FS-DOC-05Technical
Integration and API Spec
System Backup Monitoring
[YourCompany.com] · IT Department · Prepared by FullSpec · [Today's Date]
This document defines every integration point, authentication requirement, field mapping, and error-handling rule for the System Backup Monitoring automation. It is written for the FullSpec build team and covers all five named tools plus the orchestration layer. Every scope string, credential key, rate limit, and retry behaviour stated here is authoritative for the build. Where configuration must be completed outside the automation platform (PagerDuty on-call schedules, Jira project setup, Google Sheets schema), the exact requirements are specified so the IT team can prepare those environments in parallel with the build.
01Tool inventory
Tool
Role in automation
Auth method
Min plan required
Used by agents
Veeam
Backup job data source: REST API polling for job results
Basic Auth (API username + password) over HTTPS, token exchange to Bearer JWT
Veeam Backup and Replication v11 or later (Community edition insufficient; requires licensed instance with REST API enabled)
Agent 1 (Backup Status Collector)
Google Sheets
Expected job list store and central audit log destination
OAuth 2.0 via Google Service Account (JSON key); no user consent flow required
Google Workspace (any tier) or personal Google account with Sheets API enabled in Google Cloud project
Agent 1 (read expected list), Agent 3 (Audit Log Writer)
Jira
Incident ticket creation and tracking per backup failure
API Token + Basic Auth (user email : API token) over HTTPS; or OAuth 2.0 (3-LO) if org policy requires
Jira Cloud Free or above; Jira Software or Service Management project required
Agent 2 (Failure Classifier and Escalation Agent)
Slack
On-call failure alerts via channel post or direct message
OAuth 2.0 Bot Token (xoxb-) installed to workspace via Slack App
Slack Free or above; Bot must be added to target channels
Agent 2 (Failure Classifier and Escalation Agent)
PagerDuty
Critical failure escalation and on-call routing
Events API v2: Integration Key (service-level routing key), no OAuth required
PagerDuty Professional ($21/user/month) or above; on-call schedule and escalation policy must be pre-configured
Agent 2 (Failure Classifier and Escalation Agent)
Gmail
Critical failure email escalation to IT manager or business owner
OAuth 2.0 (Google Identity; offline access + refresh token); or Google Workspace Service Account with domain-wide delegation
Any Google account with Gmail API enabled; Workspace recommended for service account delegation
Agent 2 (Failure Classifier and Escalation Agent)
Workflow automation platform (orchestration layer)
Scheduling, credential store, inter-agent handoff, retry and error handling
Internal to platform; each external tool credential stored in encrypted platform credential store
Platform tier supporting scheduled triggers, HTTP request nodes, and at minimum 5 active workflows
All agents (1, 2, 3)
Before you connect anything: every tool connection must be validated against a sandbox or staging environment before production credentials are entered. For Veeam, use a dedicated read-only API account against a non-production backup server. For Google Sheets, use a copy of the production sheet with dummy job data. For Jira, use a dedicated test project. For Slack, use a private test channel. For PagerDuty, set the integration key to a low-urgency test service. Never test with production routing keys, live on-call schedules, or the production audit log sheet.
Integration and API SpecPage 1 of 4
FS-DOC-05Technical
02Per-tool integration detail
Veeam REST API
Polled daily by Agent 1 (Backup Status Collector) to retrieve all backup job session results from the past 24 hours. The API returns structured job session objects including job name, status, start time, end time, and error message. No webhook is available on-premises; the automation platform initiates the connection on schedule.
Auth method
HTTP POST to /api/oauth2/token with grant_type=password, username, and password. Response returns access_token (Bearer JWT) and token_type. Token must be refreshed before expiry (default 30 minutes). Store refresh_token in credential store; do not hardcode credentials.
Required scopes
RestApiSession.Read BackupJob.Read BackupJobSession.Read BackupJobSession.List Repository.Read
Trigger / polling setup
Scheduled poll only. No inbound webhook. The automation platform fires a GET request to /api/v1/backupSessions?createdAfter={ISO8601_24h_ago}&limit=500 at the configured daily time (default 06:00 local). Pagination: follow nextPageLink if present. Max 500 records per page.
Required configuration
VEEAM_HOST (base URL, e.g. https://veeam.internal:9419), VEEAM_API_USER, VEEAM_API_PASSWORD all stored in credential store. API port confirmed with IT team (default 9419). TLS certificate must be trusted by the automation platform or certificate verification explicitly configured. API user account must have the Veeam Backup Operator role at minimum.
Rate limits
Veeam REST API imposes a default session concurrency limit of 10 simultaneous connections and no documented per-minute call rate limit. At current volume (~120 jobs/month, one poll per day), a single paginated request sequence will complete in under 5 seconds. Throttling is not required at this volume.
Constraints
API must be enabled in Veeam configuration (disabled by default in some versions). On-premises deployments require network reachability from the automation platform host. Cloud Veeam instances may require VPN or allowlisted egress IP. API returns UTC timestamps; the automation layer must convert to local timezone for log entries.
// Input
GET /api/v1/backupSessions
?createdAfter=2024-05-05T06:00:00Z
&limit=500
Authorization: Bearer {access_token}
// Output (per session object)
{
"id": "uuid",
"name": "Job_ServerName_Daily",
"jobId": "uuid",
"result": "Success | Warning | Failed | None",
"startTime": "2024-05-05T01:00:00Z",
"endTime": "2024-05-05T01:45:00Z",
"message": "Error string or empty"
}Google Sheets API (Sheets v4)
Used by Agent 1 to read the expected job list and by Agent 3 (Audit Log Writer) to append daily result rows to the audit log. Two separate sheet tabs are used: one read-only reference tab (expected jobs) and one append-only log tab (audit log). The service account must have Editor access to the spreadsheet.
Auth method
Google Service Account with a downloaded JSON key file. The automation platform uses the private key to generate a self-signed JWT, exchanges it for a short-lived access token at https://oauth2.googleapis.com/token. Store the full service account JSON as GOOGLE_SERVICE_ACCOUNT_JSON in the credential store. Never hardcode the private key.
Required scopes
https://www.googleapis.com/auth/spreadsheets
Webhook / trigger setup
No inbound webhook. Sheets is read and written by the automation platform on schedule. Agent 1 reads the expected job list after the Veeam poll. Agent 3 appends log rows after Agent 2 completes. Both operations are synchronous REST calls.
Required configuration
GOOGLE_SPREADSHEET_ID stored in credential store (not hardcoded). EXPECTED_JOBS_RANGE stored as config variable (default: ExpectedJobs!A2:D). AUDIT_LOG_RANGE stored as config variable (default: AuditLog!A:H). Sheet structure: ExpectedJobs tab columns: job_name, system_type, criticality, owner_slack_id. AuditLog tab columns: run_date, job_name, status, severity, jira_ticket_key, pagerduty_incident_id, error_message, logged_at. Service account email must be added as Editor to the spreadsheet by the IT team before build.
Rate limits
Google Sheets API quota: 300 write requests per minute per project; 60 read requests per minute per user. At current volume (one read of ~120 rows and one batch append of ~120 rows per day), usage is well under 1% of quota. Throttling is not required. Batch append using values.append with valueInputOption=RAW is preferred over individual row writes.
Constraints
Sheet must exist before the automation runs; the build team creates it during the Discovery and Access Setup stage. Column order is fixed; any manual reordering of the sheet will break field mapping. The AuditLog tab must never have data filters or protected ranges that block append operations.
// Input: Read expected job list
GET https://sheets.googleapis.com/v4/spreadsheets/{SPREADSHEET_ID}
/values/{EXPECTED_JOBS_RANGE}
// Output: Expected job array
{ "values": [
["Job_ServerName_Daily", "server", "critical", "U012AB3CD"],
["Job_CloudVolume_Nightly", "cloud", "warning", "U098XY7ZZ"]
]
}
// Input: Append audit log row (Agent 3)
POST https://sheets.googleapis.com/v4/spreadsheets/{SPREADSHEET_ID}
/values/{AUDIT_LOG_RANGE}:append
?valueInputOption=RAW&insertDataOption=INSERT_ROWS
Body: { "values": [["2024-05-05", "Job_ServerName_Daily",
"Failed", "critical", "OPS-1042", "PD-8821",
"Agent service not running", "2024-05-05T06:03:12Z"]] }Jira Cloud REST API (v3)
Used by Agent 2 (Failure Classifier and Escalation Agent) to create one incident ticket per failed or missing backup job. Ticket fields are pre-populated from the structured result list produced by Agent 1. No webhook from Jira is consumed by the automation; all calls are outbound from the platform.
Auth method
HTTP Basic Auth: base64(user_email:api_token) in the Authorization header. API token generated at id.atlassian.com and stored as JIRA_API_TOKEN in the credential store. JIRA_USER_EMAIL and JIRA_BASE_URL (e.g. https://yourcompany.atlassian.net) also stored in credential store. Do not hardcode.
Required scopes
read:jira-work write:jira-work read:jira-user (OAuth 2.0 3-LO scopes if switching to OAuth: read:issue:jira write:issue:jira read:project:jira)
Webhook / trigger setup
No inbound webhook consumed. The automation platform posts to /rest/api/3/issue to create tickets. The Jira ticket key returned in the response (e.g. OPS-1042) is captured and passed downstream to Agent 3 for the audit log entry.
Required configuration
JIRA_PROJECT_KEY stored in credential store (e.g. OPS). JIRA_ISSUE_TYPE_ID stored in credential store (confirm numeric ID for the Bug or Incident issue type in the target project; do not use the display name string). Priority name-to-severity mapping configured as: critical -> Highest, warning -> Medium, informational -> Low. Custom field IDs for Affected System (customfield_10050) and Backup Job Name (customfield_10051) must be confirmed against the live Jira instance during Discovery.
Rate limits
Jira Cloud rate limit: 10,000 API requests per 10 minutes per account. At current volume (~120 jobs/month, with failures typically a small subset), the automation will create at most a few tickets per run. Throttling is not required. If a run produces more than 20 failure tickets, add a 200ms delay between successive POST calls as a precaution.
Constraints
The Jira project must exist and the API user must have Create Issue and Edit Issue permissions in that project. Required fields beyond summary and description must be identified during Discovery and included in the POST body; Jira rejects requests with missing required fields with a 400 error. Jira Service Management projects may have additional mandatory fields (e.g. Request Type).
// Input: Create issue
POST {JIRA_BASE_URL}/rest/api/3/issue
Authorization: Basic base64(email:token)
Content-Type: application/json
{
"fields": {
"project": { "key": "{JIRA_PROJECT_KEY}" },
"issuetype": { "id": "{JIRA_ISSUE_TYPE_ID}" },
"summary": "Backup failure: Job_ServerName_Daily [critical]",
"description": { "type": "doc", "version": 1,
"content": [{ "type": "paragraph", "content":
[{ "type": "text", "text": "Error: Agent service not running. Run date: 2024-05-05" }]
}]
},
"priority": { "name": "Highest" },
"customfield_10050": "ServerName",
"customfield_10051": "Job_ServerName_Daily"
}
}
// Output
{ "id": "10204", "key": "OPS-1042",
"self": "https://yourcompany.atlassian.net/rest/api/3/issue/10204" }Slack Web API
Used by Agent 2 to post failure alerts to a designated channel or as a direct message to the on-call owner identified in the expected job list. The Slack app is installed to the workspace once during setup; the bot token is stored in the credential store.
Auth method
OAuth 2.0 Bot Token (xoxb- prefix). Token obtained by installing the Slack App to the workspace via the Slack App management console. Store as SLACK_BOT_TOKEN in credential store. Do not hardcode or commit the token.
Required scopes
chat:write chat:write.public users:read users:read.email channels:read groups:read im:write
Webhook / trigger setup
No inbound webhook consumed. The automation platform calls chat.postMessage. For direct messages, the owner_slack_id from the expected job list is used as the channel value (Slack accepts a user ID as the channel parameter for DMs). For channel posts, the SLACK_ALERT_CHANNEL_ID stored in the credential store is used.
Required configuration
SLACK_BOT_TOKEN in credential store. SLACK_ALERT_CHANNEL_ID in credential store (the #backup-alerts channel ID, not the name; obtain from channel details). owner_slack_id per job stored in the ExpectedJobs Google Sheet column D. Message template: include job name, severity, Jira ticket link, error summary, and run timestamp. Do not post raw API credentials or full error stack traces to Slack.
Rate limits
Slack Web API rate limit: Tier 3 methods (chat.postMessage) allow 50 requests per minute. At current volume, a worst-case run posting alerts for all 120 jobs would require a 200ms delay between posts. In practice, only failed or missing jobs trigger alerts; this is expected to be under 10 messages per run. Throttling is not required at current volume but a 100ms inter-message delay should be implemented defensively.
Constraints
The bot must be invited to any private channel it needs to post in. Public channels work with chat:write.public without an invitation. The Slack App must not be revoked or reinstalled during operation; doing so invalidates the bot token and all alerts will fail silently unless error handling catches the 401 response.
// Input: Post alert message
POST https://slack.com/api/chat.postMessage
Authorization: Bearer {SLACK_BOT_TOKEN}
Content-Type: application/json
{
"channel": "U012AB3CD",
"text": "Backup failure detected",
"blocks": [
{ "type": "section", "text": { "type": "mrkdwn",
"text": ":red_circle: *CRITICAL* | Job_ServerName_Daily failed\n
Error: Agent service not running\n
Ticket: <https://yourcompany.atlassian.net/browse/OPS-1042|OPS-1042>\n
Run date: 2024-05-05 06:00 UTC" } }
]
}
// Output
{ "ok": true, "channel": "U012AB3CD", "ts": "1714893712.000100" }PagerDuty Events API v2
Used by Agent 2 for critical-severity failures only. The automation platform sends a trigger event to the PagerDuty Events API using a service-level integration key. PagerDuty then applies the pre-configured on-call schedule and escalation policy to route the incident to the correct responder. The automation does not manage PagerDuty schedules; those must be configured in PagerDuty independently.
Auth method
Events API v2 uses an Integration Key (routing_key), not OAuth. The routing_key is a 32-character string obtained from the PagerDuty service integration settings. Store as PAGERDUTY_ROUTING_KEY in credential store. No Authorization header is used; the key is included in the JSON body.
Required scopes
No OAuth scopes required for Events API v2. The routing_key implicitly scopes the event to the target service. If the REST API v2 is used for status reads (optional), an API key with Read access is required.
Webhook / trigger setup
No inbound webhook consumed by the automation. The platform POSTs a trigger event to https://events.pagerduty.com/v2/enqueue. The dedup_key field should be set to the Jira ticket key (e.g. OPS-1042) to prevent duplicate PagerDuty incidents if the automation retries.
Required configuration
PAGERDUTY_ROUTING_KEY in credential store. PagerDuty service, on-call schedule, and escalation policy must be created and active in PagerDuty before the first production run. The automation does not create or modify these. PAGERDUTY_SEVERITY_MAP config: critical -> critical, warning -> warning (PagerDuty severity enum). dedup_key must be unique per failure event; use jira_ticket_key + run_date as the composite key.
Rate limits
PagerDuty Events API v2 accepts up to 2,000 events per minute per integration key. At current volume (at most a handful of critical failures per day), rate limiting is irrelevant. No throttling required.
Constraints
Events API v2 is fire-and-forget: the API returns 202 Accepted immediately but does not confirm that the on-call engineer was reached. The automation must not treat a 202 response as confirmation of human acknowledgement. The routing_key must correspond to an active, non-disabled PagerDuty service or events will be silently dropped by PagerDuty.
// Input: Trigger critical incident
POST https://events.pagerduty.com/v2/enqueue
Content-Type: application/json
{
"routing_key": "{PAGERDUTY_ROUTING_KEY}",
"event_action": "trigger",
"dedup_key": "OPS-1042-2024-05-05",
"payload": {
"summary": "CRITICAL backup failure: Job_ServerName_Daily",
"severity": "critical",
"source": "FullSpec Backup Monitor",
"timestamp": "2024-05-05T06:03:12Z",
"custom_details": {
"job_name": "Job_ServerName_Daily",
"error_message": "Agent service not running",
"jira_ticket": "OPS-1042",
"system_type": "server"
}
}
}
// Output
{ "status": "success", "message": "Event processed",
"dedup_key": "OPS-1042-2024-05-05" }Gmail API (Google Workspace)
Used by Agent 2 for critical-failure email escalation to the IT manager or business owner, supplementing the PagerDuty alert. The Gmail API sends a structured email from a designated service account or delegated mailbox. This channel is used only for critical severity classifications.
Auth method
OAuth 2.0 with offline access and refresh token stored in credential store as GMAIL_REFRESH_TOKEN plus GMAIL_CLIENT_ID and GMAIL_CLIENT_SECRET. If using Google Workspace Service Account with domain-wide delegation, store GOOGLE_SERVICE_ACCOUNT_JSON and set GMAIL_DELEGATE_EMAIL to the sending address. Refresh tokens must be rotated if the authorised user changes.
Required scopes
https://www.googleapis.com/auth/gmail.send
Webhook / trigger setup
No inbound webhook consumed. The automation platform calls users.messages.send via POST to https://gmail.googleapis.com/gmail/v1/users/me/messages/send. The message body is MIME-formatted and base64url-encoded before sending.
Required configuration
GMAIL_FROM_ADDRESS stored in credential store (the sending address). GMAIL_ESCALATION_TO stored in credential store (IT manager email; can be a comma-separated list). Email subject template: '[CRITICAL] Backup failure: {job_name} - {run_date}'. Body must include job name, system type, error message, Jira ticket link, PagerDuty incident URL, and run timestamp. Do not include raw credentials or internal hostnames in the email body.
Rate limits
Gmail API quota: 250 quota units per second per user; users.messages.send costs 100 units. Effective limit is approximately 2 sends per second. At current volume (critical failures are a small subset of a small daily failure count), no throttling is required.
Constraints
Gmail API send scope must be explicitly granted; it is not included in the broader gmail readonly or gmail.modify scopes. If Google Workspace security policy restricts less secure app access or OAuth app verification, the Google Cloud project hosting the OAuth credentials must be verified or internal-only approval applied. The sending address must be an active mailbox.
// Input: Send escalation email
POST https://gmail.googleapis.com/gmail/v1/users/me/messages/send
Authorization: Bearer {access_token}
{ "raw": "<base64url-encoded MIME message>" }
// Decoded MIME structure
From: backup-monitor@yourcompany.com
To: itmanager@yourcompany.com
Subject: [CRITICAL] Backup failure: Job_ServerName_Daily - 2024-05-05
Content-Type: text/plain; charset=utf-8
CRITICAL backup failure detected.
Job: Job_ServerName_Daily | System: server
Error: Agent service not running
Jira: https://yourcompany.atlassian.net/browse/OPS-1042
PagerDuty: https://yourcompany.pagerduty.com/incidents/PD-8821
Run date: 2024-05-05 06:00 UTC
// Output
{ "id": "18f3c2d9a1b4e5f6", "threadId": "18f3c2d9a1b4e5f6",
"labelIds": ["SENT"] }Integration and API SpecPage 2 of 4
FS-DOC-05Technical
03Field mappings between tools
The following tables define the exact field-level data flow at each agent handoff point. All source and destination field names are shown in monospace as they appear in the API response or request body. Any transformation applied between source and destination is noted in the Destination field column.
Handoff 1: Veeam API to Agent 1 structured result list (Backup Status Collector output)
Source tool
Source field
Destination tool
Destination field
Veeam
`backupSession.name`
Internal result object
`job_name`
Veeam
`backupSession.result`
Internal result object
`raw_status` (Success / Warning / Failed / None)
Veeam
`backupSession.startTime`
Internal result object
`start_time` (ISO 8601 UTC)
Veeam
`backupSession.endTime`
Internal result object
`end_time` (ISO 8601 UTC)
Veeam
`backupSession.message`
Internal result object
`error_message` (empty string if none)
Veeam
`backupSession.jobId`
Internal result object
`veeam_job_id`
Google Sheets (ExpectedJobs tab)
`Column A: job_name`
Internal result object
`expected_job_name` (used for missing-job detection)
Google Sheets (ExpectedJobs tab)
`Column B: system_type`
Internal result object
`system_type` (server / cloud / endpoint)
Google Sheets (ExpectedJobs tab)
`Column C: criticality`
Internal result object
`configured_criticality` (critical / warning / informational)
Google Sheets (ExpectedJobs tab)
`Column D: owner_slack_id`
Internal result object
`owner_slack_id`
Derived (missing job logic)
`expected_job_name NOT IN veeam results`
Internal result object
`raw_status` set to `Missing`
Handoff 2: Agent 1 structured result list to Agent 2 (Failure Classifier and Escalation Agent)
Source tool
Source field
Destination tool
Destination field
Internal result object
`job_name`
Jira (issue fields)
`fields.summary` (prefixed with severity label)
Internal result object
`job_name`
Jira (custom field)
`customfield_10051` (Backup Job Name)
Internal result object
`system_type`
Jira (custom field)
`customfield_10050` (Affected System)
Internal result object
`error_message`
Jira (description body)
`fields.description.content[0].content[0].text`
Internal result object
`severity` (derived)
Jira (priority)
`fields.priority.name` (Highest / Medium / Low)
Internal result object
`job_name`
Slack message
`blocks[0].text.text` (job name in alert body)
Internal result object
`error_message`
Slack message
`blocks[0].text.text` (error line in alert body)
Internal result object
`owner_slack_id`
Slack API
`channel` parameter (direct message target)
Internal result object
`severity` (critical only)
PagerDuty payload
`payload.severity`
Internal result object
`job_name`
PagerDuty payload
`payload.summary`
Internal result object
`error_message`
PagerDuty payload
`payload.custom_details.error_message`
Internal result object
`system_type`
PagerDuty payload
`payload.custom_details.system_type`
Internal result object
`severity` (critical only)
Gmail body
Subject line prefix `[CRITICAL]` and body severity field
Internal result object
`start_time`
Gmail body
`Run date:` field in email body
Jira API response
`key` (e.g. OPS-1042)
Internal result object
`jira_ticket_key` (passed to Agent 3)
PagerDuty API response
`dedup_key`
Internal result object
`pagerduty_incident_id` (passed to Agent 3)
Handoff 3: Agent 2 output to Agent 3 (Audit Log Writer) for Google Sheets append
Source tool
Source field
Destination tool
Destination field
Internal result object
`run_date` (derived from trigger timestamp)
Google Sheets AuditLog!A
`run_date` (YYYY-MM-DD)
Internal result object
`job_name`
Google Sheets AuditLog!B
`job_name`
Internal result object
`raw_status`
Google Sheets AuditLog!C
`status` (Success / Warning / Failed / Missing)
Internal result object
`severity`
Google Sheets AuditLog!D
`severity` (critical / warning / informational / none)
Jira API response
`jira_ticket_key`
Google Sheets AuditLog!E
`jira_ticket_key` (empty string for successful jobs)
PagerDuty API response
`pagerduty_incident_id`
Google Sheets AuditLog!F
`pagerduty_incident_id` (empty string if not escalated)
Internal result object
`error_message`
Google Sheets AuditLog!G
`error_message` (empty string for successful jobs)
Platform runtime
`current UTC timestamp`
Google Sheets AuditLog!H
`logged_at` (ISO 8601 UTC)
All field names in the AuditLog tab are positional (column order matters). Do not insert, delete, or reorder columns after the sheet is created. If new fields are needed, append them to the right and update the AUDIT_LOG_RANGE config variable and the Handoff 3 mapping accordingly.
Integration and API SpecPage 3 of 4
FS-DOC-05Technical
04Build stack and orchestration
Orchestration layout
Three separate workflows are created in the automation platform, one per agent: Workflow 1 (Backup Status Collector), Workflow 2 (Failure Classifier and Escalation Agent), Workflow 3 (Audit Log Writer). A shared credential store holds all API keys, tokens, and configuration constants. Workflows 2 and 3 are triggered by the successful completion output of the preceding workflow, not by an independent schedule. This ensures sequential execution and that Agent 3 only logs results after Agent 2 has finished routing all alerts and tickets.
Agent 1 trigger mechanism
Scheduled trigger. Workflow 1 fires at a fixed daily time configured during setup (default 06:00 local time). No inbound webhook. The schedule is managed by the automation platform's built-in cron-equivalent scheduler. The trigger time must be at least 30 minutes after the latest expected backup job completion time to ensure all overnight results are available in Veeam.
Agent 2 trigger mechanism
Event-based trigger fired by Workflow 1 completion. When Workflow 1 finishes successfully, it passes the structured result list (array of job result objects) to Workflow 2 via the platform's inter-workflow data passing mechanism (e.g. shared in-memory variable, workflow output object, or platform execution chaining). No external webhook is involved. If Workflow 1 fails, Workflow 2 must not execute; the orchestration layer must enforce this dependency.
Agent 3 trigger mechanism
Event-based trigger fired by Workflow 2 completion. When Workflow 2 finishes (including partial completion where some alerts were sent but others failed), it passes the enriched result list (including jira_ticket_key and pagerduty_incident_id per job) to Workflow 3. Agent 3 must execute even if some Agent 2 alert actions failed, so that the audit log remains complete. Agent 3 marks rows with a flag if the corresponding alert or ticket action failed.
Signature validation (Veeam)
Veeam REST API uses HTTPS mutual authentication. TLS must be enforced on all requests; plaintext HTTP connections to the Veeam API must be rejected by the automation platform configuration. The Bearer JWT returned at login is validated by Veeam on each request; no additional signature validation is required on the platform side.
Signature validation (Slack)
Not applicable for outbound-only Slack usage. If inbound webhooks from Slack are added in future, Slack request signing (HMAC-SHA256 using SLACK_SIGNING_SECRET) must be implemented.
Credential store policy
All secrets are stored in the automation platform's encrypted credential store. No secret appears in workflow node configuration fields, code expressions, or log outputs. Credential names follow the convention defined in the code block below. Access to the credential store is restricted to the FullSpec build account and the designated IT admin service account.
Credential store contents (all entries encrypted at rest)
// Veeam
VEEAM_HOST = "https://veeam.internal:9419"
VEEAM_API_USER = "<api-read-only-user>"
VEEAM_API_PASSWORD = "<api-user-password>"
// Google (shared for Sheets and Gmail if using Service Account)
GOOGLE_SERVICE_ACCOUNT_JSON = "<full JSON key file contents>"
GOOGLE_SPREADSHEET_ID = "<sheets-doc-id-from-url>"
EXPECTED_JOBS_RANGE = "ExpectedJobs!A2:D"
AUDIT_LOG_RANGE = "AuditLog!A:H"
// Gmail (OAuth flow; only if NOT using service account delegation)
GMAIL_CLIENT_ID = "<oauth-client-id>"
GMAIL_CLIENT_SECRET = "<oauth-client-secret>"
GMAIL_REFRESH_TOKEN = "<offline-refresh-token>"
GMAIL_FROM_ADDRESS = "backup-monitor@yourcompany.com"
GMAIL_ESCALATION_TO = "itmanager@yourcompany.com"
// Jira
JIRA_BASE_URL = "https://yourcompany.atlassian.net"
JIRA_USER_EMAIL = "automation@yourcompany.com"
JIRA_API_TOKEN = "<atlassian-api-token>"
JIRA_PROJECT_KEY = "OPS"
JIRA_ISSUE_TYPE_ID = "<numeric-issue-type-id>"
// Slack
SLACK_BOT_TOKEN = "xoxb-<workspace-bot-token>"
SLACK_ALERT_CHANNEL_ID = "<channel-id-not-name>"
// PagerDuty
PAGERDUTY_ROUTING_KEY = "<32-char-integration-key>"
// Platform config constants (not secrets; stored as env vars)
SCHEDULE_TIME_LOCAL = "06:00"
LOOKBACK_HOURS = 24
MAX_VEEAM_PAGE_SIZE = 500
SLACK_INTER_MESSAGE_DELAY_MS = 100
JIRA_INTER_REQUEST_DELAY_MS = 200
05Error handling and retry logic
Every integration point has a defined failure behaviour. Unhandled exceptions must never fail silently. If an error cannot be resolved by the retry logic defined below, the automation platform must write the failure to a designated error log and send an alert to the IT administrator via a fallback channel (direct email to GMAIL_ESCALATION_TO) before halting the affected workflow branch.
Integration
Scenario
Required behaviour
Veeam REST API
Authentication failure (401 Unauthorized on token request)
Retry token request once after 10 seconds. If second attempt fails, halt Workflow 1 entirely, log error with timestamp, and send fallback email alert to GMAIL_ESCALATION_TO with subject '[AUTOMATION ERROR] Veeam auth failed - manual backup check required'. Do not proceed to Workflow 2 or 3.
Veeam REST API
Job results API call returns 5xx server error
Retry up to 3 times with exponential backoff: 30s, 90s, 270s. If all retries fail, halt Workflow 1, log error, and send fallback email alert. IT administrator must perform manual backup check.
Veeam REST API
API returns empty result set (no sessions in past 24 hours)
Treat as a critical anomaly, not a success. Log the empty result, generate a Missing alert for all expected jobs, and pass the full missing-job list to Workflow 2 for alerting. Do not silently skip the run.
Veeam REST API
Pagination: nextPageLink present but subsequent page request fails
Retry the failed page request up to 3 times with 15s backoff. If all retries fail, process results retrieved so far, flag the result set as incomplete in the log, and include a warning in all downstream alerts and the audit log row.
Google Sheets API
Read of ExpectedJobs tab returns 403 or 404 (permission or sheet not found)
Halt Workflow 1. Log error with full API response. Send fallback email alert. Do not proceed to Workflow 2 or 3. This error requires human intervention to restore sheet access.
Google Sheets API
Append to AuditLog tab fails (quota exceeded or 5xx error)
Retry up to 3 times with 30s backoff. If all retries fail, write the failed rows to a platform-internal error queue and send a fallback email listing all unlogged job results. An IT administrator must manually append these rows. Log the failure with full error detail.
Jira API
Ticket creation returns 400 Bad Request (missing required field or invalid field value)
Log the full 400 response body. Do not retry (retrying will produce the same error). Skip Jira ticket creation for the affected job, continue with Slack and PagerDuty alerts, and include a note in the audit log row that the Jira ticket was not created. Send a fallback email listing all jobs for which ticket creation failed.
Jira API
Ticket creation returns 429 Too Many Requests
Respect the Retry-After header value. Pause the workflow for the specified duration (minimum 10 seconds) then retry once. If the retry also returns 429, apply a 60-second wait and retry a second time. If three attempts all fail, log the error and proceed as per the 400 handling above.
Slack API
chat.postMessage returns error: channel_not_found or not_in_channel
Log the error with the channel ID that failed. Fall back to posting to SLACK_ALERT_CHANNEL_ID (the default alert channel). If the fallback channel also fails, send the alert via Gmail fallback email. Do not fail silently.
PagerDuty Events API
POST to /v2/enqueue returns 400 or 429
For 400: log the full response body; the routing key or payload is invalid. Send the critical alert via Gmail fallback email immediately and flag the incident in the audit log as 'PagerDuty failed'. For 429: retry after 60 seconds, then 120 seconds. If three retries all fail, use the Gmail fallback.
Gmail API
Send fails (OAuth token expired or 5xx)
For token expiry: attempt token refresh using GMAIL_REFRESH_TOKEN and retry once. For 5xx: retry up to 2 times with 30s backoff. If Gmail send fails completely and this is the last fallback channel, write the full failure record to the platform error log and attempt one final delivery via the Slack alert channel. If Slack also fails, the platform must not exit without writing to the internal error log.
Orchestration layer (inter-workflow handoff)
Workflow 1 completes but result list is malformed or missing expected fields
Workflow 2 must validate the input schema before processing. If any required field (job_name, raw_status, system_type) is absent, skip that job record, log the malformed entry, and continue processing the remaining records. After processing, include a count of skipped records in the fallback email summary.
The fallback email address (GMAIL_ESCALATION_TO) is the last-resort alert channel for automation failures. If the Gmail API itself is the failing integration, the Slack alert channel is the secondary fallback. At least one of these two channels must be reachable for the failure notification to be delivered. If both fail, the platform internal error log is the only record and manual review of the log must be part of the daily IT operations checklist.
Integration and API SpecPage 4 of 4