Developer Handover Pack

Developer Handover Pack

System Backup Monitoring

This agent connects to the Veeam REST API on a daily schedule and retrieves the full overnight job result set for the past 24 hours, including job name, status, start time, end time, and error message for each entry. It then reads the expected job list from a designated Google Sheets tab and cross-references the two datasets, producing a structured result list where every job carries a status of pass, fail, warning, or missing. This agent is the entry point for the entire automation cycle and must complete successfully before any downstream agent fires. Estimated build time: 10 hours. Complexity: Moderate.

// Input
schedule.trigger: { fired_at: ISO8601 }
veeam.api.GET /api/v1/jobs/states: { timeframe: 'last_24h' }
google_sheets.read: { sheet_id: SHEET_ID, tab: 'expected_jobs', range: 'A:D' }
  -> columns: [job_name, system_type, severity_default, owner_slack_handle]

// Output
result_list: [
  {
    job_name: string,
    system_type: string,           // e.g. 'server', 'cloud', 'endpoint'
    status: 'pass' | 'fail' | 'warning' | 'missing',
    start_time: ISO8601 | null,
    end_time: ISO8601 | null,
    error_message: string | null,
    severity_default: 'critical' | 'warning' | 'informational',
    owner_slack_handle: string
  }
]
// Passed directly to Failure Classifier and Escalation Agent

This agent receives the structured result list from the Backup Status Collector and applies pre-configured severity classification rules to each anomalous entry. For any job with a status of fail, warning, or missing, it determines whether the severity should be escalated to critical based on the system_type and error_message fields. It then routes each anomaly to the correct notification channel: a Jira ticket is created per failure, a Slack alert is sent to the on-call owner, and a PagerDuty incident is triggered for any critical classification. Gmail is available as a fallback escalation channel if PagerDuty is unreachable. Jobs with a status of pass are passed through to the Audit Log Writer without action. Estimated build time: 14 hours. Complexity: Complex.

// Input
result_list: [ { job_name, system_type, status, start_time, end_time,
                  error_message, severity_default, owner_slack_handle } ]

// Classification logic (applied per entry where status != 'pass')
if system_type IN ['server', 'cloud'] AND status IN ['fail', 'missing']
  -> severity = 'critical'
if system_type == 'endpoint' OR status == 'warning'
  -> severity = 'warning'
// Override: if error_message matches known-critical error codes list
  -> severity = 'critical'  (takes precedence)

// Output (per anomalous job)
jira_ticket: {
  project_key: 'IT',
  issue_type: 'Incident',
  summary: '[BACKUP FAIL] {job_name} - {severity} - {end_time}',
  priority: severity == 'critical' ? 'High' : 'Medium',
  description: 'System: {system_type}\nStatus: {status}\nError: {error_message}\nTimestamp: {end_time}',
  labels: ['backup-monitoring', 'automated']
}
-> jira_ticket_id: string  // stored for audit log

slack_message: {
  channel: owner_slack_handle,
  text: ':warning: Backup failure: {job_name} ({severity}). Jira: {jira_ticket_url}'
}

// On critical severity only
pagerduty_incident: {
  routing_key: PAGERDUTY_INTEGRATION_KEY,
  event_action: 'trigger',
  payload: {
    summary: 'Critical backup failure: {job_name}',
    severity: 'critical',
    source: 'backup-monitoring-automation',
    custom_details: { job_name, system_type, error_message, jira_ticket_id }
  }
}

// Fallback: if PagerDuty API returns non-2xx, send Gmail alert
gmail_fallback: {
  to: IT_MANAGER_EMAIL,
  subject: '[CRITICAL BACKUP FAILURE - PAGERDUTY UNREACHABLE] {job_name}',
  body: 'PagerDuty escalation failed. Manual review required. Job: {job_name}, Error: {error_message}'
}

This agent fires after both the Backup Status Collector and the Failure Classifier and Escalation Agent have completed their run for the day. It appends one row to the central Google Sheets audit log tab for every job in the result_list, whether the job passed or failed. Each row records the job name, system type, outcome status, severity classification, Jira ticket reference (if one was created), PagerDuty incident ID (if triggered), and the run timestamp. This produces a complete, tamper-evident daily audit trail without any manual data entry. Estimated build time: 6 hours. Complexity: Simple.

// Input
enriched_result_list: [
  {
    job_name: string,
    system_type: string,
    status: 'pass' | 'fail' | 'warning' | 'missing',
    severity: 'critical' | 'warning' | 'informational' | null,
    start_time: ISO8601 | null,
    end_time: ISO8601 | null,
    error_message: string | null,
    jira_ticket_id: string | null,
    jira_ticket_url: string | null,
    pagerduty_incident_id: string | null,
    run_date: ISO8601  // date of the automation cycle
  }
]

// Output
google_sheets.appendRow: {
  sheet_id: SHEET_ID,
  tab: 'audit_log',
  row: [
    run_date,          // column A
    job_name,          // column B
    system_type,       // column C
    status,            // column D
    severity,          // column E
    error_message,     // column F
    jira_ticket_id,    // column G
    jira_ticket_url,   // column H
    pagerduty_incident_id  // column I
  ]
}
// One appendRow call per job in enriched_result_list
// On completion: emit audit_write_confirmed: { rows_written: integer, run_date: ISO8601 }

// ─────────────────────────────────────────────────────────────────────
// TRIGGER
// ─────────────────────────────────────────────────────────────────────
schedule.daily_trigger: {
  fired_at: '2025-05-05T06:00:00Z'  // configurable; default 06:00 local
}

// ─────────────────────────────────────────────────────────────────────
// AGENT 1: Backup Status Collector
// ─────────────────────────────────────────────────────────────────────

// Step 1a: Pull job results from Veeam REST API
GET https://{veeam_host}/api/v1/jobs/states
  Authorization: Bearer {VEEAM_API_TOKEN}
  Query: { startTime: 'now-24h', endTime: 'now' }
  Response fields used: [
    job.name           -> job_name
    job.result         -> raw_status  // 'Success','Warning','Failed','Running'
    job.startTime      -> start_time
    job.endTime        -> end_time
    job.lastResult.message -> error_message
  ]

// Step 1b: Read expected job list from Google Sheets
GET google_sheets.values/{SHEET_ID}/expected_jobs!A:D
  Response columns: [job_name, system_type, severity_default, owner_slack_handle]

// Step 1c: Cross-reference and assign status
FOR each row IN expected_jobs:
  IF row.job_name NOT IN veeam_results -> status = 'missing'
  ELSE IF veeam_result.raw_status == 'Failed' -> status = 'fail'
  ELSE IF veeam_result.raw_status == 'Warning' -> status = 'warning'
  ELSE -> status = 'pass'

// Handoff 1 -> 2: result_list[]
result_list: [
  { job_name, system_type, status, start_time, end_time,
    error_message, severity_default, owner_slack_handle }
]

// ─────────────────────────────────────────────────────────────────────
// AGENT 2: Failure Classifier and Escalation Agent
// ─────────────────────────────────────────────────────────────────────

// Step 2a: Filter anomalies
anomalies = result_list.filter(r => r.status != 'pass')

// Step 2b: Classify severity per entry
FOR each entry IN anomalies:
  IF entry.system_type IN ['server','cloud'] AND entry.status IN ['fail','missing']
    -> entry.severity = 'critical'
  ELSE IF entry.error_message MATCHES known_critical_error_codes[]
    -> entry.severity = 'critical'  // override
  ELSE IF entry.status == 'warning' OR entry.system_type == 'endpoint'
    -> entry.severity = 'warning'
  ELSE
    -> entry.severity = 'informational'

// Step 2c: Create Jira ticket per anomaly
POST https://{jira_host}/rest/api/3/issue
  Authorization: Basic {JIRA_API_TOKEN}
  Body: {
    project.key: 'IT',
    issuetype.name: 'Incident',
    summary: '[BACKUP FAIL] {job_name} - {severity} - {end_time}',
    priority.name: severity=='critical' ? 'High' : 'Medium',
    description: { system_type, status, error_message, end_time },
    labels: ['backup-monitoring','automated']
  }
  Response: { id: jira_ticket_id, key: jira_ticket_key, self: jira_ticket_url }
  // Dedupe check: skip creation if open issue with same job_name label exists < 24h

// Step 2d: Send Slack alert per anomaly
POST https://slack.com/api/chat.postMessage
  Authorization: Bearer {SLACK_BOT_TOKEN}
  Body: {
    channel: entry.owner_slack_handle,
    text: ':warning: Backup failure: {job_name} ({severity}). Ticket: {jira_ticket_url}'
  }

// Step 2e: PagerDuty escalation (critical only)
IF entry.severity == 'critical':
  POST https://events.pagerduty.com/v2/enqueue
    Body: {
      routing_key: {PAGERDUTY_INTEGRATION_KEY},
      event_action: 'trigger',
      payload: {
        summary: 'Critical backup failure: {job_name}',
        severity: 'critical',
        source: 'backup-monitoring-automation',
        custom_details: { job_name, system_type, error_message, jira_ticket_id }
      },
      dedup_key: '{job_name}-{run_date}'  // prevents duplicate PD incidents
    }
  Response: { status: 'success', message: 'Event processed', incident_key: pd_incident_id }
  // On non-2xx after 2 retries -> Gmail fallback to IT_MANAGER_EMAIL

// Handoff 2 -> 3: enriched_result_list[]
enriched_result_list: [
  { job_name, system_type, status, severity, start_time, end_time,
    error_message, jira_ticket_id, jira_ticket_url,
    pagerduty_incident_id, run_date }
]

// ─────────────────────────────────────────────────────────────────────
// AGENT 3: Audit Log Writer
// ─────────────────────────────────────────────────────────────────────

// Step 3a: Append one row per job (pass AND fail) to audit_log tab
POST google_sheets.spreadsheets/{SHEET_ID}/values/audit_log!A:I:append
  Authorization: Bearer {GOOGLE_SERVICE_ACCOUNT_TOKEN}
  Body: {
    valueInputOption: 'USER_ENTERED',
    values: [
      [run_date, job_name, system_type, status, severity,
       error_message, jira_ticket_id, jira_ticket_url, pagerduty_incident_id]
    ]
  }
  // Batch all rows in one batchUpdate call

// Final output
audit_write_confirmed: {
  rows_written: integer,  // total jobs logged this cycle
  run_date: ISO8601
}
// Cycle complete. IT Administrator receives no output unless a ticket,
// Slack alert, or PagerDuty incident was triggered by Agent 2.