FS-DOC-06Quality
Test and QA Plan
System Backup Monitoring Automation
[YourCompany.com] · IT Department · Prepared by FullSpec · [Today's Date]
This document defines the complete quality assurance and testing programme for the System Backup Monitoring automation. It is written for the FullSpec build team and covers every test case required to verify that the three agents, Backup Status Collector, Failure Classifier and Escalation Agent, and Audit Log Writer, behave correctly in isolation, in combination, and under real production conditions. Testing runs in three sequential phases. No phase begins until all cases in the previous phase have passed. The process owner joins Phase 3 to review and formally approve the live run.
01Testing approach
Testing proceeds through three phases in strict sequence. No phase begins until the previous phase fully passes, meaning every test case in that phase reaches its defined pass condition with no outstanding failures or unresolved ambiguities. This sequencing ensures that integration and end-to-end tests always run on a verified unit-level foundation, and that production credentials are never used until the full flow is confirmed stable in a sandbox environment.
1Phase 1 — Unit Testing
Estimated time: 1 to 2 business days
Scope
Each agent tested individually in isolation. Veeam API retrieval, Google Sheets comparison logic, severity classification rules, Jira ticket creation, Slack posting, PagerDuty incident creation, and Google Sheets log writing are each verified as discrete units.
Environment
Sandbox only. All Veeam, Jira, Slack, PagerDuty, and Google Sheets connections use sandbox or test-account credentials. No production data is read or written.
Credential rule
Sandbox API keys and test OAuth tokens only. Production credentials must not be loaded into any environment variable or secrets store used in Phase 1.
Pass condition
All cases T01 through T14 return the expected output and meet their individual pass criteria with zero unresolved failures.
Who runs it
FullSpec team only.
2Phase 2 — Integration and Edge Case Testing
Estimated time: 1 to 2 business days
Scope
Agent-to-agent handoffs verified end to end in the sandbox. Edge cases covering duplicates, missing data, unavailable external tools, and human timeout scenarios are tested with injected fault conditions.
Environment
Sandbox only. Fault injection used to simulate API timeouts, empty result sets, and missing expected-job entries.
Credential rule
Sandbox credentials only. Edge-case tests that simulate tool unavailability must not use production endpoints even for connectivity checks.
Pass condition
All cases T15 through T20 and E01 through E06 pass. Every edge case either recovers gracefully with a logged error or routes correctly to the defined fallback behaviour.
Who runs it
FullSpec team only.
3Phase 3 — End-to-End Production Test
Estimated time: 1 business day
Scope
Full production run traced from scheduled trigger through Veeam API retrieval, severity classification, Jira ticket creation, Slack and PagerDuty alerting, and Google Sheets audit log writing. Error log integrity and alert routing verified against live credentials.
Environment
Production. All credentials switched to live accounts. A controlled test failure is injected into the Veeam sandbox job before the run and voided from the audit log and Jira board afterward.
Credential rule
Production credentials active. The FullSpec team confirms all secrets are loaded from the production secrets store before Phase 3 begins.
Pass condition
All cases T21 through T25 pass. Process owner review and approval recorded as the final case (T25) in the FullSpec confirmation record.
Who runs it
FullSpec team runs the automation. Process owner joins to observe, verify alert receipt, and record approval at T25.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
Sandbox credentials only for all Phase 1 tests. Do not load production API keys, OAuth tokens, or service account credentials into any environment used in this phase. All Veeam, Jira, Slack, PagerDuty, and Google Sheets connections must point to test instances or sandbox accounts. If a test tool does not offer a sandbox, use a dedicated non-production test account isolated from live data.
Backup Status Collector — unit tests (T01 to T05)
ID
Scenario
Input
Expected output
Pass criteria
T01
Happy path: all jobs succeeded
Veeam API returns 10 job results, all with status Success; Google Sheets expected list contains the same 10 jobs
Structured result list with 10 entries, each labelled Pass; no failure or missing flags set
Result list contains exactly 10 Pass entries and zero Fail, Warning, or Missing entries
T02
One job failed
Veeam API returns 10 results; 1 has status Failed with error code 0x80070005; expected list contains all 10
Structured result list with 9 Pass entries and 1 Fail entry carrying the error code and job name
Fail entry includes job name, error code, start time, end time, and failure flag set to true
T03
One job missing from results
Veeam API returns 9 results all Success; Google Sheets expected list contains 10 jobs; job BackupJob_DB_Prod absent from API response
Structured result list with 9 Pass entries and 1 Missing entry for BackupJob_DB_Prod
Missing entry exists, job name matches expected list entry, status labelled Missing, no ticket reference yet
T04
One job in Warning state
Veeam API returns 10 results; 1 has status Warning with message Retried once, completed with warnings; expected list contains all 10
Structured result list with 9 Pass entries and 1 Warning entry
Warning entry carries status Warning, original error message, and job name; not classified as Fail
T05
Veeam API returns empty response
Veeam API responds HTTP 200 with an empty jobs array; expected list contains 10 jobs
Agent logs an empty-result error; all 10 expected jobs flagged as Missing; downstream agents not triggered
Error logged with timestamp; missing count equals expected-list length; no partial result list passed downstream
Failure Classifier and Escalation Agent — unit tests (T06 to T10)
ID
Scenario
Input
Expected output
Pass criteria
T06
Happy path: Warning severity routes to Slack only
Result list with 1 Warning entry for a non-critical endpoint backup job
Slack direct message sent to on-call owner; no Jira ticket created; no PagerDuty incident raised
Slack API call succeeds with HTTP 200; Jira and PagerDuty APIs not called for this entry
T07
Critical severity routes to Jira and PagerDuty
Result list with 1 Fail entry for BackupJob_DB_Prod classified as Critical by severity rules
Jira ticket created with priority Critical; PagerDuty incident created; Slack channel alert posted
Jira ticket ID returned and stored; PagerDuty incident ID returned; Slack post confirms channel delivery
T08
Informational entry produces no alert
Result list with 1 Pass entry (all fields present, status Pass)
No Jira ticket, no Slack message, no PagerDuty incident created
Zero outbound API calls to Jira, Slack, or PagerDuty for this entry
T09
Multiple failures in single run: three different severities
Result list with 1 Critical Fail, 1 Warning, and 1 Missing entry
1 Jira ticket at Critical priority, 1 Jira ticket at Warning priority, 1 PagerDuty incident (Critical only), 2 Slack alerts
Ticket count equals 2; PagerDuty incident count equals 1; Slack message count equals 2; each message references the correct job name
T10
Jira API unavailable during ticket creation
Result list with 1 Critical Fail; Jira API returns HTTP 503
Error logged with timestamp and job name; PagerDuty incident still created; Slack alert still sent; retry queued
Jira failure does not block PagerDuty or Slack calls; error entry present in log with status Jira_Failed
Audit Log Writer — unit tests (T11 to T14)
ID
Scenario
Input
Expected output
Pass criteria
T11
Happy path: all-success run logged correctly
10 Pass result entries; no ticket references
10 new rows appended to the Google Sheets audit log; each row contains date, job name, status Pass, and empty ticket reference column
Row count in sheet increases by exactly 10; all status cells read Pass; timestamp column populated with today's date
T12
Failed job logged with ticket reference
1 Fail result entry; Jira ticket ID BACK-1042 returned by Escalation Agent
1 new row appended with status Fail, ticket reference BACK-1042, error code, and timestamp
Row ticket reference column reads BACK-1042; status column reads Fail; no data truncation in error message field
T13
Missing job logged with no ticket reference
1 Missing result entry; no Jira ticket created (Warning-level missing job)
1 new row appended with status Missing, empty ticket reference, and timestamp
Row status column reads Missing; ticket reference column is empty string, not null or error
T14
Google Sheets API write fails
10 result entries; Google Sheets API returns HTTP 429 (rate limit exceeded)
Agent retries after 60 seconds; if second attempt fails, error is logged to the platform error log and an alert is sent to the FullSpec monitoring channel
Retry attempt is logged with timestamp; on second failure an internal alert fires; no data is silently dropped
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Phase 2 verifies that agents hand off correctly to one another and that the combined flow handles fault conditions gracefully. Integration tests continue the T sequence from T15. Edge case tests use a separate E sequence starting at E01.
Integration tests — agent handoffs (T15 to T20)
ID
Scenario
Input
Expected output
Pass criteria
T15
Full chain happy path: Collector to Classifier to Log Writer
Sandbox Veeam returns 10 results (9 Pass, 1 Warning); expected list complete; all downstream sandbox APIs healthy
Classifier receives result list from Collector; Log Writer receives completed classification list; 10 rows written to sandbox sheet; 1 Slack alert posted
Each agent fires in correct sequence; no agent starts before its predecessor marks completion; all 10 rows present in sheet after run
T16
Critical failure end to end: Collector detects, Classifier escalates, Log Writer records
Sandbox Veeam returns 1 Critical Fail for BackupJob_DB_Prod; Classifier severity rules classify as Critical
Jira ticket created at Critical priority; PagerDuty incident opened; Slack channel alert posted; log row written with ticket reference
Jira ticket ID present in log row; PagerDuty incident ID present; Slack message delivered; all three agents confirm completion status
T17
Collector completes but Classifier receives malformed result list
Collector produces result list with one entry missing the status field (schema violation)
Classifier logs a schema validation error for the affected entry; skips processing that entry; processes remaining entries normally; error surfaced in platform log
Malformed entry does not crash the Classifier; remaining entries processed correctly; error log entry present with field name and entry reference
T18
Log Writer fires before Classifier has completed (timing test)
Artificial delay injected into Classifier; Log Writer scheduled to fire at fixed interval overlapping with Classifier run
Log Writer waits for Classifier completion signal; does not write partial results; writes complete result set after Classifier finishes
Zero partial rows in sheet; Log Writer start timestamp is later than Classifier completion timestamp
T19
Multiple concurrent critical failures in a single run
Sandbox Veeam returns 3 Critical Fail entries simultaneously
3 separate Jira tickets created; 3 PagerDuty incidents created; 3 Slack alerts posted; 3 log rows written, each with its own ticket reference
Ticket count equals 3; incident count equals 3; Slack message count equals 3; no entries share a ticket reference; log row count equals 3
T20
Collector fails and downstream agents are not triggered
Veeam API returns HTTP 401 Unauthorized during Collector run
Collector logs authentication error; Classifier and Log Writer do not fire; platform error log records the failure with timestamp and error code
Classifier execution count equals zero for this run; Log Writer execution count equals zero; error log entry present; no empty or partial rows written to sheet
Edge case tests (E01 to E06)
ID
Scenario
Input
Expected output
Pass criteria
E01
Duplicate job result returned by Veeam API
Veeam API returns two identical entries for BackupJob_FS_01 with the same job ID and timestamp
Collector deduplicates by job ID; only one entry appears in the result list; deduplication event logged
Result list contains exactly one entry for BackupJob_FS_01; deduplication log entry present with job ID and count of duplicates removed
E02
Expected job list in Google Sheets is empty
Collector retrieves Veeam results successfully; Google Sheets expected-job tab returns zero rows
Collector logs a configuration warning: expected job list is empty; no comparisons run; downstream agents not triggered; alert sent to FullSpec monitoring channel
No result list produced; no Jira tickets or Slack alerts fired; configuration warning entry present in platform log
E03
PagerDuty API unavailable during critical escalation
Classifier identifies a Critical Fail entry; PagerDuty API returns HTTP 503 Service Unavailable
Jira ticket created successfully; Slack alert sent successfully; PagerDuty failure logged; Gmail fallback alert sent to the IT Manager email address configured during setup
Gmail fallback message delivered to configured address; PagerDuty failure logged with timestamp; Jira and Slack calls unaffected
E04
Slack workspace token expired during alert posting
Classifier attempts to post Slack alert; Slack API returns HTTP 401 invalid_auth
Slack failure logged; Jira ticket creation proceeds; PagerDuty incident proceeds (if Critical); error surfaced in platform log with token expiry indication
Jira and PagerDuty calls not blocked by Slack failure; error log entry reads Slack_Auth_Failure with timestamp
E05
New backup job added to Veeam but not yet added to the expected job list in Google Sheets
Veeam API returns a Success result for BackupJob_NewServer_01 which does not appear in the Google Sheets expected list
Collector logs the result as an unrecognised job; result is written to the audit log with status Unrecognised; no failure alert raised; informational note added to platform log
Audit log row present for BackupJob_NewServer_01 with status Unrecognised; no Jira ticket or PagerDuty incident created; log note present
E06
Human timeout: IT administrator does not acknowledge a Jira ticket within the configured SLA window
Critical Jira ticket created at 06:00; no status update by 07:00 (60-minute SLA threshold configured in setup)
Automation sends a follow-up Slack direct message to the IT Manager; PagerDuty escalation policy advances to secondary on-call contact
Second Slack message timestamp is at least 60 minutes after ticket creation; PagerDuty escalation record shows secondary contact notified
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: use an internal test address and a controlled sandbox Veeam job for any injected failure. The test Jira ticket must be labelled TEST in the summary field and voided from the Jira backlog immediately after the run. The test PagerDuty incident must be resolved and suppressed from reporting. The Google Sheets audit log row written during Phase 3 testing must be marked TEST and may be deleted after the process owner confirms the row is correct. Never use a real on-call engineer's PagerDuty contact for the injected failure test without their prior agreement.
ID
Scenario
Input
Expected output
Pass criteria
T21
Full production run: all jobs succeed overnight
Production Veeam API queried at configured schedule time; all expected jobs present and succeeded; Google Sheets expected list current
Zero Jira tickets created; zero Slack failure alerts posted; zero PagerDuty incidents raised; all job rows written to production audit log with status Pass
Production Google Sheets audit log row count increases by the number of expected jobs; all status cells read Pass; no alert API calls recorded in the platform execution log
T22
Controlled critical failure injected: end-to-end alert chain verified
One sandbox Veeam job pre-configured to report Failed with a Critical error code (simulating BackupJob_DB_Prod failure); all production APIs live
Jira ticket created in production Jira project with summary TEST: BackupJob_DB_Prod Failed at [timestamp]; PagerDuty incident created and immediately resolved by test; Slack alert posted to designated test channel; audit log row written with ticket reference
Jira ticket ID visible in production Jira board; PagerDuty incident ID confirmed by PagerDuty API response; Slack message visible in test channel within 5 minutes of trigger; audit log row contains Jira ticket reference; all test artefacts voided after confirmation
T23
Error log verification: platform execution log reviewed post-run
Both T21 and T22 runs completed; platform execution log retrieved
Execution log contains one entry per agent per run, with start timestamp, completion timestamp, status (Success or Error), and any error codes; no silent failures present
Log entry count equals six for the two-run Phase 3 session (three agents times two runs); no entry with status Error exists unless associated with the controlled T22 test failure, which itself must be logged with correct error code
T24
Audit log integrity check: production Google Sheets audit log reviewed
Audit log sheet opened after T21 and T22 runs; all rows inspected
Every job from the expected list has a row for today's date; T22 test row is clearly labelled TEST and contains the correct Jira ticket reference; no duplicate rows; no missing rows
Row count for today equals expected job count plus one (the T22 test row); no duplicate job-name rows for the same date; test row ticket reference matches the Jira ticket ID confirmed in T22
T25
Process owner review and approval
Process owner (IT Manager) observes the Phase 3 run, reviews the Jira ticket created in T22, the Slack alert received in T22, the PagerDuty incident log, and the audit log rows written in T21 and T24
Process owner confirms that all alert content is accurate, the audit log is complete, the Jira ticket fields are correctly populated, and the overall automation behaviour matches the agreed specification
Process owner verbally or in writing confirms approval; FullSpec team records the confirmation in the FullSpec project confirmation log with the process owner's name, role, date, and approval status; no separate signoff block is required
After T25 is recorded, the FullSpec team voids all test artefacts: the TEST Jira ticket is closed and labelled as a test record, the PagerDuty incident is suppressed from SLA reporting, and the TEST audit log row is either deleted or flagged as non-production in the sheet. The automation then runs unattended on its production schedule from the following morning. Any issues identified after go-live should be raised with the FullSpec team at support@gofullspec.com.
Test and QA PlanPage 4 of 4