FS-DOC-06Quality
Test and QA Plan
Weekly Business Review — Automation QA
[YourCompany.com] · Management Department · Prepared by FullSpec · [Today's Date]
This document defines every test case the FullSpec team runs before the Weekly Business Review automation goes live. It covers three sequential phases: unit tests per agent, integration and edge-case tests across agent handoffs, and a final end-to-end production run. No phase begins until the previous phase has fully passed. Your team joins Phase 3 to verify the production output and record approval. Reach the FullSpec team at support@gofullspec.com with any questions.
01Testing approach
Testing is structured in three sequential phases. Phase 1 validates each agent in isolation using sandbox credentials. Phase 2 validates agent-to-agent handoffs and exercises edge cases including missing data, duplicate records, unavailable tools, and human timeout scenarios. Phase 3 runs a single full end-to-end pass against production systems with a test record that is voided immediately after the run. No phase begins until every test case in the previous phase carries a Pass status.
1Phase 1 — Unit Testing
Each agent tested independently in sandbox
Scope
Data Collection Agent, Review Narrative Agent, Distribution Agent tested individually. No inter-agent calls exercised.
Environment
Sandbox only. Xero demo company, HubSpot sandbox portal, test Google Sheet, test Google Slides deck, Gmail test alias, Slack test workspace.
Credential rule
No production API keys, OAuth tokens, or live distribution lists may be used during Phase 1. Sandbox credentials must be stored in the automation platform's test-environment variable set.
Pass condition
All cases T01 through T14 return the expected output with no unhandled errors. Any failure blocks progression to Phase 2.
Who runs it
FullSpec team only.
2Phase 2 — Integration and Edge-Case Testing
Agent handoffs and failure-mode coverage
Scope
End-to-end agent chain in sandbox, plus isolated edge-case scenarios covering duplicate data, missing fields, tool unavailability, and human review timeout.
Environment
Sandbox throughout. Same credential set as Phase 1. No production systems contacted.
Credential rule
Sandbox credentials only. Edge-case scenarios that simulate tool failure must use mocked or stubbed responses, not deliberate calls to live endpoints.
Pass condition
All cases T15 through T21 and E01 through E06 pass or produce the documented graceful-failure behaviour. Any unhandled exception is treated as a failure.
Who runs it
FullSpec team only.
3Phase 3 — End-to-End Production Test
Live systems, single run, owner sign-off
Scope
Single complete automation run against production Xero, HubSpot, Google Sheets, Google Slides, Gmail, and Slack. Full trace from trigger to archive.
Environment
Production. Internal test addresses and a clearly labelled test record are used. The test Slides deck link is not sent to the real leadership distribution list.
Credential rule
Production OAuth tokens and API keys in use. The Gmail send must target an internal test alias only. The Slack post must target a private test channel, not the live leadership channel.
Pass condition
All cases T22 through T26 pass, including the process owner review-and-approval case (T26). Approval is recorded in the FullSpec confirmation log.
Who runs it
FullSpec team executes T22 through T25. Process owner (Operations Manager) participates in T26 to review output and confirm go-live readiness.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
All Phase 1 tests must be run against sandbox credentials only. Do not connect production Xero, HubSpot, or Google accounts at this stage. Sandbox API keys must be stored in the test-environment variable set within the automation platform and must not be substituted with live tokens for convenience.
Data Collection Agent — estimated build time: 7 hours — complexity: Moderate
ID
Scenario
Input
Expected Output
Pass Criteria
T01
Happy path: scheduled trigger fires, Xero and HubSpot return full data sets
Sunday 8 PM cron fires; Xero demo returns invoiced revenue $42,500, payments $38,000, AR $4,500; HubSpot sandbox returns 14 open deals, pipeline $310,000, 3 closed, 5 stage-moved
New row appended to master Google Sheet with all 8 fields populated; timestamp column set to trigger date
Row exists in Sheet; all 8 fields non-null; no error log entry; execution time under 45 seconds
T02
Xero API returns HTTP 200 but empty invoice list (no invoices this week)
Xero sandbox configured to return zero invoices for the test period
Row appended with revenue fields set to 0; zero-value flag written to Sheet column M; alert triggered to error log
Row present; revenue fields equal 0; flag column M = TRUE; error log entry: 'Zero revenue returned from Xero'
T03
Xero API returns HTTP 401 (expired OAuth token)
Xero sandbox OAuth token manually expired before run
Agent halts; error written to run log with message 'Xero authentication failed — token refresh required'; HubSpot call does not proceed; no partial row written
No new Sheet row; run log contains 401 error entry; downstream agents not triggered
T04
HubSpot API returns HTTP 429 (rate limit hit)
HubSpot sandbox rate-limit mock returns 429 on first call; succeeds on retry after 10 seconds
Agent retries after 10-second back-off; second call succeeds; full row appended to Sheet
Row appended with all fields; run log shows one 429 retry event; total execution under 90 seconds
T05
Google Sheets append fails due to permission error (HTTP 403)
Test Google Sheet set to read-only for the automation service account
Agent logs 'Google Sheets write failed — permission denied'; no partial row written; run halted; notification queued for operator
No new row in Sheet; error log entry with 403 code; no downstream agents triggered
Review Narrative Agent — estimated build time: 9 hours — complexity: Complex
ID
Scenario
Input
Expected Output
Pass Criteria
T06
Happy path: all metrics within target thresholds
Sheet row with revenue $42,500 (target $40,000), pipeline $310,000 (target $280,000), 3 deals closed (target 2); all thresholds in reference tab
Narrative text written to Sheet column N: 2 to 4 sentences, no exception flags, positive tone confirmed; exception flag column O = FALSE
Column N non-empty; length 80 to 400 characters; column O = FALSE; no critical-metric flag in column P
T07
One metric below threshold: invoiced revenue $31,000 against target $40,000 (miss by 22.5%)
Sheet row with revenue $31,000; all other metrics on target; threshold tab unchanged
Narrative flags revenue shortfall; exception summary written to column N; column O = TRUE; column P lists 'Invoiced Revenue'
Column O = TRUE; column P contains 'Invoiced Revenue'; narrative text references revenue shortfall specifically; human-review flag written to run metadata
T08
Multiple metrics below threshold: revenue miss and pipeline miss simultaneously
Revenue $29,000 (target $40,000); pipeline $190,000 (target $280,000); deals closed 1 (target 2)
Narrative addresses all three exceptions; column P lists all three metric names; column O = TRUE; human-review flag set
Column P lists three metric names; narrative body references each; column O = TRUE; run metadata human_review_required = true
T09
Reference tab thresholds missing (tab deleted or renamed)
Sheet row present; reference tab 'Targets' renamed to 'Targets_OLD' to simulate misconfiguration
Agent logs 'Threshold reference tab not found — narrative generation halted'; no narrative written; critical error raised
Column N empty; error log entry referencing missing tab name; Distribution Agent not triggered
T10
AI narrative generation returns empty string (model timeout or API error)
Simulate AI model endpoint returning empty response after 30-second timeout
Agent writes fallback message to column N: 'Narrative unavailable this week — please review metrics manually.'; column O = TRUE; human-review flag set
Column N equals fallback string exactly; column O = TRUE; Distribution Agent proceeds with fallback text; error logged
Distribution Agent — estimated build time: 6 hours — complexity: Moderate
ID
Scenario
Input
Expected Output
Pass Criteria
T11
Happy path: deck updated, email sent, Slack posted, archive saved
Populated Sheet row; narrative in column N; Google Slides sandbox deck with named placeholders; Gmail test alias; Slack test channel
Slides placeholders replaced with current figures and narrative; email delivered to test alias with correct subject and deck link; Slack message posted to test channel; deck archived to Drive test folder with date in filename
Slides diff shows all placeholders replaced; Gmail delivery confirmed in sent items; Slack message present in test channel; Drive file exists with correct naming convention
T12
Google Slides placeholder name mismatch (one placeholder renamed in template)
Slide deck has placeholder {{revenue_this_week}} renamed to {{rev_week}}; agent config still maps to {{revenue_this_week}}
Agent logs unmapped placeholder warning; inserts value into slide at best-match location or skips and flags; email still sends with a warning note in the body
Run log contains placeholder mismatch warning; email body contains note 'One slide field could not be mapped — please verify deck'; Slack post includes same note
T13
Gmail send fails (SMTP error / quota exceeded)
Gmail sandbox configured to reject outbound send
Agent logs Gmail failure; Slack message still posts successfully with a note that email delivery failed; error written to run log
Slack message present in test channel referencing email failure; error log entry for Gmail; no duplicate Slack posts
T14
Slack API returns HTTP 404 (channel not found)
Slack test channel ID set to an invalid value
Agent logs 'Slack channel not found'; email still sent successfully; error entry written to run log; no retry loop
Email delivered to test alias; Slack error logged with channel ID; no infinite retry; run marked as completed with warnings
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Phase 2 first validates the handoffs between agents in sequence, then exercises the edge-case scenarios that fall outside normal operation. All tests remain in the sandbox environment. Test IDs continue from where Phase 1 ended.
ID
Scenario
Input
Expected Output
Pass Criteria
T15
Full agent chain: Data Collection fires, triggers Review Narrative, which triggers Distribution — all agents pass in sequence
Valid sandbox data from Xero and HubSpot; all metrics on target; all credentials valid
Data Collection writes Sheet row; Review Narrative writes narrative and no exception flag; Distribution sends email and Slack; archive created — end to end under 4 minutes
Sheet row present; narrative in column N; column O = FALSE; email in sent items; Slack post in test channel; Drive archive file exists; total run time under 4 minutes
T16
Data Collection writes partial row, then fails — Review Narrative must not fire
Xero call succeeds; HubSpot call returns 500 mid-run; Sheet append attempted with incomplete data
Incomplete row rolled back or flagged as incomplete; Review Narrative trigger condition not met; Distribution Agent not triggered; error logged
No complete row in Sheet; Review Narrative run log shows no execution; error log entry for HubSpot 500; Distribution Agent not triggered
T17
Review Narrative writes exception flag: human review step fires before Distribution Agent
Sheet row with one metric below threshold; human-review flag = true; test configured to simulate owner approves within 5 minutes
Automation pauses after narrative write; human review notification sent to test alias; owner approval captured; Distribution Agent fires after approval
Run log shows paused state; approval notification delivered to test alias; Distribution runs only after approval event; timestamp difference between pause and resume recorded
T18
Human review step: owner does not respond within configured timeout window (30 minutes in test)
Exception flag raised; human review notification sent; no approval action taken for 30 minutes
Automation sends a single reminder at 15 minutes; at 30-minute timeout, escalation email sent to secondary contact; run flagged as timed-out in log; Distribution Agent does not fire
Reminder email at 15 minutes confirmed; escalation email at 30 minutes confirmed; Distribution Agent not triggered; run log status = 'Timed Out — Human Review'
T19
All three agents complete successfully but Drive archive folder path does not exist
Drive archive folder deleted before run; all other sandbox conditions valid
Email and Slack post complete successfully; Drive write fails with logged error; fallback: file saved to root of service account Drive with warning note
Email and Slack confirmed; Drive file in root folder with warning filename suffix '_ARCHIVE_FOLDER_MISSING'; error log entry present
T20
Second trigger fires before first run completes (overlapping execution)
First run started manually; second cron trigger fires 2 minutes later before first run finishes
Second run queued or rejected with a 'Run already in progress' log entry; no duplicate Sheet row; no duplicate email or Slack post
Only one Sheet row for the week; one email in sent items; one Slack post; second run log entry shows 'skipped — concurrent run active'
T21
Narrative agent produces output exceeding Slides character limit for commentary placeholder (over 800 characters)
AI model returns 950-character narrative for a multi-exception week
Distribution Agent truncates narrative to 800 characters and appends '... (see full notes in Sheet)'; Slides updated; email body contains full narrative
Slides commentary placeholder contains exactly 800 characters plus ellipsis suffix; email body contains full untruncated narrative; run log records truncation event
Edge-case test table — scenarios outside normal agent flow
ID
Scenario
Input
Expected Output
Pass Criteria
E01
Duplicate week row already exists in Google Sheet before trigger fires
Sheet already contains a row for the current ISO week number (manually inserted to simulate duplicate)
Data Collection Agent detects existing row for current week; skips append; logs 'Duplicate week detected — row not overwritten'; flags for manual review
No second row for same week; error log entry with ISO week reference; run halted cleanly; no downstream agents triggered
E02
Xero returns data but HubSpot pipeline value field is null (field removed from API response)
HubSpot sandbox returns deal objects with pipeline_value field absent from response payload
Data Collection writes row with pipeline_value = NULL; null flag written to column M; Review Narrative writes narrative noting missing pipeline data; Distribution proceeds with caveat note
Sheet row pipeline_value = NULL; column M flag present; narrative references missing pipeline data; email and Slack include 'Pipeline data unavailable this week' note
E03
Google Sheets API is unavailable for 10 minutes mid-run (HTTP 503 service unavailable)
Sheets API mock returns 503 for first three calls; recovers on fourth attempt after 10 minutes
Agent retries with exponential back-off (30s, 60s, 120s); succeeds on fourth attempt; row appended; run log records retry sequence
Sheet row appended; run log shows three retry events with timestamps; total run time extended but under 15 minutes; no error raised to operator
E04
Slack workspace token revoked between Phase 1 and Phase 2 test run
Slack API returns HTTP 401 invalid_auth on post attempt
Distribution Agent logs 'Slack token invalid'; email send completes successfully; operator alert raised via run log; no retry attempted
Email delivered; Slack post absent; run log entry: 'Slack 401 invalid_auth — re-authenticate Slack integration'; operator notified
E05
Google Slides template has been replaced with a version that contains no named placeholders
Distribution Agent attempts to update a Slides deck where all placeholder strings have been removed
Agent detects zero placeholder matches; logs 'No placeholders found in Slides template'; email sends with narrative in body only and a warning that the deck was not updated; Slack post includes same warning
Slides deck unchanged; email body contains full narrative plus warning text; Slack post references deck update failure; run log records zero-placeholder event
E06
Human review step owner approves pack but then immediately revokes approval (edge case: double-click or accidental confirm)
Approval webhook fires twice within 3 seconds with the same owner identifier
Automation deduplicates approval events by webhook ID; Distribution Agent fires once only; second approval event logged as duplicate and discarded
One Distribution run; one email; one Slack post; run log shows duplicate approval event discarded with webhook ID reference
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: use an internal test email address (not the live leadership distribution list) as the Gmail recipient for all Phase 3 runs. Post to a private Slack test channel, not the live leadership channel. After the run completes, void the test record in the master Google Sheet (mark the row with 'TEST — DELETE' in column A and archive it to the test folder). Delete the test Slides deck from the production archive immediately. Do not allow a test record to remain in the live Sheet or Drive, as it will corrupt the next automated comparison.
Phase 3 uses live production credentials, live API connections to Xero and HubSpot, and the real Google Sheets master document. Every action is observed end to end and traced in the run log. Test IDs continue the sequential numbering from Phase 2.
ID
Scenario
Input
Expected Output
Pass Criteria
T22
Full production run: trigger fires, all three agents complete, pack delivered to test alias and test Slack channel
Manual trigger fired against production Xero and HubSpot; real current-week data; master Google Sheet in production; real Slides template; Gmail test alias; Slack test channel
Live data written to Sheet; narrative generated with real figures; Slides updated with production placeholders; email delivered to test alias within 4 minutes of trigger; Slack post in test channel; archive file created in production Drive test folder
All six tool interactions logged as successful; Sheet row contains real figures; email received at test alias; Slack message present; Drive archive file timestamped correctly; total run time under 5 minutes
T23
Production run with at least one metric below threshold: human review step fires correctly
Temporarily lower one target threshold in the reference tab to guarantee an exception flag on the live data; restore threshold after test
Exception flag raised; human review notification sent to test alias (Operations Manager); automation pauses; owner approves via test mechanism; Distribution Agent fires post-approval
Pause state confirmed in run log with timestamp; notification received at test alias; post-approval Distribution run starts within 60 seconds of approval; threshold restored in reference tab post-test
T24
Error log verification: confirm all run events, warnings, and decisions are written to the production run log with correct structure
Review the run log from T22 and T23 after completion
Run log contains: trigger timestamp, Xero API call result, HubSpot API call result, Sheet write confirmation, narrative generation result, exception flag value, distribution results for Gmail and Slack, archive confirmation, and total run duration
Every expected log event present; no log entries missing; timestamps sequential and consistent; no unhandled exception stack traces present in log
T25
Post-test cleanup: test record voided and removed from production Sheet and Drive
After T22 and T23, locate test row in master Sheet and test file in Drive archive folder
Test Sheet row marked 'TEST — DELETE' and moved to archive test subfolder; Drive Slides file deleted from production archive; no test artefacts remain in live data paths
Master Sheet contains no row marked as test data; Drive production archive folder contains no file from this test run; cleanup confirmed in run log
T26
Process owner review and approval: Operations Manager reviews the full production output from T22 and confirms go-live readiness
Operations Manager reviews: email received at test alias, Slack message in test channel, updated Slides deck, Sheet row data accuracy, narrative quality, and exception-flag behaviour observed in T23
Operations Manager confirms all outputs are accurate, the narrative is fit for purpose, the human review step behaves as expected, and the automation is approved to go live. Approval recorded in the FullSpec confirmation log. No separate sign-off block is required.
Process owner verbal or written confirmation received; FullSpec team records approval in the project confirmation log with the owner's name, role, and timestamp. Go-live authorised.
Once T26 is recorded as passed and the process owner approval is logged, the FullSpec team will switch the automation trigger from manual to the live Sunday 8 PM schedule, update the Gmail recipient to the real leadership distribution list, and point the Slack post to the live leadership channel. The first fully automated run will execute on the next scheduled Sunday. Contact support@gofullspec.com if any Phase 3 case fails or if the go-live date needs to be deferred.
Test and QA PlanPage 4 of 4