FS-DOC-06Quality
Test and QA Plan
Business KPI Dashboard
[YourCompany.com] · Operations Department · Prepared by FullSpec · [Today's Date]
This document defines the full test and quality assurance plan for the Business KPI Dashboard automation. It covers three sequential phases: unit testing of individual agents in sandbox, integration and edge-case testing across agent handoffs, and end-to-end production verification. The FullSpec team runs all phases and records results against the test cases listed here. The process owner joins Phase 3 to review the live output and record their approval before go-live is confirmed. No phase begins until every test case in the previous phase has reached a passing state.
01Testing approach
Testing proceeds through three phases in strict sequence. No phase begins until the previous phase fully passes. Each phase has a defined scope, environment, credential rule, pass condition tied to the test-case ID range, and a clear statement of who runs it. This structure ensures regressions are caught at the earliest possible layer and that the production environment is never touched until agent-level and integration-level behaviour is confirmed stable.
1Phase 1 — Unit Testing
Each agent tested in isolation before any handoff is wired
Scope
Individual agent logic: Data Collection Agent, KPI Commentary Agent, Distribution Agent. Each tested independently with stubbed inputs and outputs.
Environment
Sandbox only. No production credentials or live data are used at this stage.
Credential rule
Sandbox API keys for Xero and HubSpot; a dedicated test Google Sheet; a private Notion workspace; a test Slack channel. Production credentials must not be present in the environment during Phase 1.
Pass condition
All test cases T01 through T12 reach a Pass status with no open failures or skipped assertions.
Who runs it
FullSpec team. Process owner is not required to participate.
2Phase 2 — Integration and Edge Case Testing
Agent handoffs and failure scenarios validated end-to-end in sandbox
Scope
Agent-to-agent handoffs: Data Collection Agent to KPI Commentary Agent, and KPI Commentary Agent to Distribution Agent. Edge cases including duplicates, missing data, unavailable tools, and human-timeout scenarios.
Environment
Sandbox only. Responses from Xero and HubSpot APIs may be simulated via mock payloads where rate limits or sandbox restrictions apply.
Credential rule
Same sandbox credentials as Phase 1. No production credentials introduced.
Pass condition
All integration tests T13 through T19 and all edge-case tests E01 through E06 reach a Pass status. Any failure blocks progression to Phase 3.
Who runs it
FullSpec team. Process owner may observe but is not required.
3Phase 3 — End-to-End Production Test
Full live run on production credentials with process owner sign-off
Scope
Complete workflow run on live production credentials: Xero and HubSpot live APIs, the production Google Sheet, the live Notion workspace, and the designated Slack channel. Includes error-log verification and owner review.
Environment
Production. A controlled test run is executed using live credentials; the resulting record is voided or archived as a test entry immediately after verification.
Credential rule
Production API credentials active. A dedicated internal test Slack address and a clearly labelled Notion test page must be used so no test output reaches live stakeholders.
Pass condition
All test cases T20 through T24 reach a Pass status, including the final process owner review and approval recorded in the FullSpec confirmation log.
Who runs it
FullSpec team executes the run. The process owner (Ops Manager) joins to review the live output and record approval in the FullSpec confirmation. Phase 3 cannot be closed without that approval.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
Sandbox credentials only during Phase 1. Use the dedicated sandbox API keys for Xero and HubSpot, the test Google Sheet (not the production sheet), the private Notion test workspace, and the QA Slack channel. Do not load or reference production credentials at any point during this phase. Any test that touches a production endpoint is immediately invalid and must be re-run in the correct environment.
Data Collection Agent — connects to Xero and HubSpot on schedule, retrieves financial and sales metrics, and writes values to Google Sheets. Estimated build time: 10 hours. Complexity: Moderate.
ID
Scenario
Input
Expected output
Pass criteria
T01
Happy path: Xero revenue fetch
Scheduled trigger fires; valid sandbox Xero OAuth token present; reporting period set to current week.
Revenue, outstanding invoices, and cash position values returned and written to the correct Google Sheets input cells (B4, B5, B6).
All three cells populated with non-null numeric values; no API error logged; execution time under 30 seconds.
T02
Happy path: HubSpot pipeline fetch
Xero step completed; valid sandbox HubSpot API key present; date filter set to current reporting window.
Deal count, pipeline value, and conversion rate written to cells B9, B10, B11 in Google Sheets.
All three cells populated; values match the sandbox HubSpot deal fixture; no error in run log.
T03
Happy path: Google Sheets write and formula trigger
Both Xero and HubSpot values written; Sheets formulas configured for margin and growth rate in column C.
Derived KPIs (margin, growth rate) calculated automatically by Sheets formulas; no overwrite of formula cells.
Column C cells display calculated values; input cells match API-returned figures exactly; Sheets version history shows one write event.
T04
Failure: Xero OAuth token expired
Xero OAuth token expired or revoked before the trigger fires.
Automation platform logs an authentication error; run halts gracefully; no partial data written to Sheets.
Error event captured in the run log with code 401; Sheets cells remain at previous values; alert routed to FullSpec monitoring.
T05
Failure: HubSpot API returns 429 rate limit
HubSpot sandbox configured to return HTTP 429 on the pipeline endpoint.
Retry logic fires with exponential back-off (3 attempts, 10s/30s/60s intervals); if all retries fail, error logged and run halts.
Three retry attempts visible in logs; final error recorded with HTTP 429; Sheets not partially written with stale HubSpot data.
T06
Failure: Google Sheets write target cell protected or missing
Target cell range in sandbox Sheets sheet is protected by a conflicting permission rule.
Write step fails with a permission error; run halts; no silent overwrite or data corruption.
Permission error logged; cells unchanged; error message references the specific cell range that blocked the write.
KPI Commentary Agent — reads updated KPI values from Google Sheets, detects significant changes versus prior period, and writes plain-English commentary to Notion. Estimated build time: 10 hours. Complexity: Moderate.
ID
Scenario
Input
Expected output
Pass criteria
T07
Happy path: commentary drafted from updated Sheets values
Data Collection Agent confirms all cells updated; Sheets values differ from prior-period snapshot by more than 5% on at least one metric.
AI agent produces a commentary of 80 to 200 words referencing at least two KPI changes by name; commentary published to the designated Notion page.
Notion page created or updated in the correct database; word count within range; no placeholder or template text visible in output.
T08
Happy path: no significant change detected
All current-period KPI values within 2% of prior-period values; no anomaly flags.
Agent produces a brief summary noting stable performance; still publishes to Notion.
Notion page updated; commentary does not fabricate movements that did not occur; stable-state language used.
T09
Failure: Google Sheets read returns empty cells
Data Collection Agent step skipped or failed; input cells in Sheets are empty.
Commentary Agent detects empty read; halts without drafting; error logged noting missing source data.
No Notion page created or overwritten with blank content; error event logged with reference to the empty cell range.
T10
Failure: Notion API write fails (503 service unavailable)
Notion API returns HTTP 503 during the publish step.
Two retry attempts made; if both fail, commentary text is held in the automation platform's memory and an error alert is raised.
Retry attempts logged; Notion page not partially created; error alert captured; commentary text preserved in run log for manual recovery.
Distribution Agent — sends headline KPIs, the AI commentary, and the Looker Studio dashboard link to the configured Slack channel. Estimated build time: 8 hours. Complexity: Simple.
ID
Scenario
Input
Expected output
Pass criteria
T11
Happy path: Slack message delivered with all components
KPI Commentary Agent confirms Notion page published; Slack bot token valid; target channel ID configured.
Slack message posted to QA test channel containing: headline KPI figures, AI commentary excerpt, and hyperlinked Looker Studio dashboard URL.
Message visible in the test Slack channel within 60 seconds of Notion publish confirmation; all three content components present; tagged stakeholder handles resolved correctly.
T12
Failure: Slack bot token invalid or channel not found
Slack bot token revoked or target channel ID misconfigured.
Distribution step fails with a Slack API error; error logged with the channel ID and error code; no silent failure.
Error event logged with HTTP 404 or 401 from Slack API; no message posted; error visible in FullSpec monitoring dashboard.
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Integration tests verify that each agent correctly hands off to the next and that the full chain behaves as designed when run in sequence in the sandbox environment. Test IDs continue from Phase 1.
ID
Scenario
Input
Expected output
Pass criteria
T13
Handoff: Data Collection Agent success triggers KPI Commentary Agent
Data Collection Agent completes successfully; all Sheets cells written; completion event emitted.
KPI Commentary Agent starts automatically within 10 seconds of the completion event without manual intervention.
Commentary Agent run log shows trigger source as Data Collection Agent completion event; no manual start required; timestamp gap under 10 seconds.
T14
Handoff: KPI Commentary Agent success triggers Distribution Agent
KPI Commentary Agent publishes Notion page successfully; completion event emitted.
Distribution Agent starts automatically and sends the Slack message referencing the Notion page URL just published.
Distribution Agent log shows trigger source as KPI Commentary Agent completion; Notion URL in the Slack message matches the page created in the same run.
T15
Handoff blocked: Data Collection Agent fails; KPI Commentary Agent must not start
Data Collection Agent returns an error on the Xero API call; run halts.
KPI Commentary Agent does not start; no stale or partial commentary drafted.
KPI Commentary Agent shows no run initiated in the log for that cycle; error from Data Collection Agent is the only event recorded for that trigger.
T16
Handoff blocked: KPI Commentary Agent fails; Distribution Agent must not start
KPI Commentary Agent encounters a Notion write failure on the second retry.
Distribution Agent does not start; no Slack message sent with incomplete or missing commentary.
Distribution Agent log shows no run initiated; Slack channel shows no message for the affected cycle; error alert raised.
T17
Anomaly detection branch: metric out of expected range routes to Ops Manager review
One metric in Google Sheets exceeds the configured anomaly threshold (e.g. revenue drops more than 40% week-on-week in the sandbox fixture).
Workflow pauses before KPI Commentary Agent fires; an alert is sent to the Ops Manager review step; KPI Commentary Agent does not proceed until the review is resolved.
Pause event logged at the decision node; review alert delivered to the configured review channel or email; workflow status shows 'awaiting review' until manually cleared.
T18
Anomaly branch resolved: Ops Manager approves and workflow resumes
Ops Manager receives the anomaly alert and confirms the figure is correct; approval action triggered.
Workflow resumes from the decision node; KPI Commentary Agent fires with the reviewed values; full chain completes normally.
KPI Commentary Agent run log shows post-review trigger; Notion page reflects the anomalous but confirmed metric; Slack message delivered.
T19
Full sandbox chain run: all three agents in sequence
Trigger fires with valid sandbox credentials; all three agents configured and healthy.
Complete end-to-end chain completes in sandbox: Sheets written, Notion page published, Slack message sent. Total elapsed time under 5 minutes.
All three agent run logs show success; Sheets, Notion, and Slack test channel all updated correctly; elapsed time recorded and within target.
Edge case tests cover conditions that fall outside the normal happy path and are most likely to cause silent errors or data quality issues in production. These are tested in sandbox with simulated or fixture-based inputs.
ID
Scenario
Input
Expected output
Pass criteria
E01
Duplicate trigger: schedule fires twice in the same reporting window
Trigger misfires and executes two runs within the same hour.
Second run is detected as a duplicate by a run-guard check on the Sheets timestamp; second run aborts without overwriting the first run's data.
Sheets cells show values from the first run only; second run log shows 'duplicate run aborted'; no double Notion page or double Slack message.
E02
Missing data: Xero returns zero revenue for the period
Xero sandbox returns a valid but zero-value response for total revenue (legitimate zero, not a fetch error).
Zero is written to the Sheets cell; anomaly threshold check flags the drop for review; workflow pauses for Ops Manager confirmation.
Cell B4 contains 0.00; anomaly flag triggered; workflow does not silently proceed with a zero-revenue report to Slack without human review.
E03
Missing data: HubSpot returns no deals for the period
HubSpot sandbox returns an empty deals array for the reporting window (no closed deals that week).
Empty array handled gracefully; deal count written as 0; pipeline value and conversion rate written as 0; no null reference error.
Sheets cells B9, B10, B11 contain 0 values; KPI Commentary Agent notes the absence of deals in the commentary without crashing; no unhandled null in logs.
E04
Unavailable tool: Notion API is unreachable for the full run window
Notion API returns errors for more than 10 minutes (simulated outage beyond retry window).
KPI Commentary Agent retries twice then halts; commentary text is preserved in the run log; FullSpec monitoring alert raised; Distribution Agent does not fire.
Error alert generated; commentary text recoverable from run log; no Slack message sent without Notion confirmation; incident logged in FullSpec monitoring.
E05
Human timeout: Ops Manager does not respond to anomaly review within 2 hours
Anomaly is flagged; Ops Manager review step receives no response for 120 minutes.
Escalation reminder sent to the Ops Manager (via Slack direct message or email); workflow remains paused; no auto-progression past the review gate.
Escalation message sent at the 120-minute mark; workflow log shows 'escalation sent, awaiting review'; no downstream agents triggered.
E06
Stale prior-period data: first two cycles have no baseline for week-on-week comparison
KPI Commentary Agent runs for the first time with no prior-period snapshot available in Sheets.
Agent detects absence of prior-period data; drafts commentary noting it is the first tracked cycle; does not fabricate percentage changes.
Commentary text references 'initial baseline cycle'; no calculated percentage movements appear in the output; Notion page published without errors.
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: Use a dedicated internal test address for all Slack notifications during this run. The target channel must be a private QA channel, not the live team channel. Tag only FullSpec team members and the process owner, not the full stakeholder list. The Notion page created during this run must be clearly labelled 'TEST RUN — DO NOT DISTRIBUTE' and must be deleted or moved to a test archive immediately after verification is complete. Any Xero or HubSpot data written to Google Sheets from this run should be cleared from the input cells and replaced with the standard prior-period values before the first scheduled live run. The process owner must confirm this cleanup has been completed before sign-off is recorded.
Phase 3 uses live production credentials across all connected tools. The FullSpec team executes two complete workflow runs to confirm reliability under real conditions. The process owner reviews the output of the second run and records their approval in the FullSpec confirmation log. Test IDs continue from Phase 2.
ID
Scenario
Input
Expected output
Pass criteria
T20
Production run 1: full chain on live credentials
Trigger fired manually by FullSpec team at agreed test time; live Xero and HubSpot OAuth tokens active; production Sheets, Notion workspace, and internal QA Slack channel configured.
Data Collection Agent fetches live financial and pipeline data; Sheets cells populated with real current-period values; KPI Commentary Agent drafts commentary; Notion page created (labelled as test); Slack message posted to QA channel with dashboard link.
All three agents complete without error in the production environment; total elapsed time under 5 minutes; all output channels (Sheets, Notion, Slack) show the correct content from run 1.
T21
Production run 2: repeated run to confirm stability
Trigger fired again 30 minutes after run 1; prior-period snapshot from run 1 available for comparison.
Full chain completes again; KPI Commentary Agent performs week-on-week comparison against the run 1 snapshot; no duplicate Notion pages or Slack messages from run 1 interfere.
Second run completes cleanly; Notion page for run 2 is distinct from run 1 entry; Slack message references run 2 data; run-guard logic prevents double-posting.
T22
Error log verification: confirm monitoring captures any warnings from production runs
Review the FullSpec monitoring dashboard after both production runs complete.
All events from T20 and T21 appear in the monitoring log with correct timestamps, agent names, and success or warning status. Any warnings (e.g. slow API response) are documented.
Monitoring log is complete for both runs; no unlogged events; any warnings have an associated description and do not indicate a structural failure.
T23
Post-run cleanup: test records voided and environment reset
Both production test runs complete; FullSpec team and process owner confirm outputs have been reviewed.
Test Notion pages deleted or moved to the test archive; Google Sheets input cells cleared and reset to prior-period values; QA Slack channel messages archived; no test artefacts remain in live channels.
Production Sheets reflects pre-test state; no test-labelled Notion pages visible in the live reporting database; QA Slack channel confirmed clean; FullSpec team records cleanup completion in the run log.
T24
Process owner review and approval
Process owner (Ops Manager) reviews the Slack message, Notion page, and Google Sheets output from production run 2. FullSpec team presents the monitoring log and confirms all prior test phases passed.
Process owner confirms: data is accurate, commentary is appropriate, dashboard link resolves correctly, and the Slack message format meets expectations. Approval recorded in the FullSpec confirmation log. Go-live authorised.
FullSpec confirmation log entry shows process owner name, review date, and explicit approval status. No outstanding failures across T01 through T24 or E01 through E06. Go-live date confirmed and scheduled automated trigger set to the agreed production schedule (Monday 7am weekly).
Once T24 is marked Pass and the FullSpec confirmation log is updated, the automation is considered live. The scheduled weekly trigger will fire automatically at the agreed time. Any issues after go-live should be reported to support@gofullspec.com with the run ID from the monitoring log.
Test and QA PlanPage 4 of 4