FS-DOC-06Quality
Test and QA Plan
Fulfilment Performance Reporting
[YourCompany.com] · Fulfilment Department · Prepared by FullSpec · [Today's Date]
This document defines the full quality assurance process for the Fulfilment Performance Reporting automation. It is written for the FullSpec build team and covers every test case that must pass before the system goes live. Testing is structured across three sequential phases: unit tests against individual agents, integration and edge-case tests across agent handoffs, and a final end-to-end production run. No phase begins until the previous phase has fully passed. The process owner joins Phase 3 to review and confirm sign-off.
01Testing approach
All testing follows three phases in sequence. No phase begins until every test case in the previous phase carries a confirmed pass status. Phase 1 isolates each agent in a sandbox environment. Phase 2 verifies cross-agent handoffs and exercises edge cases. Phase 3 runs a full end-to-end cycle in production using a controlled test record. The FullSpec team owns all three phases; the process owner participates in Phase 3 only.
1Phase 1: Unit Testing
Each agent tested in isolation before any handoff is exercised
Scope
Data Collection Agent, KPI and Commentary Agent, Report Distribution Agent — each tested independently with mocked inputs where required
Environment
Sandbox only. Shopify, ShipStation, and Xero sandbox/test accounts. Google Sheets test workbook. Slack test channel. Gmail test address.
Credential rule
Sandbox credentials exclusively. No production API keys, no live Slack channels, no real recipient email addresses at any point in this phase.
Pass condition
All cases T01 through T12 return the expected output with no errors. Any failure blocks progression to Phase 2.
Who runs it
FullSpec team only.
2Phase 2: Integration and Edge Case Testing
Agent handoffs and failure scenarios validated across the full data pipeline
Scope
End-to-end handoffs between all three agents; duplicate records, missing data, unavailable tools, and human review timeout scenarios
Environment
Sandbox for all API calls. Google Sheets test workbook carries forward from Phase 1. Slack test channel and Gmail test address remain in use.
Credential rule
Sandbox credentials only. No production keys or live recipient addresses introduced at this stage.
Pass condition
All integration cases T13 through T18 and all edge cases E01 through E06 pass. Any failure blocks progression to Phase 3.
Who runs it
FullSpec team only.
3Phase 3: End-to-End Production Test
One full run in the live environment with a controlled test record before go-live
Scope
Single full production run traced from scheduled trigger through report delivery and log entry; error log verification; process owner review and sign-off
Environment
Production APIs for Shopify, ShipStation, and Xero. Live Google Sheets workbook. Live Slack channel. Live Gmail, but using an internal test address for recipients.
Credential rule
Production API credentials are active. The test record created during this phase must be voided or deleted immediately after the phase concludes. No real customer-facing distribution occurs.
Pass condition
All cases T19 through T22 pass, including the final process owner review case. Sign-off is recorded in the FullSpec confirmation system.
Who runs it
FullSpec team executes T19 through T21. Process owner participates in T22 for review and approval.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
Sandbox credentials only. All Phase 1 tests must be run against sandbox or test accounts for Shopify, ShipStation, and Xero. Do not connect production API keys, write to live Google Sheets workbooks, post to live Slack channels, or send email to real recipients during this phase. Any test run that accidentally touches a production system must be flagged to the FullSpec team lead immediately.
Data Collection Agent — connects to Shopify, ShipStation, and Xero on the configured schedule and writes aligned data to the Google Sheets test workbook. Estimated build complexity: Moderate.
ID
Scenario
Input
Expected Output
Pass Criteria
T01
Happy path: scheduled trigger fires, all three APIs respond
Scheduled trigger fires at configured time; sandbox credentials valid for all three sources
Google Sheets test workbook populated with order, shipment, and returns rows for the closed period; row count matches sandbox data set
Sheet contains correct row count; all required columns present; no empty mandatory fields; run completes within 90 seconds
T02
Shopify API returns orders for the correct date window only
Shopify sandbox contains orders spanning two reporting periods
Only orders within the configured period date range are written to the sheet
Row timestamps all fall within the period boundary; no out-of-range rows present
T03
ShipStation API returns shipment records aligned to the same period
ShipStation sandbox contains dispatch and delivery records for the period
Shipment rows written to the sheet with carrier, dispatch date, and delivery status populated
All shipment rows have non-null carrier and dispatch date values; delivery status is one of the accepted enum values
T04
Xero API returns credit notes and returns for the period
Xero sandbox contains three credit notes within the period
Returns rows written to the sheet with credit note ID, value, and date
Exactly three returns rows present; values match sandbox credit note amounts
T05
Shopify API returns a 401 Unauthorized error
Shopify sandbox credential revoked before trigger fires
Agent logs the authentication error and halts; no partial data written to the sheet; alert raised
Error entry present in the run log with status 401 and source label 'Shopify'; sheet remains empty for this run
T06
ShipStation API returns a 429 rate-limit response
Request rate exceeds ShipStation sandbox limit of 40 requests/minute
Agent applies exponential backoff and retries up to three times; on third failure it logs the error and halts
Retry attempts logged with timestamps showing increasing back-off intervals; final log entry records failure with status 429
T07
Xero API returns zero results (no returns in period)
Xero sandbox has no credit notes within the reporting period date range
Agent writes zero returns rows; a zero-value placeholder row is written to ensure KPI formulas do not break; run continues
Sheet contains the placeholder returns row; subsequent KPI calculation receives a valid zero-value input
KPI and Commentary Agent — reads the populated Google Sheets test workbook, calculates on-time dispatch rate, fill rate, average dispatch lead time, and returns rate, then drafts a plain-English variance commentary. Estimated build complexity: Moderate.
ID
Scenario
Input
Expected Output
Pass Criteria
T08
Happy path: all KPIs calculated from a fully populated sheet
Sheet populated with 50 order rows, 48 shipment rows, 2 returns rows from T01
KPI block written: on-time dispatch rate, fill rate, average lead time, returns rate; commentary paragraph drafted referencing all four metrics
All four KPI fields populated with numeric values; commentary is non-empty; no formula errors present in the sheet
T09
KPI breach detected: on-time dispatch rate below 90% threshold
Sheet data produces an on-time dispatch rate of 82%
Commentary flags the breach, names the metric, states the threshold, and includes the actual value
Commentary contains the phrase 'on-time dispatch' and a value below the 90% threshold; breach flag field set to true in the output schema
T10
All KPIs within target: commentary reflects no breaches
Sheet data produces all metrics within configured thresholds
Commentary acknowledges strong performance and notes no exceptions; no breach flags set
Breach flag fields all set to false; commentary does not contain exception language
T11
Sheet contains a column that is empty due to a missing Xero row
Returns rate formula references an empty returns column
Agent detects the missing column, logs a data-quality warning, and substitutes zero for the returns value; commentary notes the data gap
Run does not crash; log entry records the data quality warning; returns rate in output shows 0%; commentary references the gap
T12
Commentary agent receives an empty sheet (Data Collection Agent failed upstream)
Sheet contains no data rows
Agent detects the empty sheet, logs an error, and does not attempt KPI calculation or commentary; alert raised for the FullSpec team
Agent halts without writing KPI values; log entry records 'empty sheet' error; no commentary output produced
Report Distribution Agent — posts the KPI summary to the Slack test channel, sends the report link and PDF export to the test Gmail address, and logs a new row in the historical performance Google Sheet. Estimated build complexity: Simple.
ID
Scenario
Input
Expected Output
Pass Criteria
T13
Happy path: Slack post and Gmail send both succeed; log row written
Approved commentary and KPI block passed from KPI and Commentary Agent
Slack test channel receives formatted KPI summary message; test Gmail address receives email with report link and PDF attachment; new row appended to historical log sheet
Slack message delivered within 30 seconds; Gmail delivery confirmed via send receipt; log sheet row contains correct period date and all KPI values
T14
Slack API unavailable (connection timeout)
Slack test workspace returns a 503 timeout on the post request
Agent logs the Slack failure, retries once after 60 seconds, and on second failure logs the error and continues to Gmail send
Retry attempt logged; Slack failure recorded; Gmail send still executed; log row still written
T15
Gmail SMTP authentication fails
Gmail OAuth token expired before the send step
Agent logs the Gmail authentication error, skips the send, and writes the failure to the run log; Slack post already delivered is not reversed
Run log records Gmail failure with OAuth error detail; Slack post confirmed delivered; log row written with a 'gmail-failed' status flag
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Integration tests verify that the three agents hand off to each other correctly under normal and partial-failure conditions. Edge case tests exercise the failure modes most likely to occur in production: duplicate records, missing data, tool unavailability, and the human review timeout path.
All Phase 2 tests continue to use sandbox credentials. Production API keys must not be introduced until Phase 3. The Google Sheets test workbook from Phase 1 carries forward; reset it to a known clean state before each Phase 2 run.
ID
Scenario
Input
Expected Output
Pass Criteria
T16
Full agent chain: Data Collection passes to KPI and Commentary, which passes to Distribution
Scheduled trigger fires; all three sandbox APIs healthy; commentary approved at the review gate
Data Collection Agent completes and fires the trigger for KPI and Commentary Agent; KPI and Commentary Agent completes and fires the trigger for Report Distribution Agent; distribution completes and log row is written
Three sequential trigger events logged with correct agent-source labels; final run status is 'complete'; no manual intervention required
T17
Data Collection Agent fails mid-run: Xero returns 500; KPI and Commentary Agent must not fire
Shopify and ShipStation succeed; Xero sandbox returns HTTP 500
Data Collection Agent marks the run as failed after the Xero error; the trigger for KPI and Commentary Agent is not fired; alert is raised
KPI and Commentary Agent execution log shows no run initiated for this cycle; alert record contains the Xero 500 error detail
T18
Commentary review gate: ops manager does not approve within the configured timeout window
KPI and Commentary Agent outputs draft commentary; review gate opens; no approval action taken within 30 minutes
Automation platform sends a reminder notification to the configured reviewer after 15 minutes; at 30-minute timeout the run is paused and a timeout alert is logged
Reminder notification delivered at the 15-minute mark; timeout alert logged at exactly the 30-minute mark; Report Distribution Agent not triggered
T19
Commentary review gate: ops manager edits and approves the draft
Draft commentary contains a factual error; reviewer edits the text and submits approval
Edited commentary text replaces the original draft in the output; Report Distribution Agent fires with the updated text
Distribution Slack post and Gmail email both contain the reviewer-edited commentary text, not the original AI draft
Edge case tests below use the ID prefix E to distinguish them from the main T sequence. These cases focus on real-world data quality problems and tool availability issues that fall outside the primary happy path.
ID
Scenario
Input
Expected Output
Pass Criteria
E01
Duplicate order records returned by Shopify API (same order ID in two pages of results)
Shopify sandbox returns order ID ORD-1042 twice across paginated API responses
Data Collection Agent deduplicates on order ID before writing to the sheet; only one row for ORD-1042 appears in the output
Sheet row count matches the unique order ID count; no duplicate rows; deduplication event logged
E02
ShipStation shipment record missing dispatch date field
ShipStation sandbox returns a shipment record with a null dispatch_date value
Agent writes the row with a null dispatch_date; the null cell is flagged in the data quality log; KPI calculation excludes the row from on-time dispatch rate calculation and notes the exclusion
Null dispatch_date row present in the sheet; data quality log entry references the shipment ID; KPI commentary notes that one record was excluded from on-time calculation
E03
Google Sheets API quota exhausted mid-write
Sheet write operation hits the Google Sheets API write limit (300 write requests/minute) before all rows are written
Agent pauses writes, waits 60 seconds, then resumes from the last successfully written row; full data set eventually written; no rows duplicated
Final sheet row count matches expected total; no duplicate rows; quota-pause event logged with resume timestamp
E04
Xero API returns a credit note that falls outside the reporting period
Xero sandbox contains one credit note dated one day before the period start date
Agent filters the out-of-period record and does not write it to the sheet; filter event is logged with the credit note ID and its actual date
Sheet contains no rows with dates outside the reporting window; log entry records the filtered credit note ID
E05
All three source APIs unavailable simultaneously (network outage scenario)
Trigger fires; all three API endpoints return connection timeout
Agent logs three separate timeout errors, marks the run as failed, and sends an alert to the configured operations notification address; no partial data written
Run log shows three timeout entries; alert delivered within 5 minutes of the trigger fire time; sheet remains unchanged
E06
Human review timeout followed by an out-of-hours approval (late approval scenario)
Commentary review gate times out after 30 minutes; reviewer approves the draft 2 hours later
The late approval is accepted; the automation resumes from the paused state and triggers the Report Distribution Agent with the approved commentary
Report Distribution Agent fires after the late approval; distribution completes normally; run log records the approval timestamp and the total elapsed time
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: use an internal test email address (such as a FullSpec team inbox or a dedicated QA alias at the client domain) as the sole recipient for all Gmail and Slack outputs during this phase. Do not configure real leadership or external recipients. After Phase 3 concludes, the test order records written to the production Google Sheets workbook and the historical log must be voided or deleted immediately. Confirm deletion before marking Phase 3 complete.
Phase 3 runs one complete cycle against production APIs using live credentials. The trigger fires at the configured production schedule time (Monday 7 a.m.). All agents execute in sequence. The process owner reviews the commentary output and records approval. The FullSpec team traces every step in the run log to confirm end-to-end correctness before go-live is authorised.
ID
Scenario
Input
Expected Output
Pass Criteria
T20
Full production run: trigger fires, all three agents complete, distribution delivered to internal test address
Production Shopify, ShipStation, and Xero APIs; live Google Sheets workbook; Slack test channel; internal test Gmail address
Google Sheets workbook populated with live order, shipment, and returns data for the closed period; KPIs calculated; commentary drafted; Slack post delivered to test channel; Gmail delivered to internal test address; historical log row appended
All five output conditions confirmed: sheet populated, KPIs present, commentary non-empty, Slack message received, Gmail received, log row written. Total run time under 3 minutes.
T21
Error log verification: run log reviewed for unexpected warnings or silent failures
Run log from T20 production execution
Run log contains only expected entries: trigger fired, three API calls succeeded, sheet write confirmed, KPI calculation confirmed, commentary generated, Slack delivered, Gmail delivered, log row written. No warnings or error entries present.
Zero error or warning entries in the run log for the T20 execution. Each expected log entry is present with a valid timestamp. FullSpec team lead signs off on the log review before T22 proceeds.
T22
Process owner review and approval: owner reviews the production commentary output and confirms the automation is fit for go-live
Commentary and KPI block produced during T20, presented to the process owner for review
Process owner reviews the commentary and KPI figures, confirms accuracy against known period data, and records approval in the FullSpec confirmation system. Any requested edits are made and the final approved output is confirmed.
Process owner approval recorded in the FullSpec confirmation system with the reviewer name, timestamp, and approval status set to 'Approved for go-live'. This is the final gate before the automation is switched to its live schedule. No separate sign-off document is required.
After T22 approval is recorded, the FullSpec team will void all test records written to the production Google Sheets workbook and the historical log sheet, confirm the internal test address is removed from the recipient configuration, and replace it with the agreed live recipient list. The automation is then activated on its live Monday 7 a.m. schedule. Any issues after go-live should be reported to support@gofullspec.com.
Test and QA PlanPage 4 of 4