FS-DOC-06Quality
Test and QA Plan
Support Reporting and Insights
[YourCompany.com] · Customer Support Department · Prepared by FullSpec · [Today's Date]
This document defines the complete test and quality assurance plan for the Support Reporting and Insights automation. It is written for the FullSpec team responsible for building and validating the three-agent pipeline: Data Fetch Agent, Reporting Agent, and Distribution Agent. Testing runs in three sequential phases. Unit tests verify each agent in isolation; integration and edge-case tests confirm that agents hand off correctly and handle abnormal inputs safely; end-to-end production tests confirm the live system delivers accurate, timely reports across all three cadences. No phase begins until every test case in the preceding phase has passed.
01Testing approach
Testing runs in three sequential phases. Each phase must reach a full pass on every test case before the next phase opens. Phase 1 uses only sandbox credentials and synthetic data. Phase 2 introduces agent handoffs and deliberate fault injection. Phase 3 uses live production credentials with controlled test records that are voided after the run. The process owner joins the FullSpec team in Phase 3 to confirm the output matches expectations and to record sign-off.
1Phase 1: Unit Testing
Each agent tested in isolation against sandbox data
Scope
Individual agent logic: Data Fetch Agent, Reporting Agent, and Distribution Agent tested separately with no inter-agent dependencies.
Environment
Sandbox only. Zendesk sandbox, a test Google Sheet, a Looker Studio preview source, and test Gmail, Slack, and Notion workspaces.
Credential rule
No production API tokens or real account credentials may be used in Phase 1. All tokens must point to sandbox or development environments.
Pass condition
All cases T01 through T12 return the expected output and no case is flagged as a partial pass. Failures must be resolved and re-run before Phase 2 opens.
Who runs it
FullSpec team only. Process owner is not required during Phase 1.
2Phase 2: Integration and Edge-Case Testing
Agent handoffs and fault-injection scenarios across the full pipeline
Scope
Agent-to-agent handoffs, conditional routing (monthly deep-dive branch), and deliberate edge-case inputs including duplicates, missing fields, unavailable tools, and human timeout.
Environment
Sandbox for edge-case and fault injection tests. A staging environment that mirrors production credentials is used for handoff timing tests (T13 through T19).
Credential rule
Staging credentials may mirror production scopes but must not point to live distribution lists, live Slack channels, or the active Notion reporting log.
Pass condition
All integration cases T13 through T19 pass and all edge cases E01 through E06 produce the documented safe-fail or retry behaviour with no unhandled exceptions.
Who runs it
FullSpec team only. Process owner is not required during Phase 2.
3Phase 3: End-to-End Production Testing
Full pipeline run on live credentials with controlled test records
Scope
A complete production run for each cadence (daily, weekly, and monthly), error-log verification, and final process-owner review and approval.
Environment
Live production credentials. Test records are introduced into Zendesk using an internal test address and are voided or archived after the run. Distribution is directed to an internal test distribution list, not real stakeholders.
Credential rule
Live API tokens used. The Zendesk token must have read scope only during Phase 3. Gmail, Slack, and Notion credentials must point to test-only recipients and channels until T20 through T24 are fully passed.
Pass condition
All cases T20 through T24 pass, including the final process-owner review case. Sign-off is recorded in the FullSpec confirmation system. No separate signature block is required.
Who runs it
FullSpec team runs T20 through T23. The process owner (Support Manager) joins for T24 to review output and record approval.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
Sandbox credentials only: every test in Phase 1 must use the Zendesk sandbox environment, a dedicated test Google Sheet, and test accounts for Gmail, Slack, and Notion. No production API token may be present in the environment during Phase 1. If a production token is accidentally used, the run is invalidated and Phase 1 must be restarted from T01.
Data Fetch Agent — estimated build time: 8 hours, complexity: Moderate
ID
Scenario
Input
Expected Output
Pass Criteria
T01
Happy path: scheduled trigger fires on time
Cron schedule fires at configured daily time (07:00); Zendesk sandbox contains 45 tickets in the target date range.
All 45 ticket records written to the raw-data tab of the test Google Sheet within 90 seconds of trigger.
Row count in Sheet equals 45; no blank rows; timestamp column populated for all rows.
T02
Happy path: weekly cadence with larger dataset
Weekly trigger fires; Zendesk sandbox contains 280 tickets across 7 days.
All 280 records written correctly; existing tab data from prior week is replaced, not appended.
Row count equals 280; prior-period rows absent; sheet write completes within 3 minutes.
T03
Zendesk API returns 401 Unauthorized
Trigger fires but API token is expired or invalid.
Agent logs a credential error, halts execution, and sends an internal alert to the configured error notification channel.
No partial data written to Sheet; error event present in the automation platform run log; alert delivered within 2 minutes.
T04
Zendesk API rate limit hit mid-fetch
429 Too Many Requests returned after page 3 of a paginated response (simulated via sandbox throttle).
Agent pauses, retries with exponential backoff (30 s, 60 s, 120 s), and resumes pagination from the last successful page.
Final Sheet row count matches expected total; no duplicate rows; retry events logged.
T05
Google Sheets write fails (quota exceeded)
Sheets API returns 429 or 503 during the write step.
Agent retries up to 3 times at 30-second intervals; if all retries fail, logs the error and halts with an alert.
Error entry in run log; no silent failure; alerting channel notified.
Reporting Agent — estimated build time: 12 hours, complexity: Complex
ID
Scenario
Input
Expected Output
Pass Criteria
T06
Happy path: metric calculation from clean dataset
280-row test Sheet with pre-validated values for resolution time, CSAT, and ticket category.
Metrics summary tab populated: first response time, full resolution time, CSAT score, ticket volume by category, and SLA breach count all calculated correctly.
Each metric value matches the hand-calculated reference figure within 0.1% tolerance; summary tab timestamp updated.
T07
Happy path: AI commentary drafted for weekly report
Metrics summary tab populated as per T06; prior-period benchmark values present.
Two-to-three sentence plain-English commentary noting the largest change from prior period, any CSAT drop, and backlog movement written to the commentary cell.
Commentary cell is non-empty; references at least one specific metric value from the summary tab; no placeholder text present.
T08
Monthly deep-dive branch: human review flag set
Report cadence parameter set to 'monthly'; Reporting Agent completes metric calculation and commentary draft.
Workflow pauses at the human-review decision node; Support Manager receives a review notification; report is not distributed until approval is recorded.
Distribution Agent does not fire; review notification delivered to the correct channel; workflow status shows 'awaiting approval'.
T09
CSAT column missing from Sheet (header renamed)
Raw-data tab has CSAT column renamed to 'Customer Score' due to a header change in a hypothetical Zendesk export.
Agent detects the missing expected header, logs a schema mismatch warning, and halts metric calculation with an alert rather than producing a zero or null CSAT figure.
Run log contains schema mismatch event; metrics summary tab not partially written; alert delivered.
T10
Zero tickets in date range
Zendesk returns an empty result set (bank holiday or no tickets created).
Agent writes a zero-row placeholder to the Sheet and generates a commentary note stating no tickets were recorded for the period; does not error.
Metrics summary shows zeros, not nulls or errors; commentary cell populated with a no-data statement; Distribution Agent fires normally.
Distribution Agent — estimated build time: 8 hours, complexity: Moderate
ID
Scenario
Input
Expected Output
Pass Criteria
T11
Happy path: all three channels delivered successfully
Completed metrics summary and commentary; Looker Studio link generated; Gmail, Slack, and Notion credentials valid.
Email sent to test distribution list with correct subject line, metric callouts, and Looker Studio link; Slack message posted to test channel; Notion row appended with correct period label and headline figures.
Email delivery confirmed in Gmail sent folder; Slack message visible in test channel within 60 seconds; Notion database contains new row with correct date and metric values.
T12
Slack API unavailable (503 returned)
Gmail and Notion steps complete successfully; Slack API returns 503.
Agent retries Slack delivery twice at 60-second intervals; if all retries fail, logs the Slack failure, marks the Slack step as failed in the run log, and continues to complete the Notion step. Email delivery is not blocked.
Email delivered; Notion row appended; Slack failure recorded in run log; no unhandled exception; alerting channel notified of partial delivery.
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Integration tests confirm that each agent handoff transfers the correct payload and that the conditional monthly-review branch routes correctly. IDs continue from the Phase 1 sequence.
ID
Scenario
Input
Expected Output
Pass Criteria
T13
Data Fetch Agent to Reporting Agent handoff: payload completeness
Data Fetch Agent completes a full write of 280 rows to the test Sheet.
Reporting Agent triggers automatically within 30 seconds of the Sheet write confirmation event; reads the correct tab; does not process stale data from a prior run.
Reporting Agent run log shows trigger timestamp within 30 seconds of Data Fetch completion; row count read by Reporting Agent equals 280.
T14
Reporting Agent to Distribution Agent handoff: commentary and Looker Studio link passed
Reporting Agent completes metric calculation and commentary for a weekly report.
Distribution Agent receives the metrics summary object and the Looker Studio shareable link; email body and Slack message both include the link.
Looker Studio URL present in email body and Slack message text; commentary text in email matches the value written to the Sheet commentary cell.
T15
Monthly branch: human approval triggers Distribution Agent
Monthly cadence; Support Manager records approval in the review notification.
Distribution Agent fires within 60 seconds of approval being recorded; distributes the approved commentary, not a cached or prior draft.
Distribution timestamp is after approval timestamp; commentary in email matches the approved version.
T16
Monthly branch: human approval not received within 24 hours
Monthly cadence; review notification sent; no approval recorded after 24 hours.
Automation platform sends a follow-up reminder to the Support Manager; workflow remains paused; report is not distributed.
Reminder notification delivered at the 24-hour mark; Distribution Agent has not fired; workflow status still shows 'awaiting approval'.
T17
Daily, weekly, and monthly cadences do not conflict
All three schedule triggers configured simultaneously in the automation platform; daily and weekly triggers fire on the same day as the monthly trigger.
Each cadence runs its own isolated pipeline instance; no shared state between runs; all three complete without interference.
Three separate run instances visible in the platform run log; no metric values from one cadence overwriting another; all three Notion rows appended with the correct cadence label.
T18
Looker Studio dashboard reflects refreshed Sheet data
Data Fetch Agent writes a new dataset that changes the CSAT average by more than 5 percentage points from the prior period.
Looker Studio dashboard connected to the Sheet updates to show the new CSAT figure without manual re-authorisation or cache clearing.
Looker Studio shareable link, when opened within 5 minutes of the Sheet write, displays the updated CSAT value.
T19
Notion log append does not overwrite existing rows
Notion database already contains 12 prior reporting entries.
Distribution Agent appends a new row without modifying or deleting any existing rows.
Notion database row count increases by exactly 1; all 12 prior rows intact; new row contains correct period label, date, and headline metric values.
The following edge-case tests use the E-series ID sequence and cover abnormal inputs, duplicate data, tool unavailability, and human-step timeout scenarios.
ID
Scenario
Input
Expected Output
Pass Criteria
E01
Duplicate ticket records in Zendesk export
Zendesk API returns a paginated response where 15 tickets appear on both page 3 and page 4 due to a cursor pagination edge case.
Data Fetch Agent deduplicates records by ticket ID before writing to the Sheet; Sheet contains no duplicate rows.
Sheet row count matches the unique ticket count; no duplicate ticket IDs present in the raw-data tab.
E02
Missing CSAT data for all tickets in the period
All tickets in the date range have no CSAT response recorded (e.g. surveys not sent or all unanswered).
Reporting Agent writes 'N/A' to the CSAT metric cell and notes the absence in the commentary; does not calculate a zero score or leave the cell blank.
CSAT metric cell contains 'N/A'; commentary references the missing CSAT data; no formula error in the Sheet.
E03
Zendesk API completely unavailable at scheduled trigger time
Zendesk returns a 504 Gateway Timeout on the first request and all three retry attempts.
Entire pipeline halts after the third retry; an error alert is sent to the configured internal notification channel; no partial data is written; the next scheduled run attempts a fresh fetch.
Run log shows three retry attempts with timestamps; error alert delivered; Sheet raw-data tab unchanged from prior run.
E04
Gmail credential token expired at distribution time
Gmail OAuth token has expired between build and the first live run.
Distribution Agent detects the auth failure on the Gmail step, logs a credential expiry error, and sends an alert via Slack (if Slack is available) or to the Notion log; does not silently skip email delivery.
Email not sent; Slack alert or Notion error entry created; run log contains Gmail OAuth failure event.
E05
Reporting Agent generates commentary with a hallucinated metric value
Metrics summary tab contains a CSAT of 87%; AI commentary draft states 'CSAT improved to 92%'.
A validation step compares the commentary's cited metric values against the metrics summary tab; if a discrepancy greater than 1% is detected, the commentary is flagged for human review before distribution, regardless of cadence.
Commentary flagged in the run log; distribution paused; review notification sent to Support Manager; metric discrepancy documented in the alert.
E06
Human review timeout for monthly report (Support Manager unavailable)
Monthly deep-dive review notification sent; no approval or rejection recorded within 48 hours.
After 48 hours the automation sends a second escalation notification to the backup reviewer defined in the configuration; workflow remains paused until approval is received.
Escalation notification delivered at the 48-hour mark to the backup contact; Distribution Agent has not fired; workflow status shows 'escalated'.
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: all test records introduced into Zendesk must use an internal test address (e.g. qa-test@[YourCompany.com]) and must be tagged with 'qa-test-run' so they can be identified and voided after the run. Distribution during Phase 3 must be directed to an internal test distribution list only. Do not use real stakeholder email addresses, the live Slack support channel, or the active Notion reporting log until T24 is passed and the process owner has recorded approval. Any Phase 3 run that sends to real recipients before T24 is complete is considered a test failure and must be documented in the run log.
ID
Scenario
Input
Expected Output
Pass Criteria
T20
Full production run: daily cadence end-to-end
Daily trigger fires at 07:00 in the live automation platform; Zendesk production environment contains real tickets from the prior day; Gmail, Slack, and Notion credentials are live and pointing to test recipients.
Data Fetch Agent pulls yesterday's tickets, writes to the production Sheet; Reporting Agent calculates all metrics and drafts commentary; Distribution Agent sends the daily summary email to the test list, posts to the test Slack channel, and appends a Notion row. Total elapsed time from trigger to Notion append is under 5 minutes.
Email received by test address with correct date label and at least one non-zero metric value; Slack message posted; Notion row appended; elapsed time confirmed in run log as under 5 minutes; all three qa-test-run tickets voided in Zendesk after confirmation.
T21
Full production run: weekly cadence end-to-end
Weekly trigger fires; production Zendesk contains the prior week's tickets; metrics for first response time, resolution time, CSAT, volume by category, and SLA breach count all calculable.
All five core metrics populated in the production metrics summary tab; Looker Studio dashboard reflects the updated values; email contains the Looker Studio link; commentary references at least one metric change versus the prior week.
All five metric cells non-null and non-zero (or explicitly N/A where no data exists); Looker Studio link in email opens the updated dashboard; commentary text mentions the prior-period comparison; run completes within 5 minutes.
T22
Full production run: monthly deep-dive with human review
Monthly trigger fires; Reporting Agent completes metric calculation and drafts extended commentary; monthly-review branch routes to the human-review step.
Support Manager receives the review notification on the test channel; reviews the draft commentary; records approval; Distribution Agent fires and sends the monthly report to the test distribution list.
Review notification received by Support Manager within 2 minutes of Reporting Agent completion; approval recorded; Distribution Agent fires within 60 seconds of approval; email subject line contains 'Monthly' and the correct period label.
T23
Error log verification: confirm all failure events are captured
Simulate a Zendesk API 503 error by temporarily revoking the API token mid-run in the test environment, then restoring it and triggering a retry.
Run log in the automation platform contains: the initial failure event with timestamp, each retry attempt with timestamp and HTTP status code, and the final success event after the token is restored.
At least three log entries present for the failure sequence; final success entry confirms row count written to Sheet; no uncaught exceptions in the platform error console.
T24
Process owner review and final approval
Support Manager (process owner) is presented with the outputs from T20, T21, and T22: the daily email, the weekly email with Looker Studio link, and the monthly email with approved commentary.
Support Manager confirms that metric values are accurate, commentary reads correctly, formatting matches the agreed template, and all three distribution channels worked as expected. Approval is recorded in the FullSpec confirmation system.
Process owner sign-off recorded in the FullSpec confirmation system. No separate signature block required. Phase 3 is closed and the automation is approved for go-live.
Once T24 is recorded as passed in the FullSpec confirmation system, the FullSpec team will update all credentials to point to the live distribution list, the active Slack support channel, and the production Notion reporting log. The qa-test-run tag filter will be removed and the schedule will be set to the agreed live cadence. Any issues discovered after go-live should be reported to support@gofullspec.com.
Test and QA PlanPage 4 of 4