FS-DOC-06Quality
Test and QA Plan
Client Reporting Automation
[YourCompany.com] · Operations Department · Prepared by FullSpec · [Today's Date]
This document defines the full test and quality assurance programme for the Client Reporting Automation. It covers three sequential phases: unit testing of each individual agent in a sandbox environment, integration and edge-case testing of agent handoffs and exception paths, and a final end-to-end production run with process owner sign-off. The FullSpec team runs Phases 1 and 2 independently. The process owner joins Phase 3 to verify real-world behaviour before the automation goes live. No phase begins until all cases in the previous phase have passed.
01Testing approach
Testing runs in three phases in strict sequence. No phase begins until every test case in the previous phase carries a confirmed pass status. This prevents integration tests from running against an agent whose core logic has not yet been validated, and prevents a production run from proceeding until all known edge cases have been handled. Each phase has a defined environment, credential rule, pass condition, and a nominated team responsible for execution.
1Phase 1: Unit Testing
Individual agent logic validated in isolation
Scope
Each agent tested independently: Report Data Collector and Report Builder and Distributor. No live inter-agent handoffs in this phase.
Environment
Sandbox only. HubSpot sandbox account, Google Sheets test workbook, Looker Studio dev connector, Gmail sandbox sending identity, test Slack workspace.
Credential rule
No production credentials may be loaded. All API keys, OAuth tokens, and service accounts must reference sandbox or test environments exclusively.
Pass condition
All cases T01 through T12 return the expected output with no unhandled errors. Any failure blocks progression to Phase 2.
Who runs it
FullSpec team. Process owner not required in this phase.
2Phase 2: Integration and Edge-Case Testing
Agent handoffs, data flow, and exception paths
Scope
End-to-end agent handoff from Report Data Collector to Report Builder and Distributor, plus all identified edge cases including duplicate records, missing data, unavailable tools, and human review timeout.
Environment
Sandbox for the automation platform orchestration layer. External tool sandboxes remain active. No emails sent to real client addresses.
Credential rule
Sandbox credentials only. Production HubSpot, Gmail, and Google Drive accounts remain disconnected throughout Phase 2.
Pass condition
Integration cases T13 through T18 and edge cases E01 through E06 all pass. All exception paths terminate gracefully with a logged error and no orphaned records.
Who runs it
FullSpec team. Process owner may observe but is not required to action anything in this phase.
3Phase 3: End-to-End Production Test
Full production run with process owner review and approval
Scope
One complete production cycle per report variant: schedule trigger through HubSpot data fetch, Sheets staging, Looker Studio refresh, AI commentary draft, optional human review gate, PDF export, Gmail send, HubSpot activity log, and Slack notification.
Environment
Live production credentials and accounts. Internal test client records used. All outbound emails directed to an internal test address before go-live is confirmed.
Credential rule
Production credentials active. Every action taken during Phase 3 against a real system must be voided or deleted after the test run unless it is explicitly retained as a reference record.
Pass condition
Cases T19 through T22 all pass. Process owner confirms accuracy, completeness, and delivery behaviour and records sign-off in the FullSpec confirmation log.
Who runs it
FullSpec team executes. Process owner joins to review outputs, approve commentary samples, and record final approval in the FullSpec system.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
Sandbox credentials only throughout Phase 1. No production API keys, OAuth tokens, or service account files may be present in the automation platform environment during these tests. Any test that accidentally touches a production system is automatically voided and must be re-run in a clean sandbox state before a pass can be recorded.
Report Data Collector Agent — Complexity: Moderate. Estimated build time: 12 hours.
ID
Scenario
Input
Expected Output
Pass Criteria
T01
Happy path: scheduled trigger fires for a single client with a complete HubSpot record
Client ID from Google Sheets schedule config; HubSpot sandbox account with all expected metric fields populated for the reporting period
Google Sheets staging tab written with all metric fields populated; no error flags raised
All mapped HubSpot fields present in staging tab; row count matches expected client entry; completion status logged as success
T02
Multiple clients due on the same date processed sequentially
Three client IDs in the schedule config with identical reporting dates
Three separate staging tab rows written, one per client, each with correct client code and period label
No row overwrites another; each client ID maps to its own tab row; all three completion statuses logged
T03
HubSpot API authentication succeeds with a valid OAuth token
Valid sandbox OAuth token; HubSpot sandbox account accessible
API returns a 200 response; data object populated with account metrics
HTTP 200 received; response body non-empty; token expiry not triggered during the request cycle
T04
HubSpot API returns 401 Unauthorized due to an expired token
Expired or revoked OAuth token injected into the credential store
Agent catches the 401, logs an authentication error to the error log, and halts without writing to Google Sheets
No partial data written to staging tab; error log entry present with timestamp and client ID; workflow marked as failed
T05
One or more expected HubSpot metric fields are absent from the API response
HubSpot sandbox record with two of the standard metric properties removed
Staging tab row written with present fields; absent fields flagged as MISSING in their respective cells; error flag raised on the row
MISSING flag visible in the correct cells; row-level error flag set to true; no default or zero values silently substituted
T06
Google Sheets write fails due to a permissions error on the staging workbook
Service account token with read-only scope applied to the staging workbook
Agent catches the write error, logs it with the affected client ID and sheet reference, and halts without partial write
No partial data visible in staging tab; error log entry includes sheet URL and permission scope error message
T07
Trigger fires outside the expected schedule window (off-cycle manual trigger)
Manual trigger call with a valid client ID outside the scheduled date window
Agent runs normally and completes the data pull; no schedule-guard rejection
Staging tab written correctly; no schedule validation error raised; useful for operator-initiated reruns
Report Builder and Distributor Agent — Complexity: Moderate. Estimated build time: 16 hours.
ID
Scenario
Input
Expected Output
Pass Criteria
T08
Happy path: staging tab fully populated, account not flagged for review, standard delivery
Fully populated staging tab row; account review flag set to false; sandbox Gmail, Drive, HubSpot, and Slack credentials active
Looker Studio refreshes; PDF exported to Drive with correct file name; email sent to sandbox test address; HubSpot activity logged; Slack message posted
All five downstream actions confirmed in their respective sandbox environments; no manual intervention required; workflow marked complete
T09
Account flagged for human review: commentary draft visible before send step
Staging tab populated; account review flag set to true
Workflow pauses after AI commentary draft is generated; draft visible in the review interface; send step does not execute until approval action is received
Send step not triggered before approval; draft text present and readable; approval action unblocks the remainder of the workflow
T10
Looker Studio report refreshes successfully and reflects updated staging data
Staging tab with a specific metric value updated from the previous cycle value
Refreshed Looker Studio report shows the new value in the correct chart or summary field
Looker Studio connector confirms refresh completion; updated value visible in the report data source within the expected latency window (under 90 seconds)
T11
PDF export fails: Looker Studio connector timeout during export step
Connector timeout simulated by temporarily revoking the export service account scope
Agent logs the export failure with connector error detail; workflow halts before Gmail send step; no email dispatched
Gmail send step not triggered; error log entry includes connector name, error type, and client ID; staging tab row status updated to export-failed
T12
Gmail send fails due to a recipient address not found in sandbox
Malformed or non-existent sandbox recipient email address in the client record
Gmail API returns a 422 or bounce equivalent; agent logs the delivery failure; HubSpot activity record updated with failure status
No email delivered; HubSpot activity entry shows delivery-failed status; Slack notification not posted; error log entry present with recipient address and error code
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Phase 2 validates the handoff between the two agents and the behaviour of the full pipeline when external conditions deviate from the happy path. Integration tests continue the T sequence. Edge-case tests use a separate E sequence.
ID
Scenario
Input
Expected Output
Pass Criteria
T13
Happy path handoff: Report Data Collector completion event triggers Report Builder and Distributor without delay
Report Data Collector completes successfully for a single client; staging tab confirmed populated
Report Builder and Distributor begins within the configured trigger latency (under 60 seconds); no manual kick required
Trigger latency under 60 seconds confirmed in execution log; Report Builder and Distributor run ID linked to the originating Report Data Collector run ID
T14
Handoff for a flagged review account: pipeline pauses correctly between agents at the commentary review gate
Handoff event fires for an account with review flag true; human reviewer assigned in config
Report Builder and Distributor reaches the review gate and waits; reviewer receives a notification with the draft; send step remains blocked
Send step not triggered within a 10-minute observation window; reviewer notification delivered to the correct sandbox channel or email address
T15
Full pipeline completes for five clients triggered in the same batch run
Five client IDs with the same reporting date in the schedule config; all HubSpot records complete
All five reports built, exported, emailed, logged in HubSpot, and posted to Slack; no client record processed twice
Five distinct run entries in the execution log; five Drive files with unique file names; five HubSpot activity records; five Slack messages; no duplicate client processing
T16
Report Data Collector fails for one client in a multi-client batch; remaining clients continue
One client HubSpot record deliberately missing required fields; four other clients complete
Failing client logged as error and skipped; the four successful clients complete the full pipeline independently
Error log contains exactly one failure entry with the affected client ID; four successful run entries confirmed; no cross-client data contamination
T17
HubSpot activity log is written after Gmail send in the correct sequence
Standard handoff for a non-flagged account; execution log timestamps captured
HubSpot activity record timestamp is later than the Gmail send timestamp in the execution log
Execution log shows Gmail send event preceding HubSpot log event; time delta is positive and under 30 seconds
T18
Slack delivery notice contains the correct client name, reporting period, and timestamp for each completed report
Batch of three clients completed; Slack sandbox workspace monitored
Three Slack messages posted; each contains the correct client display name, period label, and a timestamp matching the Gmail send time
Three messages visible in the Slack sandbox channel; client name, period, and timestamp fields validated against the execution log for each
Edge-case tests cover the failure modes and boundary conditions most likely to arise in live operation. Each case must terminate gracefully with a logged outcome and no orphaned records in any connected system.
ID
Scenario
Input
Expected Output
Pass Criteria
E01
Duplicate trigger fires for the same client and period within the same run window
Schedule config entry duplicated for one client ID on the same date; trigger fires twice
Second trigger detected as a duplicate; second run aborted before HubSpot query; error log entry created noting the duplicate
Only one staging tab row written for the affected client; only one report generated and delivered; duplicate detection log entry confirmed
E02
Client contact email address missing from HubSpot record at send time
HubSpot sandbox record with the primary contact email field empty
Gmail send step skipped; error logged with client ID and field name; HubSpot activity record updated with missing-recipient status; Slack alert posted to the internal error channel
No email dispatched; HubSpot activity shows missing-recipient; Slack error alert visible in the designated channel; staging tab row not deleted
E03
Google Sheets staging tab unavailable at the time of write (service outage simulation)
Google Sheets API returning 503 for the target workbook during the write step
Agent retries the write up to three times with exponential backoff; after three failures, logs a permanent error and halts the run for the affected client
Three retry attempts visible in the execution log with increasing delay intervals; permanent error logged after the third failure; no partial row written; other clients in the batch unaffected
E04
Looker Studio connector unavailable at the time of report refresh (third-party connector timeout)
Looker Studio connector endpoint returning a timeout error during the refresh call
Refresh step logs the timeout; workflow halts before PDF export; no email sent; error notification posted to the internal Slack error channel
PDF export step not triggered; Slack error alert posted with connector error detail; HubSpot activity not logged (run incomplete); staging tab data preserved for retry
E05
Human reviewer does not action the commentary review gate within the configured timeout window
Account flagged for review; reviewer notification sent; no action taken within the 24-hour timeout window
Workflow automatically escalates after timeout: a second Slack alert sent to the process owner; run status updated to escalated-pending-review; send step remains blocked
Second Slack alert confirmed in the sandbox channel after the 24-hour mark; send step not triggered; run status in execution log shows escalated-pending-review; original draft preserved
E06
Report period data in HubSpot returns values that are statistical outliers (e.g. zero across all metrics)
HubSpot sandbox record returning all numeric metric fields as zero for the reporting period
Staging tab written with the zero values; a data-quality warning flag raised on the row; AI commentary draft notes the anomaly explicitly; reviewer alert triggered regardless of account review flag setting
Zero values written without substitution; data-quality flag visible on the staging row; commentary draft contains an explicit anomaly note; reviewer alert triggered and confirmed in the sandbox notification channel
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: all outbound emails during this phase must be directed to an internal test address, not to any real client contact. Confirm the test address is set in the Gmail send step configuration before the run begins. Any HubSpot activity records, Google Drive files, and Slack messages created during Phase 3 testing must be reviewed after the run and either voided, deleted, or clearly labelled as TEST records before the automation is switched to live client accounts. The FullSpec team will coordinate this cleanup with your team immediately after each Phase 3 run.
ID
Scenario
Input
Expected Output
Pass Criteria
T19
Full production run: schedule trigger through to Slack notification for a single standard (non-flagged) account
One real client record in HubSpot with a reporting date set to today; Google Sheets schedule config pointing to the production workbook; all production credentials active; Gmail send step configured to the internal test address
Trigger fires at the scheduled time; HubSpot data fetched and written to the production staging tab; Looker Studio report refreshes; AI commentary draft generated; PDF exported to Drive with the correct file name convention (ClientCode-Period-Report.pdf); email delivered to the internal test address with PDF attached; HubSpot activity record created with timestamp and Drive link; Slack message posted to the designated delivery channel
All nine pipeline steps confirmed in the execution log with sequential timestamps; PDF file name matches the convention; email received at the test address within five minutes of the trigger time; HubSpot activity record links to the correct Drive file; Slack message content matches the client name and period label
T20
Full production run for a flagged review account: commentary review gate active, reviewer approves, send proceeds
One real client record flagged for human review in the production config; reviewer is the process owner; internal test address set for Gmail send
Pipeline pauses at the commentary review gate; process owner receives the review notification via Slack or email; process owner approves the draft; send step executes; all downstream steps complete normally
Pause confirmed in execution log before the send step; reviewer notification delivered; approval action recorded in the execution log with the reviewer identity and timestamp; send completes after approval; end-to-end elapsed time from trigger to Slack notification logged
T21
Error log verification: a deliberate data-missing scenario run in production to confirm error logging and alerting behaviour
One real HubSpot record with one expected metric field temporarily cleared; production credentials active; internal test address set
Staging tab row written with the MISSING flag in the affected cell; data-quality warning raised; AI commentary notes the missing field; error log entry created; Slack error alert posted to the internal error channel; email delivery proceeds with the caveat noted in the commentary
MISSING flag visible in the correct staging tab cell after the run; error log entry confirmed with field name and client ID; Slack error alert received in the correct channel; email delivered to the internal test address with the caveat present in the commentary text; production HubSpot record restored to its original state after the test
T22
Process owner review and approval: owner confirms end-to-end accuracy, commentary quality, delivery behaviour, and records sign-off
Outputs from T19 and T20: the received test emails with PDFs, the Drive archived files, the HubSpot activity records, and the Slack delivery notices
Process owner reviews each output against the expected format and content. Owner confirms: data values in the PDF match the HubSpot source record; commentary is coherent and relevant to the period metrics; email format and subject line are correct; Drive file naming convention is consistent; HubSpot activity record is accurate; Slack message content is correct. Owner records approval in the FullSpec confirmation log.
Process owner approval recorded in the FullSpec system with the owner name, date, and confirmation that all T19 and T20 outputs were reviewed and accepted. All Phase 3 test records labelled or removed from production systems. FullSpec team confirms the automation is cleared for go-live with live client accounts.
Once T22 is recorded as approved in the FullSpec system, the automation is considered QA-complete. The FullSpec team will update the production configuration to point Gmail sends to live client addresses, remove any TEST labels from retained Drive files, and confirm the go-live status in the project dashboard. For any questions during the testing phases, contact the FullSpec team at support@gofullspec.com.
Test and QA PlanPage 4 of 4