FS-DOC-06Quality
Test and QA Plan
Campaign Performance Reporting
[YourCompany.com] · Marketing Department · Prepared by FullSpec · [Today's Date]
This document defines the structured test and quality assurance programme for the Campaign Performance Reporting automation. It is written for the FullSpec team who build and execute the tests, with the process owner joining Phase 3 for final acceptance. Testing runs across three sequential phases: unit tests against each agent in isolation, integration and edge-case tests across agent handoffs, and an end-to-end production run against live credentials. No phase begins until the previous phase fully passes. All test outcomes are logged in the FullSpec project record and any failure blocks progression until resolved.
01Testing approach
Testing is structured across three phases that must be completed in strict sequence. A phase is considered fully passed only when every test case in that phase reaches its defined pass condition with no outstanding failures or deferred items. A failure in any case causes the phase to remain open until the defect is resolved and the affected case is re-run from the start. The three phases are Unit, Integration, and End-to-end, described below.
1Phase 1: Unit Testing
Estimated time: 1 to 2 business days
Scope
Each agent tested in isolation: Data Collection Agent, Normalisation and Insight Agent, Distribution Agent
Environment
Sandbox only. No live API keys, no production Slack channels, no real HubSpot contacts
Credential rule
Use sandbox or developer-tier credentials exclusively. Meta test ad account, GA4 sandbox property, HubSpot sandbox portal, test Slack workspace
Pass condition
All cases T01 through T15 pass with expected output matching actual output exactly or within stated tolerance
Who runs it
FullSpec team only
2Phase 2: Integration and Edge-Case Testing
Estimated time: 1 to 2 business days
Scope
Agent-to-agent handoffs, staging-sheet write integrity, and all edge and failure cases including missing data, API unavailability, and human review timeout
Environment
Sandbox for edge cases; controlled staging environment with real API connections for handoff tests where sandbox cannot replicate the behaviour
Credential rule
Sandbox credentials for error-simulation cases. Staging credentials (read-only, non-production) for handoff flow tests. No writes to production sheets or Slack channels
Pass condition
All cases T16 through T22 and E01 through E07 pass. Error handling routes produce the correct fallback output with no unhandled exceptions
Who runs it
FullSpec team only
3Phase 3: End-to-End Production Test
Estimated time: 1 business day
Scope
Full automation run against live credentials, production Google Sheet, live Looker Studio dashboard, and designated Slack channel using an internal test address and a voided test record
Environment
Production. All credentials are live. Test records are flagged and voided after the run
Credential rule
Live production credentials authorised by the process owner. Post-test, any records written to HubSpot or Sheets during the test are deleted or flagged as void
Pass condition
All cases T23 through T27 pass, including process owner review and sign-off recorded in the FullSpec confirmation log
Who runs it
FullSpec team executes; process owner joins to review commentary and confirm final approval for T27
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
All Phase 1 tests must be executed using sandbox credentials only. Do not connect live Meta Business Manager, production GA4 properties, real HubSpot portals, or production Slack workspaces at any point during Phase 1. If a sandbox environment does not exist for a given tool, use mocked API responses recorded from a prior authenticated session.
Data Collection Agent — Complexity: Moderate. Estimated build time: 8 hours. Connects to Meta Ads Manager, Google Analytics, and HubSpot on schedule and writes raw campaign metrics to the staging sheet.
ID
Scenario
Input
Expected Output
Pass Criteria
T01
Happy path: Meta Ads API returns full campaign data
Scheduled trigger fires; sandbox Meta account has 3 active campaigns with impressions, clicks, spend, and conversions populated
All 3 campaigns written to staging sheet rows with correct column mapping: campaign_id, impressions, clicks, spend, conversions, date_start, date_end
Row count matches campaign count; no null values in required fields; date range matches trigger window exactly
T02
Happy path: Google Analytics API returns session and goal data
GA4 sandbox property with sessions, goal_completions, and source/medium populated for the reporting window
Sessions, goal completions, and UTM source/medium written to staging sheet aligned to the same date range as Meta data
Date range matches T01 window; utm_source and utm_medium fields populated; no truncated rows
T03
Happy path: HubSpot contacts and deals retrieved
HubSpot sandbox portal with 12 contacts created within reporting window, each with hs_analytics_source set to PAID_SOCIAL or ORGANIC_SEARCH
12 contact records written to staging with contact_id, create_date, campaign_source, and associated deal value where present
Contact count matches sandbox; campaign_source field populated on all 12 records; deal value is null where no deal exists, not zero
T04
Meta API authentication failure: invalid or expired access token
Sandbox trigger fires with Meta access token set to an expired value
Workflow halts at Meta fetch step; error logged with code 190 (OAuthException); no partial data written to staging sheet; alert triggered to FullSpec monitoring channel
Staging sheet row count is zero; error log contains token expiry code; downstream agents do not fire
T05
Google Analytics API returns zero rows for date range
GA4 sandbox property queried for a date range with no recorded sessions
Agent writes a zero-row placeholder record to staging; normalisation agent is notified of empty GA source; workflow does not halt
Staging sheet contains a zero-row marker row for the GA source; no unhandled null pointer exceptions; normalisation agent receives flag
T06
HubSpot API rate limit exceeded mid-pull
Sandbox configured to return HTTP 429 after the first 5 contact records
Agent pauses, waits the retry-after interval (10 seconds in sandbox), then resumes from the sixth record without duplicating the first 5
Final staging row count is 12 (all contacts); no duplicate contact_id values; rate-limit event recorded in error log
T07
Scheduler fires outside the expected cadence window (manual trigger during off-schedule hours)
Manual trigger sent at an off-schedule time to simulate an operator-initiated re-run
Agent accepts the trigger, runs the full collection sequence, and completes normally; report_type flag is set to manual_run in metadata
All three sources polled; staging sheet updated; report_type metadata field reads manual_run
Normalisation and Insight Agent — Complexity: Complex. Estimated build time: 10 hours. Takes the raw staging dataset, resolves column mismatches, calculates derived KPIs, flags underperforming campaigns, and drafts the commentary paragraph.
ID
Scenario
Input
Expected Output
Pass Criteria
T08
Happy path: clean three-source dataset normalised correctly
Staging sheet contains well-formed rows from all three sources with matching date ranges and campaign identifiers
Master sheet updated with standardised columns; cost_per_lead, ctr, conversion_rate, and roas calculated for each campaign; no formula errors
All four derived KPIs present and non-null for every campaign row; column headers match the agreed schema exactly
T09
Column name mismatch between Meta and GA4 source rows
Staging sheet contains Meta column header spend and GA4 column header cost (same field, different name)
Normalisation logic maps both to the canonical column name ad_spend; no duplicate or split column appears in master sheet
Master sheet contains exactly one ad_spend column; values from both sources correctly assigned; no orphan spend or cost column
T10
Underperforming campaign flag triggers correctly
Campaign C_003 has a roas of 0.8 against a configured threshold of 1.5
Campaign C_003 flagged in the underperformance_flag column as TRUE; commentary draft references C_003 as the biggest underperformer
underperformance_flag is TRUE for C_003 only; commentary paragraph contains the campaign name and the word underperforming
T11
Commentary draft generation: top performer and recommended action
Clean master sheet with campaign C_001 returning roas of 4.2 (highest), C_003 flagged underperforming, and a configured action template
Draft commentary names C_001 as top performer, C_003 as underperformer, and includes one recommended action sentence
Draft text is non-empty; contains the strings top performer and recommended action (or equivalent); length is between 80 and 300 characters
T12
Division-by-zero in derived KPI: zero clicks on a campaign
Campaign C_004 has clicks = 0; ctr formula attempts spend / clicks
Agent writes null or N/A to ctr for C_004 rather than returning a divide-by-zero error; formula error is logged
Master sheet cell for C_004 ctr is null or N/A; no #DIV/0! error visible in sheet; error log entry present for C_004
T13
HubSpot UTM variance warning surfaced
Meta records 140 clicks for campaign C_001; HubSpot records 118 contacts attributed to the same campaign source
Normalisation agent writes both values to separate columns (platform_clicks and crm_attributed_contacts) and sets a variance_note flag
Both columns populated; variance_note contains the text attribution variance; downstream commentary references the discrepancy
Distribution Agent — Complexity: Simple. Estimated build time: 4 hours. Formats the Slack message with key metric callouts and campaign flags, posts to the designated channel, and links to the Looker Studio dashboard.
ID
Scenario
Input
Expected Output
Pass Criteria
T14
Happy path: Slack message posted with correct content
Normalisation agent has confirmed master sheet update; Looker Studio dashboard has refreshed; sandbox Slack workspace active
Slack message posted to #marketing-reports-test channel containing top performer name, underperformer name, ROAS figure, and clickable dashboard URL
Message appears in channel within 60 seconds of trigger; all four content elements present; URL resolves to Looker Studio (200 response)
T15
Slack API returns 429 (rate limited) on first post attempt
Sandbox Slack app configured to return 429 on the first call then 200 on retry
Agent waits the retry-after value, retries once, and posts successfully; rate-limit event logged
Message delivered on second attempt; delivery timestamp within 2 minutes of trigger; rate-limit event in error log
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Integration tests verify that data flows correctly between agents across handoff boundaries, that the staging sheet acts as a reliable intermediate store, and that the Distribution Agent receives accurate inputs from the Normalisation and Insight Agent before posting. Test IDs continue from Phase 1.
ID
Scenario
Input
Expected Output
Pass Criteria
T16
Handoff 1: Data Collection Agent output triggers Normalisation Agent correctly
Data Collection Agent completes a full three-source pull and writes 9 rows to the staging sheet
Normalisation and Insight Agent fires automatically within 30 seconds of the staging sheet write completing; it reads exactly 9 rows
Normalisation agent execution log shows trigger event linked to staging sheet update; row count read = 9
T17
Handoff 1 failure: Data Collection Agent writes zero rows due to all-source failure
All three API calls return errors; staging sheet remains empty
Normalisation agent does not fire; a workflow halt notification is sent to the FullSpec monitoring channel; no Slack post is made
Normalisation agent execution log shows no trigger event; monitoring channel receives a failure alert within 5 minutes; Slack channel receives no report message
T18
Handoff 2: Normalisation Agent output triggers Distribution Agent after manager approval
Normalisation agent completes; commentary draft is approved via the review step (simulated yes path); master sheet confirmed updated
Distribution Agent fires; Looker Studio refresh is triggered before the Slack post; Slack message is posted after dashboard confirms refresh
Looker Studio refresh timestamp precedes Slack post timestamp; both events recorded in execution log
T19
Handoff 2 alternative: manager edits commentary before approval
Normalisation agent completes; reviewer edits the commentary draft text and then approves
Distribution Agent uses the edited commentary text in the Slack message, not the original AI draft
Slack message content matches the edited text exactly; original draft text not present in posted message
T20
Staging sheet write integrity: concurrent trigger does not cause duplicate rows
Two scheduler triggers fire within 10 seconds of each other (simulated race condition)
Deduplication logic allows only one execution to proceed; second trigger is suppressed and logged; staging sheet contains exactly one set of rows for the reporting period
No duplicate campaign rows in staging sheet; suppression event in execution log; master sheet row count unchanged from a single run
T21
Looker Studio dashboard does not refresh within the timeout window
Dashboard data source refresh takes longer than the 120-second timeout configured in the Distribution Agent
Distribution Agent logs a timeout warning, proceeds to post the Slack message with a note that the dashboard may take a few minutes to update, and does not block the report
Slack message posted with timeout caveat text; execution log records the refresh timeout; dashboard URL still included in message
T22
Full integration happy path: weekly report cycle end to end in staging
All three APIs return clean data; normalisation completes; reviewer approves without edits; Looker Studio refreshes within timeout; Slack posts
Every step executes in sequence with no manual intervention; Slack message delivered with accurate top-line metrics matching master sheet values
Execution log shows all 7 nodes completing in order; Slack message figures match master sheet calculated values to 2 decimal places
Edge-case tests cover scenarios that fall outside the normal happy path and are most likely to cause silent failures or incorrect data in production. Each uses its own ID series starting at E01.
ID
Scenario
Input
Expected Output
Pass Criteria
E01
Duplicate campaign rows in Meta export for the same date range
Meta API returns campaign C_002 twice with identical campaign_id and date range due to a pagination overlap
Deduplication logic in the Data Collection Agent or Normalisation Agent removes the duplicate before writing to the master sheet; one row per campaign per period
Master sheet contains exactly one row for C_002; deduplication event logged; no inflated spend or impression totals
E02
Missing data: HubSpot returns no contacts for the reporting period
HubSpot API returns an empty result set (0 contacts created in the window)
Normalisation agent writes zero to crm_attributed_contacts for all campaigns; commentary notes no CRM data available for this period; workflow does not halt
crm_attributed_contacts column populated with zeros; commentary contains the string no CRM data or equivalent; Slack message still posted
E03
Missing data: one campaign has no spend and no impressions (paused mid-period)
Campaign C_005 is paused; Meta API returns it with spend = 0, impressions = 0, clicks = 0
Campaign C_005 written to master sheet with zeros; not flagged as underperforming (paused status respected); commentary does not name it as the biggest underperformer
C_005 present in sheet with zero values; underperformance_flag = FALSE; commentary does not reference C_005 as underperforming
E04
Unavailable tool: Meta Marketing API returns 503 for the entire run
Meta API endpoint returns HTTP 503 Service Unavailable on all retry attempts (3 retries with exponential backoff)
After exhausting retries, agent writes a partial dataset (GA4 and HubSpot only) to staging with a meta_unavailable flag; monitoring alert sent; commentary draft notes Meta data is missing for this period
Staging sheet contains GA4 and HubSpot rows only; meta_unavailable flag = TRUE; monitoring channel receives alert; Slack message includes caveat about missing Meta data
E05
Unavailable tool: Google Sheets API returns 503 during the normalisation write
Google Sheets API returns 503 when the Normalisation Agent attempts to write the master sheet
Agent retries up to 3 times with backoff; if all retries fail, the workflow halts and a failure alert is sent; no partial or corrupt write occurs
Master sheet is either fully written or untouched (no partial rows); failure alert sent to monitoring channel; Distribution Agent does not fire
E06
Human review timeout: manager does not approve or reject commentary within the configured 4-hour window
Commentary draft posted for review at 09:00; no response received by 13:00
Workflow sends a reminder notification to the reviewer at 13:00; if no response by 17:00 (8-hour total), the workflow escalates to the process owner and halts the Slack post until a decision is made
Reminder notification sent at 4-hour mark; escalation sent at 8-hour mark; no Slack post made until approval received; all timeout events in execution log
E07
Date range mismatch between Meta and GA4 (Meta returns one extra day due to timezone offset)
Meta API returns data for Mon to Sun in UTC+0; GA4 returns data for Mon to Sun in UTC+5 (effective date range shifts by one day)
Normalisation agent detects the offset using the configured timezone parameter, trims the Meta dataset to align with the canonical UTC reporting window, and logs the offset event
Both sources cover the same canonical date range after normalisation; offset event in execution log; no data loss or double-counting for the boundary day
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: use an internal test email address as the Slack notification recipient (not the full stakeholder list) and post to a designated internal test channel, not the live #marketing-reports channel. Any records written to HubSpot, Google Sheets, or Looker Studio during this test run must be clearly tagged as TEST and voided or deleted by the FullSpec team immediately after the run is confirmed. Do not run Phase 3 until Phase 1 and Phase 2 are fully signed off with zero open failures.
Credential check before Phase 3: confirm that live Meta Business Manager API access token is valid, GA4 property ID is the production property, HubSpot portal is the production portal, and the Slack app has permission to post to the test channel. FullSpec will verify each credential in the execution log before triggering the run.
ID
Scenario
Input
Expected Output
Pass Criteria
T23
Full production run: scheduler triggers and all three APIs return live data
Scheduled trigger fires against production credentials; live Meta, GA4, and HubSpot data available for the current reporting window
All three sources polled successfully; staging sheet populated with live campaign rows; no API authentication errors in execution log
Execution log shows 200 responses from all three APIs; staging sheet row count greater than zero; timestamp matches the scheduled run window
T24
Normalisation and derived KPIs correct against live data
Staging sheet contains live production data from T23
Master sheet updated with standardised columns; cost_per_lead, ctr, conversion_rate, and roas calculated and non-null for all campaigns with sufficient data; underperformance flags applied where roas is below the configured threshold
All four derived KPI columns present; no formula errors in master sheet; underperformance flags match manual spot-check of the raw figures to within 1%
T25
Commentary draft produced and routed to reviewer
Master sheet confirmed updated after T24; commentary draft generated by Normalisation and Insight Agent
Draft commentary paragraph delivered to the designated reviewer (process owner or delegated marketing manager) via the configured review channel; draft references at least one named campaign
Commentary text is non-empty; reviewer receives the draft notification within 2 minutes of master sheet confirmation; draft contains a campaign name present in the master sheet
T26
Error log verification: no unhandled exceptions in the full production run
Execution log from T23 through T25 reviewed in full
Error log contains zero unhandled exceptions; any retries or warnings are logged with context but classified as informational, not critical
Zero entries with severity level ERROR or CRITICAL in the execution log; any WARN entries reviewed and documented as acceptable or raised as defects
T27
Process owner review and final approval: Slack message reviewed, dashboard confirmed live, and sign-off recorded
Process owner reviews the full Slack test message in the internal test channel and opens the Looker Studio dashboard link; FullSpec presents the execution log summary
Process owner confirms that the Slack message content is accurate, the dashboard reflects live data, and the commentary is appropriate for stakeholder distribution; approval is recorded by FullSpec in the project confirmation log
Process owner verbally or in writing confirms approval; FullSpec records the outcome, approver name, and timestamp in the FullSpec project confirmation log. This record constitutes the Phase 3 sign-off. No separate signature block is required.
After T27 passes, FullSpec will void all test records created during the Phase 3 run, switch the Slack destination from the internal test channel to the live #marketing-reports channel, and confirm the go-live schedule with the process owner. The first fully automated production report run will be monitored in real time by the FullSpec team. Contact support@gofullspec.com for any queries after go-live.
Test and QA PlanPage 4 of 4