FS-DOC-06Quality
Test and QA Plan
Churn Risk Identification
[YourCompany.com] · Customer Support Department · Prepared by FullSpec · [Today's Date]
This document defines every test case the FullSpec team runs before the Churn Risk Identification automation goes live. It is structured in three sequential phases: unit tests for each agent in isolation, integration and edge-case tests for agent handoffs and failure scenarios, and a final end-to-end production run with process owner sign-off. No phase begins until the previous phase fully passes. The process owner joins Phase 3 only. All sandbox and staging work is handled entirely by the FullSpec team.
01Testing approach
Testing runs in three phases in strict sequence. Each phase must fully pass before the next phase begins. Phase 1 isolates each agent and validates its individual logic. Phase 2 tests the handoffs between agents and exposes edge cases such as missing data, duplicate records, and tool unavailability. Phase 3 runs the complete flow against the production environment with real credentials and real accounts, and concludes with the process owner confirming the output is correct.
1Phase 1: Unit Tests
Each agent tested in isolation against sandbox data
Scope
Churn Signal Aggregator and Risk Scoring and Routing Agent tested independently
Environment
Sandbox only. No production credentials used.
Credential rule
All API keys must be sandbox or developer-tier tokens. Mixpanel sandbox project, Intercom test workspace, HubSpot developer sandbox account.
Pass condition
All cases T01 through T12 return expected output with zero errors logged.
Who runs it
FullSpec team only.
2Phase 2: Integration and Edge Case Tests
Agent handoffs, tool failures, and boundary conditions
Scope
End-to-end agent chain from trigger to HubSpot task creation, plus all identified edge cases
Environment
Sandbox for integration tests. Staging environment (cloned production config, sandbox credentials) for edge cases.
Credential rule
Sandbox credentials only. No live customer data accessed at this phase.
Pass condition
All cases T13 through T20 pass and all edge cases E01 through E06 produce the correct fallback or error-handling behaviour.
Who runs it
FullSpec team only.
3Phase 3: End-to-End Production Test
Live environment with real accounts; process owner joins this phase
Scope
Full workflow run against production environment including real Mixpanel events, live Intercom tickets, HubSpot CRM records, Slack channel, and Gmail outbox
Environment
Production. All live credentials active.
Credential rule
Production API keys in use. Test records must be clearly labelled and voided after the run.
Pass condition
All cases T21 through T25 pass, error logs are clean, and the process owner confirms the output at T25.
Who runs it
FullSpec team runs T21 through T24. Process owner joins for T25 review and approval.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
All Phase 1 tests must use sandbox credentials exclusively. Do not connect any Phase 1 test run to production Mixpanel projects, live Intercom workspaces, or real HubSpot portals. Confirm sandbox token configuration in the automation platform before running any case.
Agent 1: Churn Signal Aggregator
ID
Scenario
Input
Expected Output
Pass Criteria
T01
Happy path: all three sources return data
Sandbox accounts with Mixpanel events, open Intercom tickets, and HubSpot records all present
Normalised account snapshot for every active account; all fields populated
Snapshot contains usage score, ticket count, ticket age, last contact date, and HubSpot owner for 100% of test accounts
T02
Mixpanel returns no events for an account
One sandbox account with zero Mixpanel events in the past 30 days
Account snapshot includes a null or zero usage score; record still created
Snapshot is generated; usage_score field is 0 or explicitly null; no error thrown
T03
Intercom API returns an empty ticket list
Sandbox account with no open Intercom conversations
Snapshot includes ticket_count: 0 and ticket_age_days: null
Field is present in output with correct zero/null value; pipeline continues
T04
HubSpot contact record has no last contact date
HubSpot sandbox record with last_contact_date field blank
Snapshot includes last_contact_date: null and owner is still resolved from company record
Null is correctly propagated; owner field is not null if company record has an assigned rep
T05
Mixpanel API returns a 429 rate-limit error
Simulated 429 response from Mixpanel sandbox
Agent retries up to three times with exponential back-off; logs a warning if all retries fail
Retry logic fires; after three failures the run logs a warning and skips the Mixpanel pull for that account without crashing the pipeline
T06
Intercom API is unreachable (simulated timeout)
Intercom sandbox endpoint returns a 503 after 10 seconds
Agent logs a TOOL_UNAVAILABLE error; snapshot is flagged as incomplete; downstream agents are notified
Error is logged with tool name and timestamp; incomplete flag is set on the affected snapshots; no uncaught exception
T07
Large account batch (100+ accounts)
Sandbox with 120 active account records
All 120 snapshots generated within the configured time window (default: 10 minutes)
All 120 snapshots present in output; no timeout error; pagination handled correctly for Mixpanel and Intercom
Agent 2: Risk Scoring and Routing Agent
ID
Scenario
Input
Expected Output
Pass Criteria
T08
Happy path: account clearly high risk
Snapshot with usage drop >40%, 4 open tickets, oldest ticket 12 days, last contact 21 days ago
Risk tier: HIGH; Slack alert drafted; Gmail outreach email drafted; HubSpot payload prepared
All four outputs present; risk tier is HIGH; Slack payload includes account name, signals, and assigned CSM; email subject and body contain account-specific merge fields
T09
Happy path: account clearly low risk
Snapshot with no usage drop, zero open tickets, last contact 3 days ago
Risk tier: LOW; no Slack alert; no outreach email drafted; HubSpot payload prepared with LOW tier
Only HubSpot write payload generated; Slack and Gmail steps skipped; tier field is LOW
T10
Account sits exactly on the high/medium threshold
Snapshot with signals at the configured boundary values for HIGH tier
Risk tier resolves to HIGH (boundary inclusive rule confirmed in config)
Tier is HIGH; scoring log shows boundary condition was evaluated correctly
T11
Account has an incomplete snapshot (incomplete flag set)
Snapshot with incomplete: true from a T06-style Intercom failure
Risk tier: INCONCLUSIVE; Slack alert sent with data-gap warning; no outreach email drafted
Tier is INCONCLUSIVE; Slack message includes a note that Intercom data was unavailable; no Gmail payload created
T12
HubSpot write-back fails with a 403 permission error
Simulated 403 from HubSpot sandbox on the properties write endpoint
Error logged with HTTP status, property name, and account ID; Slack alert still fires; run does not halt
Error row appears in the run log; all other outputs for that account are still generated; pipeline continues to next account
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Integration tests validate that the two agents hand off correctly and that every downstream tool receives well-formed payloads. IDs continue from Phase 1.
ID
Scenario
Input
Expected Output
Pass Criteria
T13
Full handoff: Aggregator snapshot triggers Scoring Agent
Aggregator completes a full batch run in sandbox
Scoring Agent fires automatically on snapshot completion; no manual trigger required
Scoring Agent execution timestamp is within 30 seconds of Aggregator completion; all snapshot records are passed
T14
HubSpot risk tier and owner written correctly
Scoring Agent produces HIGH-tier output for a sandbox account assigned to CSM sandbox user
HubSpot contact record shows churn_risk_tier: HIGH and hubspot_owner updated to correct CSM
HubSpot GET on the contact confirms field values match the scoring output; no duplicate property writes
T15
Slack alert format and content validated
HIGH-risk account with three distinct churn signals
Slack message posted to designated channel; contains account name, risk tier badge, three signal lines, and assigned CSM handle
Message appears in sandbox Slack channel within 60 seconds; all four content elements present; no raw JSON visible in message body
T16
Gmail outreach draft created with correct merge fields
HIGH-risk account; CSM name, account name, and primary churn signal available in snapshot
Draft email in CSM Gmail sandbox contains correct salutation, account name, and signal reference; subject line populated
Draft visible in Gmail sandbox drafts folder; all merge fields resolved; no placeholder tokens remaining (e.g. no {{account_name}} visible in body)
T17
CSM approves outreach: email sends and HubSpot logs activity
CSM clicks approve in Slack or approval interface for a HIGH-risk account
Gmail sends the draft; HubSpot logs an EMAIL_SENT activity on the contact; a follow-up task is created with due date based on HIGH-tier SLA
Gmail sent items contains the email; HubSpot activity timeline shows the log entry; task due date is within the HIGH-tier window (default: 2 business days)
T18
CSM edits and resends email: edited version is sent and logged
CSM clicks edit, modifies the draft body, and confirms send
Edited version is sent via Gmail; HubSpot logs the activity with an EDITED flag or note; original draft is discarded
Sent email body reflects the CSM's edits; HubSpot activity note indicates manual edit; no duplicate sends
T19
Google Sheets audit log entry created for every run
One complete batch run covering 10 sandbox accounts
Google Sheets log appends one row per account per run: timestamp, account ID, risk tier, signals, action taken
Sheet contains exactly 10 new rows after the run; all required columns populated; no blank rows or misaligned data
T20
Daily scheduled trigger fires at configured time
Automation platform scheduler set to 07:00 in the configured timezone
Aggregator begins execution at 07:00 (plus or minus 2 minutes)
Execution start timestamp in run log is between 06:58 and 07:02; no manual trigger required
Edge Case Tests
Edge case tests deliberately introduce abnormal inputs and tool failures. Each case must confirm that the automation handles the condition gracefully and does not crash, send incorrect data to a live system, or leave a record in an inconsistent state.
ID
Scenario
Input
Expected Output
Pass Criteria
E01
Duplicate account record in HubSpot
Two HubSpot sandbox contacts share the same company domain and account name
Automation flags the duplicate in the run log; uses the most recently modified record; does not write two separate HubSpot updates
Log contains a DUPLICATE_DETECTED entry with both contact IDs; only one record is updated; no error thrown
E02
Account exists in Mixpanel but not in HubSpot
Mixpanel sandbox event data for an account ID with no matching HubSpot contact
Account is skipped for scoring; a MISSING_CRM_RECORD warning is logged; no HubSpot write attempted
Warning row in log includes the Mixpanel distinct_id; no 404 error surfaces to the run output; pipeline continues to next account
E03
Intercom ticket data contains malformed JSON
Intercom sandbox returns a conversation object with a null messages array and missing created_at field
Agent catches the parse error; account snapshot marks ticket_data_valid: false; scoring continues with available fields
No uncaught exception; ticket_data_valid flag is false in the snapshot; scoring tier is derived from Mixpanel and HubSpot data only
E04
Slack API is unavailable during alert dispatch
Simulated 503 from Slack API during message post
Automation retries twice; if both fail, logs a SLACK_ALERT_FAILED entry and continues; HubSpot update and Gmail draft still proceed
SLACK_ALERT_FAILED entry in run log with timestamp and HTTP status; HubSpot and Gmail steps are not blocked by the Slack failure
E05
CSM does not approve or reject the outreach within 48 hours
Approval request sent for a HIGH-risk account; no response after 48 hours in test simulation
Automation sends a reminder Slack message to the CSM; after a further 24 hours with no response, escalates to the Support Lead and logs a TIMEOUT_ESCALATED entry
Reminder message visible in Slack sandbox 48 hours after original alert; escalation message sent at 72 hours; HubSpot task is not marked complete; no email is sent automatically
E06
All accounts score LOW risk on a given day
Batch run where no account meets the HIGH or MEDIUM threshold
No Slack alerts posted; no Gmail drafts created; HubSpot updated with LOW tier for all accounts; Google Sheets log records the full batch as LOW
Slack channel receives no messages; Gmail drafts folder is empty; all HubSpot records show LOW; Sheets log row count matches account count
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: Use an internal test email address (not a real customer address) as the outreach recipient for all production test runs. Create a clearly labelled test account in HubSpot (e.g. TEST-CHURN-QA-001) and in Mixpanel before starting. Void and delete all test records in HubSpot, Slack, and Google Sheets immediately after Phase 3 is complete. The FullSpec team will confirm deletion in writing before go-live is approved.
Phase 3 runs the complete automation against the live production environment using real API credentials, real Slack channels, and real Gmail accounts. The FullSpec team coordinates the run timing with the process owner to avoid disrupting the normal daily workflow. All test records are clearly labelled TEST and removed from every system before go-live.
ID
Scenario
Input
Expected Output
Pass Criteria
T21
Full production run: trigger to Aggregator
Daily schedule trigger fires against production Mixpanel, Intercom, and HubSpot using the TEST-CHURN-QA-001 account plus a sample of 10 real accounts (read-only pull)
Normalised snapshots generated for all 11 accounts; test account snapshot reflects seeded usage data; run completes within the 10-minute window
Run log shows 11 snapshots; test account snapshot values match seeded Mixpanel events and Intercom tickets; no API errors in the production run log
T22
Production scoring and HubSpot write-back
TEST-CHURN-QA-001 seeded as HIGH risk (usage drop 50%, 5 open tickets, last contact 30 days ago)
Risk tier HIGH written to the test contact in production HubSpot; owner field set to the designated test CSM
HubSpot production portal shows churn_risk_tier: HIGH on TEST-CHURN-QA-001; owner is correct; no other live contact records are modified during the test run
T23
Production Slack alert and Gmail draft
HIGH-risk tier confirmed for TEST-CHURN-QA-001
Slack alert posted to the production churn-alerts channel with correct content; Gmail draft created in the test CSM's production Gmail account addressed to the internal test email address
Alert visible in the production Slack channel within 60 seconds; all content fields correct; draft in Gmail addressed to the internal test address only; no real customer email address in any field
T24
Error log verification
Review the production run log after T21 through T23
No ERROR-level entries; any WARN-level entries are documented and acknowledged; Google Sheets audit log contains a complete row for every account in the run
Run log is free of ERROR entries; Sheets log row count matches the account batch; FullSpec team reviews and signs off the log before T25 begins
T25
Process owner review and approval
Process owner (Customer Success Lead) reviews: the Slack alert content, the Gmail draft quality, the HubSpot record update, the Google Sheets audit log row, and the overall run timing
Process owner confirms all outputs are correct, the email tone and content meet the team standard, and the automation is ready for go-live
Process owner provides explicit approval recorded in the FullSpec confirmation email to support@gofullspec.com. This approval constitutes the Phase 3 pass condition and authorises go-live. No separate signature block is required.
Once T25 is approved, the FullSpec team will remove all TEST-CHURN-QA-001 records from HubSpot, Mixpanel, Intercom, Slack, Gmail, and Google Sheets, and will confirm deletion in writing. The automation will then be enabled on its live daily schedule. Contact support@gofullspec.com with any questions before or after go-live.
Test and QA PlanPage 4 of 4