Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Competitor Monitoring Automation

[YourCompany.com] · Marketing Department · Prepared by FullSpec · [Today's Date]

This document defines the complete test and quality assurance programme for the Competitor Monitoring automation. It covers three sequential phases: unit testing of individual agents, integration and edge-case testing of agent handoffs and failure modes, and a final end-to-end production run. The FullSpec team executes Phases 1 and 2 in full; your process owner joins Phase 3 to confirm that the live output meets the agreed standard before the automation is signed off for ongoing operation. No phase begins until the previous phase fully passes all of its test cases.

01Testing approach

Testing is structured in three phases that run in strict sequence. Phase 1 validates each agent in isolation using sandbox credentials. Phase 2 tests the handoffs between agents and exercises known failure modes and edge cases. Phase 3 runs a single full cycle in the production environment and requires sign-off from the process owner. No phase begins until every test case in the previous phase carries a passing result and that result has been logged by the FullSpec team.

Phase 1: Unit Testing

Each agent tested in isolation before any integration work begins

Scope

Data Collection Agent and Intelligence Summariser Agent tested independently; no live handoffs between agents.

Environment

Sandbox only. No production API keys, no live Slack channel, no live HubSpot record.

Credential rule

All API calls use sandbox or developer-tier credentials. SEMrush and Ahrefs test tokens must be scoped to read-only sandbox data. Google Sheets target must be a dedicated test sheet.

Pass condition

All cases T01 through T10 return expected outputs with no unhandled errors.

Who runs it

FullSpec team only.

Phase 2: Integration and Edge-Case Testing

Agent handoffs and failure scenarios verified before production credentials are used

Scope

End-to-end agent handoff from Data Collection Agent to Intelligence Summariser Agent, plus all defined edge cases covering duplicates, missing data, tool unavailability, and human review timeout.

Environment

Sandbox for edge-case tests. A staging Slack channel and a staging Google Sheet are used for handoff tests. HubSpot sandbox environment for CRM log tests.

Credential rule

Sandbox credentials throughout. Production API keys must not be used in Phase 2 under any circumstances.

Pass condition

Integration cases T11 through T15 and edge cases E01 through E06 all pass. No unhandled exception propagates silently.

Who runs it

FullSpec team only.

Phase 3: End-to-End Production Test

Full live run traced from trigger to HubSpot log; process owner reviews and approves

Scope

One complete production cycle starting from the Monday 7 a.m. schedule trigger and ending with a verified HubSpot note. All production tools are live: SEMrush, Ahrefs, Google Alerts RSS, Google Sheets, Slack, and HubSpot.

Environment

Full production environment. Live API keys, live Slack channel, live Google Sheet tracker, live HubSpot record.

Credential rule

Production credentials active. The test run must use an internal test Slack channel and a designated test competitor entry so that the live digest is not confused with a real weekly report. The test competitor record must be voided in Google Sheets and HubSpot after the run.

Pass condition

Cases T16 through T19 all pass, including process owner review and written approval recorded in the FullSpec confirmation log.

Who runs it

FullSpec team executes and monitors. Process owner (Marketing Manager) joins to review the live digest output and record approval.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only. All Phase 1 tests must be executed against sandbox or developer-tier API tokens for SEMrush and Ahrefs. The target Google Sheet must be the designated test sheet, not the live tracker. Do not use production Slack webhooks or live HubSpot credentials at any point during Phase 1.

Data Collection Agent — Build complexity: Moderate. Estimated build time: 12 hours.

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: weekly schedule fires and all three sources return data

Scheduled trigger fires at 07:00 Monday; sandbox SEMrush, Ahrefs, and Google Alerts RSS all respond with valid payloads for five test competitors.

A single structured JSON dataset containing SEO signals, backlink data, and news mentions for all five competitors, normalised to the agreed schema.

JSON dataset present; all five competitor objects populated; no null fields in required schema keys; execution completes within 120 seconds.

T02

SEMrush API returns data correctly for all tracked competitors

Sandbox SEMrush token; five competitor domains passed as parameters.

Keyword ranking deltas, estimated traffic change, and new paid ad keywords returned per competitor.

Response contains all required fields: domain, keyword_delta, traffic_estimate, new_ad_keywords. HTTP 200 for each call.

T03

Ahrefs API returns backlink and content data correctly

Sandbox Ahrefs token; five competitor domains.

New backlinks count, domain rating, and recently published URLs returned per competitor.

Response contains: domain, new_backlinks, domain_rating, recent_content_urls. HTTP 200 for each call.

T04

Google Alerts RSS feeds are read and filtered correctly

Five configured Google Alert RSS feed URLs for test competitor brand names.

News items published since the previous Monday are collected; items older than 7 days are excluded.

Item list non-empty for at least one feed; date filter correctly excludes items outside the 7-day window; no RSS parse errors.

T05

SEMrush API returns HTTP 429 (rate limit hit)

Sandbox token configured to return 429 after the first call.

Agent applies exponential backoff, retries up to three times, then logs a structured error and halts gracefully without crashing the run.

Error event written to the automation platform log with code SEMRUSH_RATE_LIMIT; no unhandled exception; downstream agent is not triggered.

T06

Ahrefs API returns HTTP 401 (invalid or expired token)

Invalid Ahrefs sandbox token deliberately injected.

Agent logs AUTH_ERROR_AHREFS, skips Ahrefs data collection, continues with SEMrush and Google Alerts, and flags the missing Ahrefs section in the output JSON.

Output JSON includes ahrefs_status: 'error'; SEMrush and Google Alerts sections still populated; error logged to platform.

T07

Google Alerts RSS feed returns empty or malformed XML

One RSS URL returning empty feed; one returning malformed XML.

Empty feed: section recorded as empty with zero items. Malformed XML: parse error logged, item skipped, run continues.

No unhandled exception; malformed feed logged with code RSS_PARSE_ERROR; remaining feeds processed successfully.

T08

Competitor list in Google Sheets is empty

Test Google Sheet with the competitor list tab containing zero rows of competitor data.

Agent detects empty list, logs NO_COMPETITORS_FOUND, sends an alert to the configured ops notification channel, and exits without attempting API calls.

Log entry present with code NO_COMPETITORS_FOUND; no API calls made; ops alert sent to test Slack channel.

Intelligence Summariser Agent — Build complexity: Moderate. Estimated build time: 10 hours.

Scenario

Input

Expected Output

Pass Criteria

T09

Happy path: AI agent receives complete JSON dataset and produces a valid digest

Full structured JSON dataset from T01 (five competitors, all fields populated).

A formatted plain-English weekly digest with a highlights section and one section per competitor. Significant changes are flagged. Digest is correctly structured for Slack Block Kit formatting.

Digest contains highlights block and five competitor blocks; flagged items list is non-empty where input contained a delta above threshold; no template placeholders unfilled.

T10

AI agent receives a dataset where all competitors show zero change

JSON dataset with all delta fields set to zero and no new backlinks or news items for any competitor.

Digest is produced with a highlights section stating no significant changes detected this week. No false positives flagged.

Highlights block present; flagged_items list is empty; digest still posts to test Slack channel successfully; Google Sheets row appended with change_count: 0.

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Phase 2 tests verify that the two agents hand off correctly, that downstream tool writes (Slack, Google Sheets, HubSpot) behave as expected, and that the human review branch works. Edge-case tests then exercise the scenarios most likely to cause silent failures in a production environment.

Scenario

Input

Expected Output

Pass Criteria

T11

Data Collection Agent output is correctly received by Intelligence Summariser Agent

Successful T01 JSON dataset staged by the Data Collection Agent in the automation platform's internal data store.

Intelligence Summariser Agent fires automatically upon staging event; receives the full JSON; produces digest without manual intervention.

Summariser Agent execution log shows trigger source as DATA_COLLECTION_COMPLETE; input payload matches T01 schema; no data loss between agents.

T12

Digest posted to staging Slack channel in correct format

Formatted digest from T09 passed to the Slack delivery step.

Slack message appears in the staging channel with highlights in the main message body and competitor sections in a thread. No raw JSON or unformatted text visible.

Message visible in staging channel within 10 seconds of delivery step execution; Block Kit structure renders correctly; thread contains five competitor sections.

T13

Google Sheets tracker row appended correctly

Digest run with five competitors; staging Google Sheet used.

A new dated row is appended for each competitor with columns: date, competitor_name, keyword_delta, new_backlinks, flagged (true/false), digest_link.

Five new rows present in the test sheet; date matches run date; no existing rows overwritten; all required columns populated.

T14

Significant change flagged: human review branch fires correctly

Input JSON where one competitor has a keyword_delta exceeding the flagging threshold (defined as greater than 15% in either direction).

The decision node routes to the human review branch; a Slack DM is sent to the Marketing Manager's test user account with the flagged competitor detail and a prompt to review.

Slack DM received by test user; message includes competitor name and change detail; main digest post still proceeds in parallel; no step skipped.

T15

HubSpot note logged against the correct record

Digest run complete; HubSpot sandbox environment active; target company record ID pre-configured in the automation.

A note is created on the target HubSpot company record summarising the week's competitive landscape; note body matches the digest highlights section.

Note visible on the HubSpot sandbox company record; timestamp matches run date; note body contains at least the highlights text; HTTP 201 from HubSpot API.

Edge-case test cases — the scenarios below simulate conditions that are likely to occur in real operation and must be handled gracefully without data loss or silent failure.

Scenario

Input

Expected Output

Pass Criteria

E01

Duplicate competitor entry in the Google Sheets source list

Test competitor list contains the same domain entered twice on separate rows.

Agent deduplicates by domain before making API calls; only one API call is made per domain; digest contains one section per unique competitor.

API call log shows no duplicate domain calls; digest contains five unique sections (not six); deduplication event logged with code DUPLICATE_COMPETITOR_SKIPPED.

E02

Missing required field in SEMrush API response

Sandbox SEMrush response with the traffic_estimate field absent from one competitor's payload.

Agent substitutes a null marker for the missing field, flags it in the output JSON as data_quality: 'incomplete', and continues processing remaining competitors.

Output JSON contains data_quality: 'incomplete' for the affected competitor; field is null rather than absent; digest notes data gap for that competitor; no crash.

E03

Ahrefs API is entirely unavailable (connection timeout)

Ahrefs endpoint returns a connection timeout after 30 seconds.

Agent retries twice with a 15-second delay, then logs AHREFS_TIMEOUT, marks ahrefs_status: 'unavailable' in the output JSON, and proceeds with SEMrush and Google Alerts data only.

Total retry time does not exceed 90 seconds; output JSON present with ahrefs_status: 'unavailable'; digest is produced and delivered; error event logged.

E04

Slack webhook delivery fails (channel archived or webhook revoked)

Staging Slack webhook URL returns HTTP 404.

Automation logs SLACK_DELIVERY_FAILED, retries once after 60 seconds, then falls back to sending the digest as a plain-text email to the configured fallback address, and logs the failure for ops review.

Fallback email delivered to test fallback address; SLACK_DELIVERY_FAILED logged; Google Sheets append still completes; HubSpot note still logged.

E05

Google Sheets API returns HTTP 503 (service temporarily unavailable)

Google Sheets write step encounters a 503 response.

Agent retries the write up to three times with exponential backoff (15s, 30s, 60s). If all retries fail, it logs SHEETS_WRITE_FAILED and queues the row data for a manual retry prompt sent to the ops notification channel.

Retry log shows three attempts with correct intervals; if all fail, SHEETS_WRITE_FAILED logged and ops notification sent; Slack delivery and HubSpot note not blocked.

E06

Human review step not actioned within the defined timeout window

Significant change flagged and Slack DM sent to Marketing Manager test user; DM left unacknowledged for 24 hours (simulated by timeout configuration in the automation platform).

After 24 hours, the automation escalates with a second Slack DM to the Marketing Manager and an additional DM to the Operations Lead. The digest is posted to the main Slack channel regardless, with a note that a flagged item is awaiting human review.

Second DM sent to Marketing Manager test user after timeout; DM also sent to Operations Lead test user; main digest posted with flagged_awaiting_review: true; escalation event logged.

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: Use an internal test competitor entry (a dummy domain controlled by your team, not a real rival) to avoid polluting the live digest with test data. Direct the Phase 3 Slack output to a private internal test channel, not the main marketing channel. After the run passes all test cases, void the test competitor row in the Google Sheets tracker and delete the test HubSpot note before the automation goes into regular operation. Do not use a real competitor domain or a real team-facing Slack channel for this run.

Phase 3 runs a single complete cycle from the Monday 07:00 schedule trigger through to a verified HubSpot note. Every step is traced in the automation platform's execution log. The FullSpec team monitors the run in real time and records the result of each case below. The final case, T19, requires the process owner (Marketing Manager) to review the live digest output and record written approval, which is then stored in the FullSpec confirmation log for this project.

Scenario

Input

Expected Output

Pass Criteria

T16

Full production run: schedule trigger fires and both agents complete successfully

Production Monday 07:00 schedule trigger; live SEMrush API key; live Ahrefs API key; live Google Alerts RSS feeds for the test competitor domain; production Google Sheets tracker (test row); production HubSpot account (test company record).

Data Collection Agent completes and stages a valid JSON dataset within 90 seconds of trigger. Intelligence Summariser Agent fires automatically, produces a formatted digest, and the full workflow completes (Slack post, Sheets append, HubSpot note) within 5 minutes of trigger.

Execution log shows both agents as COMPLETED with no errors; total run time under 5 minutes; all four downstream actions (flag check, Sheets append, Slack post, HubSpot note) confirmed in the log.

T17

Error log verification: no unexpected errors present in the production run log

Execution log from the T16 run.

Log contains only INFO and SUCCESS level entries. No WARN, ERROR, or unhandled exception entries present. Any expected retry events (if a transient issue occurred) are clearly labelled and resolved.

Zero ERROR-level log entries; zero unhandled exceptions; if any WARN entries are present, each has a documented resolution in the FullSpec run notes.

T18

Production outputs verified: Slack message, Sheets row, and HubSpot note all correct

Live Slack channel (test channel), live Google Sheets tracker (test row), live HubSpot test company record after the T16 run.

Slack: digest message visible in the test channel with highlights and one competitor section for the test entry. Sheets: one new row appended with correct date, competitor name, and metric values. HubSpot: one new note on the test company record with the digest highlights text and the correct timestamp.

All three outputs confirmed by visual inspection and screenshotted by the FullSpec team. Sheets row date matches run date. HubSpot note timestamp within 5 minutes of trigger time. Slack message renders without formatting errors.

T19

Process owner review and approval: Marketing Manager confirms the production digest meets the agreed standard

Live digest output from T16 presented to the Marketing Manager (process owner) via the FullSpec project confirmation link.

Marketing Manager reviews the digest format, content structure, flagging logic, and downstream outputs (Sheets and HubSpot). Confirms in writing that the output meets the agreed weekly digest standard and that the automation is approved for live operation.

Written approval recorded in the FullSpec confirmation log with the process owner's name, date, and explicit approval statement. No outstanding issues raised. FullSpec marks Phase 3 as PASSED and closes the QA programme. Test competitor row voided in Sheets; test HubSpot note deleted.

Once T19 is recorded as passed in the FullSpec confirmation log, the automation is cleared for live weekly operation beginning on the next scheduled Monday trigger. Any issues raised during the process owner review in T19 are resolved by the FullSpec team before the pass status is recorded. Contact the FullSpec team at support@gofullspec.com for any queries during the testing programme.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process