Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Email Newsletter Production

[YourCompany.com] · Marketing Department · Prepared by FullSpec · [Today's Date]

This document defines the complete testing programme for the Email Newsletter Production automation, covering the Content Assembly Agent and the Reporting Agent. Testing runs in three sequential phases: unit tests in sandbox, integration and edge-case tests across agent handoffs, and a final end-to-end production run. No phase begins until the previous phase fully passes. The FullSpec team runs Phases 1 and 2 independently; the process owner joins Phase 3 to confirm outputs and record approval. Use this document alongside the Developer Handover Pack and the Integration and API Spec.

01Testing approach

All testing follows three phases in strict sequence. No phase begins until every test case in the previous phase carries a passing status. Phase 1 isolates each agent and each tool connection individually. Phase 2 validates handoffs between agents and the orchestration layer, and stress-tests edge conditions. Phase 3 runs one full production cycle end to end in the live environment with real credentials and real data, with the process owner present to review and approve.

Phase 1: Unit Testing

Scope: individual agents and tool connections in isolation

Scope

Each agent function and each API connection tested in isolation. Content Assembly Agent and Reporting Agent are tested independently.

Environment

Sandbox only. No production Mailchimp campaigns, no live Notion workspace writes, no real Slack channels.

Credential rule

Use sandbox API keys for Mailchimp, a test Notion workspace, a dedicated test Slack workspace, and a Google Docs test folder. No production credentials permitted in Phase 1.

Pass condition

All cases T01 through T14 return expected outputs with no unhandled exceptions.

Who runs it

FullSpec team only.

Phase 2: Integration and Edge-Case Testing

Scope: agent-to-agent handoffs and failure-mode resilience

Scope

End-to-end data flow between agents and the orchestration layer, including Slack approval gate behaviour, Mailchimp draft creation pipeline, and Reporting Agent trigger logic. Plus a dedicated edge-case table covering duplicates, missing data, unavailable tools, and human timeout scenarios.

Environment

Sandbox for edge-case destructive tests; a staging Mailchimp account with a test list of internal addresses for integration flow tests.

Credential rule

Staging Mailchimp API key with campaign-write scope. All other integrations remain on sandbox credentials. No write operations to the production Notion workspace.

Pass condition

All cases T15 through T22 pass and all edge cases E01 through E07 resolve with the correct fallback behaviour documented.

Who runs it

FullSpec team only.

Phase 3: End-to-End Production Test

Scope: full live cycle with process owner review and approval

Scope

One complete newsletter production cycle run against the live Mailchimp account, live Notion workspace, live Slack channels, and live Google Analytics property. All steps from weekly trigger through Notion performance card are traced.

Environment

Production environment. Test send targets an internal-only subscriber segment containing only team addresses.

Credential rule

Production API keys and OAuth tokens used. Mailchimp campaign must be voided or deleted after the test is complete. No send to the live subscriber list.

Pass condition

All cases T23 through T27 pass, including process owner sign-off recorded in the FullSpec confirmation log.

Who runs it

FullSpec team, with the process owner present and actively reviewing Phase 3 outputs.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only: all Phase 1 tests must use non-production API keys and test workspaces. Do not connect to the live Mailchimp account, the production Notion workspace, or any real Slack channel during this phase. If a test requires a live URL check, use a controlled set of known-good and known-broken URLs in the test environment.

Content Assembly Agent: the following cases cover the full range of inputs the agent may receive during a normal production cycle, plus the key failure modes for each connected tool.

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: all sections complete and on-brief

Google Doc with all required sections at correct word counts; Notion tasks all marked complete

Fully populated Mailchimp campaign draft with correct template variables; no flags raised

Campaign draft created via Mailchimp API; all variable fields populated; no error log entries

T02

One contributor section missing from Google Doc

Google Doc with one section blank; Notion task for that contributor still open

Agent flags the missing section; draft build is paused; Slack alert sent to coordinator

Slack message received in test channel citing the missing section by name; draft not created in Mailchimp

T03

One section over the maximum word count

Google Doc section containing 1.4x the configured word-count limit

Agent flags the section as off-brief with current and target word counts; pauses that section's injection

Flag appears in the agent's output log; coordinator Slack alert includes word-count delta; other sections continue to inject

T04

Google Docs API read fails (timeout)

Simulated API timeout on the Docs read request

Agent retries up to three times with exponential backoff; raises an error event after third failure

Three retry attempts logged with timestamps; error event raised; no partial draft created in Mailchimp

T05

Mailchimp template variable mismatch

Copy payload contains a field key not present in the Mailchimp template variable map

Agent logs an unmatched field warning; skips the unrecognised variable; all matched variables inject correctly

Warning logged for unmatched key; all correctly named variables appear in the Mailchimp draft; no unhandled exception

T06

Notion task status read fails

Simulated Notion API 503 on task status query

Agent falls back to the copy deadline timestamp to determine whether to proceed

Agent log shows fallback trigger used; build proceeds or holds based on deadline logic; no unhandled exception

T07

Mailchimp API campaign create returns 400

Malformed campaign payload sent to Mailchimp API

Error captured; descriptive log entry written; no retry; alert raised

400 error captured in log with Mailchimp error code and message; no retry loop; alert sent to test Slack channel

T08

Link check: all links valid

Draft campaign containing five known-good URLs

All five URLs return HTTP 200; no broken-link flag raised; workflow continues to Slack approval step

Link-check log shows five passes; no Slack broken-link alert; workflow advances to approval notification

T09

Link check: one broken link detected

Draft campaign containing four valid URLs and one URL returning HTTP 404

Broken link flagged in coordinator Slack alert with the specific URL and anchor text; approval step paused

Slack alert delivered to test channel citing the 404 URL; approval notification not sent until flag is resolved

T10

Link check: one URL redirects (301)

Draft containing a URL that redirects to a different domain

Redirect flagged as a warning (not a hard stop); coordinator notified; workflow continues to approval

Warning logged and included in Slack message; campaign draft not blocked; approval notification delivered

Reporting Agent: the following cases test the 48-hour post-send trigger, the Mailchimp and Google Analytics data pulls, and the Notion card write.

Scenario

Input

Expected Output

Pass Criteria

T11

Happy path: campaign metrics available at 48-hour mark

Simulated Mailchimp campaign with open rate, click rate, and link click data available; matching GA session data present

Notion performance card created with open rate, click rate, top three links by clicks, and no bounce flag

Notion card exists in the test workspace; all five data fields populated; bounce flag absent; card created within 5 minutes of trigger

T12

Bounce rate exceeds four-week average

Campaign bounce rate set to 1.8x the rolling four-week average in test data

Notion card created with all standard fields plus a bounce-flag marker

Bounce flag field is set to true on the Notion card; Slack alert sent to test channel citing the elevated rate

T13

Mailchimp metrics not yet available at 48-hour trigger

Simulated Mailchimp API returning empty metrics object at trigger time

Agent retries after 30 minutes; if still empty after two retries, writes a partial card with a data-pending flag

Retry attempts logged; partial Notion card created after second retry failure; data-pending flag set; no unhandled exception

T14

Google Analytics session data unavailable

GA Data API returns no matching sessions for the campaign UTM source

Reporting Agent writes the Notion card with Mailchimp data only; GA fields marked as unavailable; no error thrown

Notion card present with Mailchimp fields populated; GA fields show a clear unavailable status; agent log records the GA miss

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Integration tests verify that data flows correctly across agent boundaries and that the orchestration layer routes events as specified. All cases continue the sequential ID from Phase 1.

Scenario

Input

Expected Output

Pass Criteria

T15

Weekly schedule trigger fires and Slack briefs are dispatched

Scheduled trigger event fired in staging environment on the configured day and time

Personalised Slack brief messages sent to each contributor entry in the test workspace; Notion brief template read successfully

Each configured contributor receives a distinct Slack message with their section name, word count, and deadline; Notion API read logged with 200 status

T16

Content Assembly Agent receives validated payload from orchestration layer

Orchestration layer passes assembled Google Docs content payload to agent after all Notion tasks marked complete

Agent accepts payload, validates structure, and begins template variable injection without error

Agent input log shows correctly structured payload; validation step completes with no schema errors; injection begins within 10 seconds

T17

Mailchimp draft created and link check triggered in sequence

Content Assembly Agent completes injection; orchestration layer triggers link-check step

Link check runs against the newly created Mailchimp draft ID; results returned to orchestration layer

Link-check step receives the correct campaign ID from the previous step; check completes; result passed to decision node

T18

Slack approval notification delivered with correct preview link

Link check passes (no broken links); orchestration layer triggers approval notification step

Coordinator receives Slack message in the test channel with a valid Mailchimp preview link and approve/request-edits prompt

Slack message received; preview link resolves to the correct staging Mailchimp draft; prompt options present

T19

Coordinator approves via Slack; Mailchimp schedule is set

Coordinator selects approve in the staging Slack prompt

Orchestration layer calls Mailchimp schedule-campaign endpoint with the configured send time

Mailchimp campaign status changes to scheduled in the staging account; send time matches configured value; confirmation log entry written

T20

Coordinator selects request-edits in Slack; workflow pauses correctly

Coordinator selects request-edits option in the staging Slack prompt

Workflow halts at approval gate; notification sent back to coordinator confirming the draft is on hold; no send scheduled

No Mailchimp schedule API call made; hold-state log entry written; coordinator receives hold-confirmation Slack message

T21

Reporting Agent trigger fires 48 hours after simulated send event

Mailchimp campaign status set to sent in staging environment; 48-hour offset trigger configured

Reporting Agent activates at the correct offset; queries Mailchimp and GA APIs with the correct campaign ID

Agent activation timestamp within 2 minutes of the 48-hour offset; both API calls include the correct campaign ID and date range parameters

T22

Notion performance card written and readable by team workspace

Reporting Agent completes data aggregation; orchestration layer calls Notion API write

Notion card created in the correct database with all required fields; card is visible and correctly formatted in the staging workspace

Notion API returns 200 on card create; all five data fields present; card appears in the staging performance database view

Edge-case tests: the following table covers conditions outside the happy path that the automation must handle gracefully without data loss, unintended sends, or unhandled exceptions.

Scenario

Input

Expected Output

Pass Criteria

E01

Duplicate weekly trigger fires twice in the same cycle

Trigger event fires a second time within the same production window (e.g. misconfigured schedule)

Orchestration layer detects an active cycle for the current week and suppresses the duplicate run; alert sent to FullSpec monitoring

Second run does not create a second Mailchimp draft or send a second set of Slack briefs; deduplication log entry written

E02

All contributor Notion tasks missing (no tasks found in the board)

Notion task query returns an empty result set for the current cycle

Agent falls back to the copy deadline trigger; Slack alert sent to coordinator warning that no tasks were found

Deadline-based fallback logged; coordinator alert delivered; build proceeds or holds according to configured fallback logic

E03

Google Doc content is present but the document is empty (zero characters)

Google Docs API returns a document with no body text

Agent treats this as a missing-section condition; raises a flag; does not attempt Mailchimp injection

Flag logged; Slack alert sent citing the empty document URL; no partial Mailchimp draft created

E04

Mailchimp API is unavailable for more than 10 minutes during draft creation

Simulated Mailchimp API returning 503 for 10+ minutes during the campaign-create step

Agent retries with backoff for up to 10 minutes; after timeout, raises a critical error event and halts the cycle; coordinator and FullSpec monitoring alerted

Retry log shows attempts with timestamps; critical error event raised after timeout; no partial draft left in Mailchimp; both alert destinations receive notification

E05

Slack API unavailable when approval notification is due

Simulated Slack API 503 when the approval message is dispatched

Orchestration layer retries Slack delivery three times; if all fail, falls back to sending an email to the coordinator's configured fallback address; workflow holds

Three Slack retry attempts logged; fallback email sent to the configured address; workflow does not auto-approve or auto-cancel; hold state maintained

E06

Coordinator does not respond to the approval Slack prompt within the configured timeout window

Approval prompt sent; no response received within the configured 4-hour window

Orchestration layer escalates: sends a second Slack message tagged to the marketing manager; send is NOT scheduled automatically

Escalation message delivered to the marketing manager's Slack handle; no Mailchimp schedule API call made; escalation event logged with timestamp

E07

Reporting Agent fires for a campaign that was cancelled before send

Mailchimp campaign status is cancelled rather than sent at the 48-hour trigger point

Reporting Agent detects non-sent status; skips metric pull; writes a cancelled-campaign note to the Notion card; no error raised

Notion card created with cancelled status noted; no Mailchimp metrics API call made; agent log records the status check outcome

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: all sends during this phase must target an internal-only subscriber segment containing only team email addresses. Do not include any real subscribers in the test send. After the test is complete, void or delete the Mailchimp campaign and remove the test Notion card from the live workspace. Confirm these cleanup steps are complete before the first real production cycle begins. Any production API keys used during Phase 3 must be the same keys that will run in live operation; do not use placeholder values.

The following test cases run one complete newsletter cycle against the live environment, tracing every step from the weekly trigger through to the Notion performance card. The process owner must be present and actively reviewing outputs from T25 onward. The final case, T27, records the process owner's approval and is the formal gate before go-live.

Scenario

Steps Covered

Expected Output

Pass Criteria

T23

Full production cycle: trigger through Mailchimp draft creation

Weekly trigger fires; Slack briefs sent to real contributor Slack handles; Google Docs copy aggregated; Content Assembly Agent validates and injects into Mailchimp via production API

Real contributor Slack messages received; Google Docs content pulled correctly; Mailchimp campaign draft created in the live account with all template variables populated; link-check report generated

Contributor Slack messages received by the correct users; Mailchimp draft visible in the live account with all variables correct; link-check log shows results for every URL in the draft; no unhandled exceptions in any step

T24

Error log verification across all production steps

Inspect the orchestration layer's run log for the T23 cycle

Run log shows a clean execution trace with no warnings or errors; all step timestamps are present; API response codes are all 2xx

Log exported and reviewed by FullSpec; zero error-level entries; all integration call response codes confirmed 2xx; log retained for the project record

T25

Slack approval notification delivered in production; coordinator reviews draft

Approval Slack message sent to the coordinator's live Slack account with the production Mailchimp preview link

Coordinator receives the message, opens the preview link, and confirms the draft renders correctly in Mailchimp's preview pane

Coordinator confirms receipt and correct rendering; preview link resolves to the correct draft; no rendering errors reported by the coordinator

T26

Reporting Agent fires against production Mailchimp and GA data; Notion card written

Test send dispatched to the internal-only segment; 48-hour trigger offset simulated; Reporting Agent pulls live Mailchimp and GA API data

Notion performance card created in the live workspace with open rate, click rate, top three links, and bounce flag assessment

Notion card visible in the live performance database; all five data fields populated with values matching the Mailchimp and GA dashboards; card created within 5 minutes of the trigger offset

T27

Process owner review and approval: go-live gate

Process owner reviews all Phase 3 outputs: Mailchimp draft quality, Slack approval flow, Notion performance card accuracy, and error log

Process owner confirms all outputs meet the agreed acceptance criteria and records approval in the FullSpec confirmation log; FullSpec marks the automation as cleared for live operation

Process owner sign-off recorded in the FullSpec confirmation log. No separate signoff block is used. FullSpec team marks Phase 3 complete and schedules the first live production cycle.

Post-Phase 3 cleanup checklist before go-live: delete or archive the test Mailchimp campaign; remove the test Notion performance card from the live database; confirm no test emails were delivered outside the internal segment; rotate any API keys that were shared in plain text during testing; and verify the live production schedule trigger is set to the correct recurring day and time.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process