Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Knowledge Base Maintenance

[YourCompany.com] · Customer Support Department · Prepared by FullSpec · [Today's Date]

This document defines the complete test and quality assurance strategy for the Knowledge Base Maintenance automation, covering all three agents: Ticket Trend Analyst, Article Draft Writer, and Publish and Audit Agent. Testing is structured across three sequential phases. No phase begins until the previous phase has fully passed. The FullSpec team executes Phases 1 and 2 in sandbox environments; the process owner joins the FullSpec team for Phase 3 production validation. All test case IDs are sequential across the full document. Edge cases are prefixed with E.

01Testing approach

Testing runs in three mandatory phases in sequence. No phase begins until the previous phase fully passes. Each phase targets a different layer of the system: isolated agent logic, cross-agent integration and edge cases, and finally a live end-to-end production run. This sequencing prevents integration failures from obscuring unit-level bugs and keeps production exposure to a single controlled run.

Phase 1: Unit Testing

Isolated agent logic in sandbox

Scope

Each agent tested independently: trigger logic, data parsing, output format, and tool calls verified in isolation.

Environment

Sandbox only. No production helpdesk, live help centre, or active Notion workspace is touched.

Credential rule

Sandbox API keys only. No production tokens or OAuth grants permitted during Phase 1.

Pass condition

All test cases T01 through T15 return the expected output with no errors. Any failure blocks progression to Phase 2.

Who runs it

FullSpec team only.

Phase 2: Integration and Edge Case Testing

Agent handoffs and failure scenarios in sandbox

Scope

Agent-to-agent handoff integrity, data passing between Zendesk, Google Docs, Notion, Intercom, and Slack. Edge cases covering duplicates, missing data, tool unavailability, and human timeout.

Environment

Sandbox with stubbed tool responses for unavailability scenarios. Sandbox OAuth credentials throughout.

Credential rule

Sandbox credentials only. Stubbed webhook endpoints used to simulate Intercom publish and Slack post where rate limits apply.

Pass condition

All integration cases T16 through T24 pass and all edge cases E01 through E08 produce the defined safe outcome. Any failure blocks Phase 3.

Who runs it

FullSpec team only.

Phase 3: End-to-End Production Test

Full pipeline on live systems with process owner present

Scope

One complete pipeline run from ticket cluster detection through to published Intercom article, completed Notion record, and Slack confirmation. Error log verification included.

Environment

Production. Live Zendesk, Intercom, Google Docs, Notion, and Slack connections using scoped production credentials.

Credential rule

Production credentials active. Intercom write scope restricted to update-existing-article only. No create or delete permissions granted.

Pass condition

All cases T25 through T28 pass, including process owner approval sign-off recorded in the FullSpec confirmation log.

Who runs it

FullSpec team, with the process owner present and actively completing the manager approval step.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only during Phase 1. Do not connect production Zendesk, Intercom, Google Docs, Notion, or Slack tokens to any agent under test in this phase. Use dedicated test workspace credentials and sandbox API keys issued during the Discovery and Tool Access stage.

Ticket Trend Analyst — Build time estimate: 8 hours. Complexity: Moderate.

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: cluster threshold met

Sandbox Zendesk returns 6 tickets tagged 'password reset' within a 7-day window

Agent produces a scored article list with at least 1 flagged article and a cluster summary

Output JSON contains article_id, staleness_score >= 0.6, and ticket_count = 6

T02

Threshold not met: fewer than 5 tickets

Sandbox Zendesk returns 4 tickets on the same topic within 7 days

Agent produces no flagged articles and exits cleanly with a 'below threshold' status

Workflow terminates after polling step; no downstream agent is triggered

T03

Multiple clusters in one poll

Sandbox Zendesk returns 7 tickets on 'billing error' and 5 tickets on 'export CSV' in the same window

Agent produces two separate scored article entries, one per topic cluster

Output contains exactly 2 article records, each with a distinct topic label and staleness score

T04

Zendesk API returns 401 Unauthorized

Invalid or expired sandbox API key passed to Zendesk polling step

Agent logs a credential error, halts the run, and emits an error event to the orchestration layer

No article list generated; error event payload includes error_code: 401 and source: zendesk

T05

Zendesk API returns empty ticket array

Sandbox Zendesk returns an empty results array for the polling window

Agent exits cleanly with a 'no tickets found' status and does not proceed

Workflow terminates; no error logged; status field equals 'no_data'

T06

Ticket text contains no identifiable topic keywords

6 tickets with only numeric reference codes and no body text in the sandbox payload

Agent flags the cluster as 'low confidence', assigns a staleness score below 0.4, and does not trigger the draft agent

Output contains confidence_flag: 'low' and downstream trigger is suppressed

Article Draft Writer — Build time estimate: 8 hours. Complexity: Moderate.

Scenario

Input

Expected Output

Pass Criteria

T07

Happy path: article draft created

Scored article record from T01 passed to agent; sandbox Google Docs OAuth active

A new Google Doc is created containing the existing article structure with tracked-change suggestions and a change summary header

Google Doc URL present in output; doc contains at least one tracked change; Notion row created with article title and doc link

T08

Notion row created with correct fields

Same input as T07

Notion database row contains: article title, ticket count, Google Doc link, suggested priority, and assigned reviewer field populated

All five fields present and non-null in the Notion API response

T09

Source article body is empty in Intercom

Flagged article ID resolves to an Intercom article with a blank body field

Agent logs a warning, creates a Google Doc with a placeholder structure and a note that the source article was empty, and still creates the Notion row

Google Doc created; doc body contains warning string 'source article body was empty'; Notion row created

T10

Google Docs OAuth token rejected

Agent attempts to create a Google Doc using an expired sandbox token

Agent logs an auth error, halts doc creation, marks the Notion row with status 'draft_failed', and emits an error event

No Google Doc created; Notion row status equals 'draft_failed'; error event logged

T11

Notion API rate limit exceeded

Sandbox Notion stub returns HTTP 429 on row creation attempt

Agent retries up to 3 times with exponential backoff; if all retries fail, logs a row-creation error and alerts via the orchestration layer

Retry count equals 3 in logs; after final failure, error event emitted with source: notion and code: 429

T12

Ticket text is very short (under 20 characters per ticket)

6 tickets each with a body of fewer than 20 characters in the sandbox payload

Agent generates a draft with a low-confidence flag in the doc header and a note recommending manual review

Google Doc contains the string 'low confidence draft' in the header section

Publish and Audit Agent — Build time estimate: 6 hours. Complexity: Moderate.

Scenario

Input

Expected Output

Pass Criteria

T13

Happy path: approval detected and article published

Manager adds a comment containing the string 'Approved' to the sandbox Google Doc; Intercom sandbox publish endpoint active

Agent publishes updated article body to sandbox Intercom, marks Notion row as 'complete', and posts a Slack confirmation to the test channel

Intercom API returns 200; Notion row status equals 'complete'; Slack message received in test channel within 60 seconds

T14

Rejection comment: manager requests revision

Manager adds a comment containing 'Needs revision' to the sandbox Google Doc

Agent does not publish; does not change Notion row status; no Slack message sent

Intercom publish endpoint not called; Notion row status unchanged; no Slack message in test channel

T15

Intercom publish API returns 403 Forbidden

Approval detected but sandbox Intercom stub returns 403 on PUT request

Agent logs a publish error, does not update Notion row to 'complete', and emits an error event with source: intercom and code: 403

Notion row status remains 'approved_pending_publish'; error event payload matches expected schema

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Phase 2 verifies that data passes correctly between all three agents and that the supporting tool connections behave as expected across the full chain. These tests run in the sandbox environment with all three agents active simultaneously. Integration test IDs continue the T sequence from T16.

Scenario

Input

Expected Output

Pass Criteria

T16

Ticket Trend Analyst output triggers Article Draft Writer correctly

Sandbox Zendesk returns a valid cluster; Ticket Trend Analyst completes successfully

Article Draft Writer receives the full scored article record including article_id, staleness_score, ticket_count, and topic_label within 30 seconds

Article Draft Writer input payload matches Ticket Trend Analyst output schema exactly; no fields missing

T17

Article Draft Writer output triggers Publish and Audit Agent listener correctly

Google Doc created by Article Draft Writer; sandbox approval comment added

Publish and Audit Agent detects the approval comment on the correct document within 2 minutes of it being posted

Agent poll log shows the correct doc_id matched; approval detection latency under 120 seconds

T18

Full three-agent handoff chain completes without manual intervention

Sandbox end-to-end: Zendesk cluster threshold met, Article Draft Writer runs, approval added to Google Doc

All three agents complete in sequence; Intercom sandbox article updated; Notion row marked complete; Slack message posted

All output records present and correct; total pipeline runtime under 10 minutes in sandbox

T19

Slack notification contains correct article metadata

Publish and Audit Agent triggers Slack post after publish

Slack message includes article title, Google Doc link, ticket count that triggered the review, and the name of the assigned reviewer

All four fields present in the Slack message body; link resolves to the correct sandbox Google Doc

T20

Notion tracker reflects correct status at each pipeline stage

Full pipeline run in sandbox

Notion row status transitions: 'pending' on creation, 'in_review' after Slack alert, 'approved_pending_publish' after approval, 'complete' after publish

Each status value is logged at the correct pipeline stage; no status is skipped

T21

Two simultaneous clusters processed independently

Sandbox Zendesk returns two separate clusters triggering at the same time

Two independent pipeline runs execute concurrently; each produces its own Google Doc, Notion row, and Slack alert without data cross-contamination

Two distinct doc_ids, two distinct Notion row_ids, and two Slack messages posted; no field values mixed between runs

T22

Ticket Trend Analyst output schema is malformed

Ticket Trend Analyst emits a payload missing the article_id field due to a simulated parsing error

Article Draft Writer detects the missing required field, halts, and emits a schema validation error to the orchestration layer

Article Draft Writer does not proceed; error event contains field_missing: article_id

T23

Slack API unavailable during notification step

Sandbox Slack stub returns HTTP 503 at the point of posting the review alert

Agent retries the Slack post up to 3 times; if all retries fail, the pipeline continues and logs a non-fatal Slack error; Notion row and Google Doc are unaffected

Notion row and Google Doc created successfully; Slack error logged as non-fatal; retry count equals 3

T24

Manager approval comment uses a non-standard approval string

Manager types 'LGTM' instead of 'Approved' in the Google Doc comment

Publish and Audit Agent does not detect this as an approval; article is not published; Notion row status unchanged

No Intercom publish call made; Notion status unchanged; no Slack completion message sent

Edge case tests cover scenarios outside the normal flow that the system must handle safely without data loss or unintended publishing.

Scenario

Input

Expected Output

Pass Criteria

E01

Duplicate cluster: same article flagged twice in the same polling window

Ticket Trend Analyst detects the same article_id in two separate cluster results within one daily poll

Agent deduplicates and produces only one flagged article record; only one Google Doc and one Notion row created

Exactly one doc_id and one Notion row_id for the article; deduplication log entry present

E02

Notion row already exists for the same article from a previous run

Article Draft Writer attempts to create a Notion row for an article_id that already has an open row with status 'in_review'

Agent skips row creation, links the existing Notion row to the new Google Doc, and logs a 'row already exists' event

No duplicate Notion row created; existing row updated with new doc_id; log entry confirms skip

E03

Missing article body in source: Intercom returns null body for flagged article

Article Draft Writer fetches article content from sandbox Intercom and receives a null body field

Agent creates a Google Doc with a placeholder template and a warning note; Notion row created with priority set to 'manual review required'

Google Doc created with warning header; Notion priority field equals 'manual_review_required'

E04

All three external tools unavailable simultaneously

Sandbox stubs for Zendesk, Google Docs, and Intercom all return 503 at pipeline start

Orchestration layer logs a system-wide unavailability error; no partial pipeline execution; no data written to Notion or Slack

Error event emitted with affected_services listing all three; no Notion rows or Google Docs created; Slack not called

E05

Manager does not approve or reject within 7 days

Google Doc created and Slack alert sent; no manager comment added within the timeout window

Orchestration layer sends a reminder Slack message at day 3 and day 7; after day 7 with no response, Notion row status set to 'stalled' and FullSpec error log records the timeout

Reminder messages sent at correct intervals; Notion row status equals 'stalled' after 7-day timeout

E06

Ticket cluster topic matches an article that was already updated within the last 24 hours

Article_id in the Ticket Trend Analyst output has a last_published timestamp within the previous 24-hour window

Agent suppresses the draft and does not trigger Article Draft Writer; logs a 'recently updated, skipping' event

Article Draft Writer not triggered; log entry contains reason: recently_updated and article_id

E07

Intercom article publish partially fails: body updated but metadata not saved

Sandbox Intercom stub returns 200 on PUT body but 500 on the metadata update call

Agent logs a partial publish error, marks Notion row as 'publish_partial_failure', and emits an alert; does not mark Notion as 'complete'

Notion row status equals 'publish_partial_failure'; error event includes failed_step: metadata_update

E08

Large cluster: 50 or more tickets in one topic within 7 days

Sandbox Zendesk returns 52 tickets on a single topic

Ticket Trend Analyst processes all 52 tickets, assigns maximum staleness score of 1.0, and sets priority to 'critical' in the output

Output staleness_score equals 1.0; priority field equals 'critical'; downstream agents proceed normally

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: Use an internal test support email address to generate or seed the trigger ticket cluster. Do not use a real customer ticket unless it is clearly marked as a test record in Zendesk. The Google Doc draft produced during this test must be voided after the run: delete or archive it from the shared Drive folder and remove the corresponding Notion row or mark it 'test_void'. The Intercom article updated during this test must be reverted to its pre-test content immediately after the run completes. Confirm all cleanup steps are completed before the pipeline is switched to live operation.

Phase 3 runs once, in production, with the process owner actively present. The FullSpec team leads the execution. The process owner completes the manager approval step inside Google Docs as they would in normal operation. All four test cases below must pass for go-live to be confirmed.

Scenario

Input

Expected Output

Pass Criteria

T25

Full production pipeline run traced end to end

A seeded ticket cluster of 5 or more tickets on a single internal test topic, entered into live Zendesk using the internal test address; all production credentials active

Ticket Trend Analyst detects the cluster on its next scheduled poll; Article Draft Writer creates a Google Doc in the production Drive folder and a Notion row in the live tracker; Slack alert posted to the support manager channel

All three outputs present within 15 minutes of cluster crossing the threshold; doc structure matches the approved template; Notion row fields all populated; Slack message received

T26

Manager approval in production triggers publish

Process owner adds an 'Approved' comment to the Google Doc produced in T25

Publish and Audit Agent detects the approval within 2 minutes; pushes updated content to the live Intercom help centre article; marks Notion row as 'complete'; posts Slack confirmation

Intercom article content matches the Google Doc body; Notion status equals 'complete'; Slack confirmation message received; all actions logged in the orchestration layer audit trail

T27

Error log verification

Review orchestration layer run log for the Phase 3 pipeline execution

Run log shows zero unhandled exceptions; all API calls return 200 or 201; retry events, if any, are logged with resolution status

Log contains no unhandled_exception entries; all external API response codes in the accepted range; any retries show final_status: success

T28

Process owner review and approval of go-live readiness

Process owner reviews the complete Phase 3 run: the Google Doc draft quality, the Notion tracker entry, the Slack alert content, and the published Intercom article

Process owner confirms the pipeline output meets operational standards and gives explicit go-live approval; FullSpec team records the approval in the FullSpec confirmation log with the reviewer name, date, and a pass or fail verdict

FullSpec confirmation log entry created with reviewer_name, approval_date, verdict: pass, and a reference to this test case ID T28. This entry constitutes the formal process owner sign-off for production launch.

Once T28 is recorded as passed in the FullSpec confirmation log, the automation is cleared for live operation. Post-launch monitoring begins immediately. Contact the FullSpec team at support@gofullspec.com for any issues observed in the first two weeks of live operation.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process