Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

FAQ & First Response Automation

[YourCompany.com] · Customer Support Department · Prepared by FullSpec · [Today's Date]

This document defines the complete test and quality assurance programme for the FAQ and First Response Automation across three sequential phases: Unit, Integration, and End-to-End. No phase begins until the previous phase fully passes. The FullSpec team runs Phases 1 and 2 in sandbox environments; the process owner joins FullSpec for Phase 3 in production. Test case IDs are sequential across all phases (T01 onward) and edge cases are prefixed E01 onward. Any failure against a pass condition must be resolved and re-tested before progression. Contact the FullSpec team at support@gofullspec.com with any questions about this plan.

01Testing approach

Testing is structured across three phases run in strict sequence. Phase 1 validates each agent in isolation against known inputs and expected outputs. Phase 2 validates the handoffs between agents and stress-tests the system against edge cases including duplicate tickets, missing data, tool unavailability, and human timeout scenarios. Phase 3 confirms full end-to-end behaviour in the live production environment with the process owner present. No phase begins until every test case in the preceding phase carries a pass status.

Phase 1: Unit Testing

Each agent tested in isolation before integration begins

Scope

Individual agent logic: Ticket Classification Agent, First Response Drafting Agent, Escalation and Logging Agent — each tested independently against controlled inputs

Environment

Sandbox only. No connection to production Zendesk, HubSpot, Notion, Slack, or Gmail instances

Credential rule

Sandbox API keys and test account tokens exclusively. No production credentials are loaded or referenced during Phase 1

Pass condition

All cases T01 through T15 return expected outputs with zero unhandled exceptions. Confidence scoring deviations within ±5% of defined thresholds are acceptable; breaches beyond ±5% require agent re-tuning

Who runs it

FullSpec team only. Process owner is not required for Phase 1

Phase 2: Integration and Edge Case Testing

Agent handoffs and failure-mode coverage validated end to end in sandbox

Scope

Cross-agent handoffs (Classification to Drafting, Drafting to Escalation/Logging), plus dedicated edge case table covering duplicates, missing fields, tool outages, and manual review timeouts

Environment

Sandbox environment with mocked API responses for outage simulation. No production data is used

Credential rule

Sandbox credentials only. Mocked OAuth tokens used for Zendesk, HubSpot, Notion, Slack, and Gmail outage scenarios

Pass condition

All integration cases T16 through T23 and all edge cases E01 through E07 resolve with correct routing, no data loss, and appropriate error logging. Any failed handoff that silently drops a ticket is an automatic block

Who runs it

FullSpec team only. Process owner may observe but sign-off is not required at this phase

Phase 3: End-to-End Production Testing

Live environment confirmation with process owner sign-off

Scope

Full production workflow traced from Zendesk webhook trigger through classification, FAQ retrieval, reply send, HubSpot logging, and Slack escalation alert using real credentials and live tool instances

Environment

Production. All five integrated tools (Zendesk, Notion, HubSpot, Slack, Gmail) connected with live credentials. Test records must be voided immediately after each run

Credential rule

Production credentials active. The FullSpec team confirms each credential is scoped to the minimum required permissions before Phase 3 begins

Pass condition

All cases T24 through T27 pass including process owner review and approval captured in the FullSpec confirmation record

Who runs it

FullSpec team runs the automation; process owner is present to observe, validate outputs, and record approval for the final case

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only during Phase 1. No production API keys, no live Zendesk tickets, no real customer data. All inputs are synthetic test payloads constructed by the FullSpec team. If any test case requires a real email address, use the designated FullSpec sandbox address (sandbox-test@gofullspec.com) only.

Ticket Classification Agent — tests T01 through T06

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: clear FAQ ticket

Ticket body: 'What is your return policy?' — no ambiguity, standard phrasing

Category: FAQ; confidence score >= 0.85; routing destination: auto-reply queue

Category label equals FAQ; confidence field is numeric and >= 0.85; routing field populated

T02

Happy path: billing issue ticket

Ticket body: 'I was charged twice for my order last week'

Category: billing; confidence score >= 0.80; routed to billing queue; flagged for human review

Category label equals billing; human-review flag set to true; no auto-reply triggered

T03

Happy path: escalation ticket

Ticket body: 'This is unacceptable, I am contacting my bank and leaving a review'

Category: escalation; priority: urgent; confidence score >= 0.80; escalation flag set

Category equals escalation; priority field equals urgent; escalation flag is true

T04

Low confidence — ambiguous ticket

Ticket body: 'I need help' — no additional context

Category: unknown; confidence score < 0.65; ticket placed in human-review queue; no auto-reply

Confidence field < 0.65; human-review queue flag set; auto-reply action not triggered

T05

Failure mode: empty ticket body

Ticket body: '' (empty string); subject: 'Hello'

Classification skipped; error logged with ticket ID; ticket routed to human-review queue

Error log entry created with ticket ID and error code EMPTY_BODY; no classification label set; human-review flag true

T06

Failure mode: non-English ticket body

Ticket body in Spanish: 'Necesito ayuda con mi pedido'

Category: unknown; confidence score < 0.65; language flag set; routed to human-review

Language flag populated; confidence < 0.65; human-review flag true; no auto-reply

First Response Drafting Agent — tests T07 through T11

Scenario

Input

Expected Output

Pass Criteria

T07

Happy path: FAQ category triggers Notion lookup and draft

Classification output: category=FAQ, confidence=0.90; customer name: 'Alex'; Notion article: return-policy.md

Personalised reply draft containing customer name, accurate return policy text, no placeholder tokens

Draft contains 'Alex'; policy text matches Notion source article; no unfilled {{placeholder}} tokens present

T08

Happy path: multiple matching Notion articles

Ticket body references both return policy and shipping times; two Notion articles match

Draft combines both topics coherently; no duplicate sentences; tone matches brand guidelines

Both topics present in draft; character count between 80 and 400 words; no repeated sentences

T09

High-confidence output triggers Zendesk send action

Draft produced with confidence = 0.92; Zendesk sandbox send action enabled

Zendesk API call made to POST /tickets/{id}/comments with is_public=true; ticket tagged auto-replied

Zendesk sandbox receives the API call; response status 200; ticket tag auto-replied present

T10

Failure mode: no matching Notion article found

Classification: FAQ; Notion query returns zero results for ticket topic

Draft not produced; ticket re-routed to human-review queue; error logged as NO_NOTION_MATCH

Error log entry NO_NOTION_MATCH created; human-review flag set; no reply sent to customer

T11

Failure mode: Notion API returns 429 rate limit

Notion API responds with HTTP 429 during article fetch

Retry logic fires up to 3 times with exponential backoff; on third failure ticket queued for human review

Three retry attempts logged with timestamps; after third failure human-review flag set; no silent drop

Escalation and Logging Agent — tests T12 through T15

Scenario

Input

Expected Output

Pass Criteria

T12

Happy path: HubSpot logging after FAQ auto-reply

Ticket closed as auto-replied; customer email present in HubSpot sandbox contact

HubSpot activity note created on matched contact: ticket category, reply timestamp, Zendesk ticket ID

HubSpot sandbox shows new activity note; note body contains category, timestamp, and Zendesk ticket ID

T13

Happy path: Slack escalation alert for urgent ticket

Ticket category: escalation; priority: urgent; Slack sandbox channel: #support-escalations

Slack message posted with ticket summary, customer name, Zendesk ticket link, and @support-lead mention

Slack sandbox channel receives message; all four required fields present; @support-lead mention included

T14

Failure mode: customer email not found in HubSpot

Ticket customer email does not match any HubSpot contact

New HubSpot contact created with email and ticket note; deduplication flag logged for post-launch review

New contact record created in HubSpot sandbox; deduplication flag written to error log; no duplicate contacts

T15

Failure mode: Slack API unavailable

Slack API returns HTTP 503 during escalation post attempt

Retry up to 2 times; on failure log SLACK_SEND_FAILED; fallback Gmail alert sent to support-lead@[YourCompany.com]

SLACK_SEND_FAILED log entry present; Gmail fallback send confirmed in sandbox outbox; no silent drop

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Phase 2 validates the live handoffs between agents and confirms that data passes correctly across the full pipeline. All tests below run in the sandbox environment with no production credentials. IDs continue from T15.

Scenario

Agents Involved

Expected Handoff Output

Pass Criteria

T16

Classification Agent passes FAQ label to Drafting Agent

Ticket Classification Agent -> First Response Drafting Agent

Drafting Agent receives structured payload: {category, confidence, ticket_id, customer_email, customer_name}; begins Notion fetch

Drafting Agent log shows payload received with all five required fields; Notion fetch initiated within 2 seconds of handoff

T17

Drafting Agent passes completed reply to Zendesk send and then triggers Logging Agent

First Response Drafting Agent -> Zendesk -> Escalation and Logging Agent

Zendesk API confirms reply sent (status 200); Logging Agent receives trigger with ticket_id and reply_timestamp

Zendesk sandbox shows reply; Logging Agent log shows trigger received with correct ticket_id and timestamp

T18

Low-confidence ticket bypasses Drafting Agent and routes directly to Logging Agent via human queue

Ticket Classification Agent -> Zendesk (human queue) -> Escalation and Logging Agent

Ticket placed in human-review queue in Zendesk; Logging Agent logs ticket as human-routed with no reply sent

Zendesk shows ticket in human-review queue; HubSpot note category = human-review; no Zendesk auto-reply present

T19

Escalation ticket triggers both HubSpot log and Slack alert in correct order

Escalation and Logging Agent: HubSpot log then Slack alert

HubSpot note created first; Slack alert posted second; both actions complete within 10 seconds of trigger

HubSpot activity timestamp precedes Slack message timestamp; both present in sandbox within 10 seconds

T20

End-to-end happy path in sandbox: new FAQ ticket flows through all three agents without human input

All three agents in sequence

Ticket classified, Notion article retrieved, reply drafted and sent, HubSpot logged, no Slack alert (non-escalation)

All pipeline steps logged; reply in Zendesk sandbox; HubSpot note present; Slack channel empty for this ticket

T21

Drafting Agent output rejected by confidence gate — handoff to human queue instead of Zendesk send

First Response Drafting Agent -> confidence gate -> Zendesk human queue

Draft produced but confidence below send threshold; draft saved as internal note; ticket flagged for human review

Zendesk sandbox shows ticket as internal note only; public reply not sent; human-review flag true; draft content preserved

T22

Parallel processing of two simultaneous tickets — no cross-contamination of ticket data

All three agents, two concurrent ticket payloads

Each ticket maintains its own isolated context throughout pipeline; replies and logs reference correct ticket IDs

Ticket A reply contains only Ticket A customer name and content; Ticket B reply contains only Ticket B data; no field bleed

T23

Correct Zendesk tags and priority applied for all category types across a batch of five test tickets

Ticket Classification Agent -> Zendesk

FAQ tickets tagged auto-reply-faq; billing tickets tagged billing-human; escalation tickets tagged escalation-urgent

All five sandbox tickets carry the correct tag set; priority field matches category rules defined in the agent specification

Edge case tests — IDs E01 through E07

Edge Case Scenario

Condition Simulated

Expected Behaviour

Pass Criteria

E01

Duplicate ticket submission

Same customer submits identical ticket twice within 60 seconds; Zendesk assigns two different ticket IDs

Second ticket detected as probable duplicate; flagged in Zendesk with duplicate tag; Logging Agent logs both but Slack alert fires only once

Both tickets in Zendesk sandbox; second ticket carries duplicate tag; only one Slack alert in sandbox channel

E02

Missing customer name field in ticket payload

Zendesk webhook delivers ticket with empty display_name field

Drafting Agent uses fallback salutation 'Hello' instead of customer name; reply still sent; no agent crash

Draft contains 'Hello' as salutation; no null reference error in agent log; reply delivered to Zendesk sandbox

E03

Zendesk webhook delivery failure

Zendesk fails to deliver webhook to the orchestration layer; platform receives no trigger

Zendesk retry policy fires (3 attempts at 5-minute intervals); FullSpec platform logs missed webhook on receipt; alert sent to support@gofullspec.com

Missed webhook alert present in platform error log; retry attempts documented; no ticket silently dropped

E04

Notion knowledge base entirely unavailable

Notion API returns HTTP 503 for all requests during a 5-minute window

All FAQ tickets during outage window routed to human-review queue; error NOTION_UNAVAILABLE logged per ticket; Slack alert sent to support team

Human-review queue in Zendesk sandbox contains all affected tickets; NOTION_UNAVAILABLE log entries present; Slack alert fired

E05

HubSpot contact record has multiple duplicate email entries

Customer email matches three HubSpot contact records

Logging Agent selects the most recently updated record; logs note to that record only; deduplication warning written to error log

Note appears on most-recently-updated contact only; deduplication warning in log with all three contact IDs listed

E06

Human review timeout: flagged ticket not actioned within SLA window

Ticket placed in human-review queue; no agent action taken within 2 hours

Automation fires a Slack reminder to the support team lead after 2 hours; reminder repeats at 4 hours; ticket status updated to overdue

First Slack reminder in sandbox at 2-hour mark; second at 4-hour mark; Zendesk ticket status updated to overdue

E07

Ticket body contains PII beyond email address (e.g. payment card digits)

Ticket body includes partial card number string '4111 1111'

PII detection flag set; ticket body not passed to AI agent; ticket routed immediately to human-review queue; no content logged to HubSpot note

PII flag in classification log; no card-number string present in HubSpot note or Slack message; human-review flag true

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: Use only the designated internal test email address when submitting test tickets in production. Do not use a real customer email address or any address that will generate a genuine customer-facing reply. Every test ticket created in Zendesk during Phase 3 must be voided (deleted or marked as test-void) immediately after the test case completes. The HubSpot activity note created during T25 must be manually deleted from the contact record once the FullSpec team has confirmed it. Do not proceed with Phase 3 until all Phase 2 cases carry a pass status.

Phase 3 confirms that the automation operates correctly in the live production environment across a complete ticket lifecycle. The FullSpec team submits each test ticket, observes all pipeline steps in real time, and documents results. The process owner is present for all Phase 3 cases and records their approval against T27. All test case IDs continue from Phase 2.

Scenario

Production Steps Traced

Expected Output

Pass Criteria

Owner Action

T24

Full production run: FAQ ticket from receipt to auto-reply

Zendesk webhook fires -> Classification Agent classifies as FAQ (confidence >= 0.85) -> Notion article retrieved -> Drafting Agent produces reply -> Zendesk sends reply and applies tags -> HubSpot note created -> No Slack alert (non-escalation)

Customer-facing reply delivered via Zendesk within 2 minutes of ticket submission; correct tags applied; HubSpot note present on contact record

Zendesk shows public reply with timestamp within 2 minutes; tags correct; HubSpot activity note present with ticket ID; Slack channel silent for this ticket

Observe and confirm outputs. No action required.

T25

Full production run: escalation ticket triggers urgent Slack alert and HubSpot log

Zendesk webhook fires -> Classification Agent classifies as escalation (priority: urgent) -> Drafting Agent bypassed -> Ticket placed in human-review queue -> HubSpot note logged -> Slack escalation alert posted to #support-escalations

No auto-reply sent; Zendesk ticket in human-review queue with escalation and urgent tags; HubSpot note category=escalation; Slack alert in #support-escalations channel within 30 seconds

Human-review queue contains ticket; correct tags present; Slack message received within 30 seconds; HubSpot note confirms escalation category

Confirm Slack alert received and readable. Void test ticket in Zendesk and delete HubSpot note after confirmation.

T26

Error log verification: confirm all pipeline events are written to the automation platform audit log

Review the automation platform audit log for T24 and T25 events: trigger receipt timestamp, agent execution times, API call statuses, output payloads

Every pipeline step for T24 and T25 appears in the audit log with a timestamp, status (success/failure), and payload summary. No gaps in the event chain.

Audit log contains entries for all seven pipeline steps across both test runs; no missing events; no unhandled exception entries; all API call statuses recorded as 200 or documented non-200 with retry outcome

Review audit log with FullSpec team. Raise any missing entries before proceeding to T27.

T27

Process owner review and approval: final sign-off recorded in FullSpec confirmation

Process owner reviews outputs from T24, T25, and T26: reply quality, tag accuracy, HubSpot note content, Slack alert format, audit log completeness

Process owner confirms all outputs meet operational requirements. Approval recorded in the FullSpec project confirmation record. No separate signoff block is required.

FullSpec confirmation record updated with process owner name, approval date, and explicit pass status for all Phase 3 cases. Automation cleared for go-live.

Review all Phase 3 outputs. Record approval in the FullSpec confirmation. Confirm go-live readiness with the FullSpec team at support@gofullspec.com.

Once T27 is recorded as approved in the FullSpec confirmation system, the automation is cleared for production go-live. The FullSpec team will enable the live Zendesk webhook, confirm all credentials are scoped correctly, and notify the process owner. The parallel run period defined in the Launch Plan begins immediately after go-live confirmation.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process