Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Delivery Exception Handling

[YourCompany.com] · Fulfilment Department · Prepared by FullSpec · [Today's Date]

This document defines the full test and quality assurance strategy for the Delivery Exception Handling automation. It covers three sequential phases: unit testing of individual agents and steps, integration and edge-case testing across agent handoffs and failure scenarios, and a final end-to-end production verification. The FullSpec team runs all phases and no phase begins until the previous one fully passes. Your process owner joins Phase 3 to confirm the automation meets operational expectations before sign-off is recorded.

01Testing approach

Testing is structured across three sequential phases. No phase begins until every test case in the previous phase has achieved a passing result. This prevents integration-layer defects from being masked by untested unit behaviour and ensures the production run reflects a fully verified system.

Phase 1: Unit Testing

Each agent and step tested in isolation

Scope

Individual agent logic: Exception Triage Agent classification, Resolution Agent decision logic, and each discrete tool action (ShipStation webhook parse, Shopify order fetch, Gorgias ticket creation, Gmail send, Slack prompt, Google Sheets write).

Environment

Sandbox only. All API calls use sandbox or test-mode credentials for every connected tool.

Credential rule

No production API keys, live Shopify stores, real customer email addresses, or live Slack channels may be used in Phase 1.

Pass condition

All test cases T01 through T14 return the exact expected output with no unhandled errors.

Who runs it

FullSpec team. Process owner is not required for this phase.

Phase 2: Integration and Edge-Case Testing

Agent handoffs and failure scenarios tested end to end in sandbox

Scope

Cross-agent handoffs (Triage Agent output consumed by Resolution Agent), all four exception-type paths, the high-value order approval branch, and edge cases including duplicate webhooks, missing order data, tool unavailability, and manager non-response.

Environment

Sandbox only. Simulated payloads are used where live carrier webhooks cannot be replicated in a test environment.

Credential rule

Sandbox credentials only. No live Shopify orders or real customer records may be touched.

Pass condition

All integration test cases T15 through T22 and all edge-case tests E01 through E06 pass with correct outputs and graceful error handling.

Who runs it

FullSpec team. Process owner is not required for this phase.

Phase 3: End-to-End Production Test

Full live-environment run verified by process owner

Scope

One complete end-to-end run per exception type in the production environment, covering the full workflow from ShipStation webhook receipt through to Google Sheets logging and Slack notification.

Environment

Production. Live API credentials active; test orders and internal test email addresses used throughout.

Credential rule

Production credentials are used but all test orders must be voided, refunded, or deleted after each run. No real customer emails may be sent; use an internal test address.

Pass condition

All test cases T23 through T27 pass and the process owner confirms the automation output matches operational expectations.

Who runs it

FullSpec team executes. The process owner (Fulfilment Manager) joins Phase 3 to observe, validate outputs, and record approval.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

All Phase 1 tests must be executed using sandbox credentials exclusively. No production API keys, live Shopify stores, real customer email addresses, or live Slack workspaces may be used at any point during this phase. If a sandbox environment is unavailable for a specific tool, use a mocked response that matches the documented API contract for that tool.

Exception Triage Agent: Complexity: Moderate. Estimated build time: 10 hours. Tests cover webhook ingestion, Shopify order fetch, and classification across all four exception types plus key failure modes.

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: address issue classification

ShipStation webhook payload with exception code ADDRESS_ISSUE; sandbox Shopify order #1001 present

Exception record: category = address_issue, resolution_path = request_correction, order value and customer email populated

All four classification fields set correctly; no null values in the output record

T02

Happy path: failed delivery attempt classification

ShipStation webhook payload with exception code DELIVERY_ATTEMPT_FAILED; sandbox Shopify order #1002 present

Exception record: category = failed_attempt, resolution_path = reship_or_contact_customer

Category and resolution_path match expected values; Shopify order fields fully populated

T03

Happy path: customs hold classification

ShipStation webhook payload with exception code CUSTOMS_HOLD; sandbox Shopify order #1003 present

Exception record: category = customs_hold, resolution_path = await_release_or_refund

Category = customs_hold; resolution_path set; no processing errors logged

T04

Happy path: lost parcel classification

ShipStation webhook payload with exception code PARCEL_LOST; sandbox Shopify order #1004 present

Exception record: category = lost_parcel, resolution_path = reship_or_refund

Category = lost_parcel; resolution_path = reship_or_refund; order value correctly passed downstream

T05

Failure: unrecognised exception code

ShipStation webhook with exception code UNKNOWN_XYZ; sandbox Shopify order #1005 present

Exception record: category = unclassified; resolution_path = manual_review; alert raised to Slack internal test channel

Workflow does not halt; unclassified record created; Slack alert fired to test channel within 30 seconds

T06

Failure: Shopify order not found

ShipStation webhook with valid exception code; order number not present in sandbox Shopify

Error record created with order_id = NOT_FOUND; workflow paused; error logged to console

Workflow does not proceed to Gorgias or Gmail steps; error entry visible in platform execution log

T07

Failure: malformed webhook payload

ShipStation webhook with missing tracking_number field

Payload validation fails at ingestion; error logged; workflow does not start downstream steps

Validation rejection is logged with field name; no partial records written to any downstream tool

Resolution Agent: Complexity: Moderate. Estimated build time: 10 hours. Tests cover the reship path, refund path, high-value approval branch, and Gorgias and Gmail step failures.

Scenario

Input

Expected Output

Pass Criteria

T08

Happy path: reship on low-value lost parcel

Exception record: category = lost_parcel; order value = $45 (below threshold); no manager approval required

Shopify replacement order created; original order timeline updated with resolution note; Gorgias ticket status set to resolved

Replacement order ID returned by Shopify sandbox API; Gorgias ticket status = closed; no approval Slack message sent

T09

Happy path: refund on low-value address issue

Exception record: category = address_issue; order value = $38; customer response indicates preference for refund

Shopify refund processed; Gorgias ticket closed; resolution_type = refund written to output record

Shopify sandbox refund confirmed; Gorgias ticket status = closed; resolution_type field = refund

T10

High-value order: Slack approval prompt sent

Exception record: category = lost_parcel; order value = $320 (above threshold)

Workflow paused; Slack message posted to test channel with order summary, recommended action, and approve/reject buttons

Slack message received in test channel within 60 seconds; workflow execution paused in platform logs; no Shopify action taken before approval

T11

High-value order: manager approves reship

Slack approval button clicked with action = approve_reship on test channel message from T10

Shopify replacement order created; Gorgias ticket updated; resolution_type = reship; audit record written

Shopify sandbox order created after Slack approval event received; Gorgias ticket updated; execution log shows approval event timestamp

T12

Gorgias ticket creation fails

Valid exception record; Gorgias sandbox API returns 503 error

Retry attempted up to three times at 30-second intervals; if all retries fail, error logged and Slack alert sent to test channel; Gmail notification still sent

Retry count visible in execution log; Gmail send not blocked by Gorgias failure; Slack error alert delivered

T13

Gmail send fails

Valid exception record; Gmail sandbox API returns authentication error

Gmail step errors; error logged to platform; Gorgias ticket note added indicating email not sent; Slack alert posted to test channel

Gorgias ticket contains note about failed email; Slack alert delivered; workflow does not halt entirely

T14

Google Sheets write fails

Valid resolved exception; Google Sheets API returns quota exceeded error

Retry attempted twice; if still failing, error logged and Slack alert sent; Shopify resolution not rolled back

Shopify resolution unaffected; Slack alert visible in test channel; error entry in platform execution log

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Phase 2 tests the handoffs between agents and tool steps in sequence, verifying that outputs produced by the Exception Triage Agent are correctly consumed by the Resolution Agent and that all downstream tool actions fire in the right order. IDs continue from Phase 1.

Scenario

Input

Expected Output

Pass Criteria

T15

Full handoff: Triage Agent to Resolution Agent on address issue

ShipStation webhook with ADDRESS_ISSUE code; sandbox Shopify order #2001 value $55

Triage Agent output record consumed by Resolution Agent; Gmail sent; Gorgias ticket opened; reship or correction path initiated

Resolution Agent reads all four classification fields from Triage Agent output without mapping errors; no null fields passed

T16

Full handoff: Triage Agent to Resolution Agent on customs hold

ShipStation webhook CUSTOMS_HOLD; sandbox order #2002 value $90

Exception classified as customs_hold; Gmail template for customs delay sent; Gorgias ticket tagged customs_hold; workflow pauses for carrier update

Correct Gmail template variant sent; Gorgias tag = customs_hold; no premature Shopify action taken

T17

Gorgias ticket created before Gmail send

Valid exception record from Triage Agent; sandbox Gorgias and Gmail credentials active

Gorgias ticket ID generated and stored in workflow context before Gmail send step executes

Gorgias ticket ID present in workflow context at time of Gmail step; Gmail message body includes Gorgias ticket reference number

T18

Slack approval followed by Shopify write followed by Google Sheets log

High-value exception; manager approval via Slack test channel; order value $410

Shopify reship order created after Slack approval; Google Sheets row written with resolution_type, timestamp, and order value; Slack summary posted

All three downstream steps execute in correct sequence after Slack approval event; Google Sheets row contains all required columns with no blanks

T19

Google Sheets log row schema validation

Completed exception resolution record passed to Sheets step

Row written with columns: order_id, exception_type, resolution_type, customer_email (masked), timestamp, resolved_by (system or manager_name), order_value

All seven columns present and non-null; customer_email stored as domain-masked value only (e.g. c***@example.com)

T20

Slack channel summary message format

Resolved exception record with resolution_type = refund; order #2005

Slack message posted to fulfilment test channel: order number, exception type, resolution applied, timestamp, Gorgias ticket link

Message appears in test Slack channel within 30 seconds of Sheets write; all five fields present in message body

T21

ShipStation polling fallback triggers correctly

15-minute polling job fires; ShipStation returns two new exception events not previously received via webhook

Both exceptions processed through full workflow as if received by webhook; no duplication with existing records

Two new Gorgias tickets created; two Gmail notifications sent; no duplicate records for orders already in Sheets log

T22

Manager rejection via Slack routes to manual review

High-value exception Slack prompt; manager selects reject action on test channel message

Workflow flags case as manual_review_required; Gorgias ticket updated with note; no Shopify action taken; Slack message updated to show rejected status

No Shopify order or refund created; Gorgias ticket note contains rejection timestamp and action = manual_review; Slack message status updated

Edge-case tests cover scenarios that fall outside normal happy-path and failure-mode behaviour: duplicate events, structurally incomplete data, tool outages, and human timeout conditions.

Scenario

Input

Expected Output

Pass Criteria

E01

Duplicate webhook for same shipment within 5 minutes

Two identical ShipStation webhooks received for tracking number TN-88001 within 3 minutes

Second webhook detected as duplicate using tracking number deduplication check; second run suppressed; one Gorgias ticket and one Gmail send only

Execution log shows deduplication triggered on second webhook; only one Gorgias ticket and one Sheets row for TN-88001

E02

Shopify order record missing customer email

Valid webhook; Shopify order found but customer.email field is null

Gmail step skipped; Gorgias ticket created with note 'Customer email unavailable'; Slack alert posted to test channel; exception still logged to Sheets

Gmail step does not fire; Gorgias note present; Sheets row written with customer_email = NULL flagged; Slack alert received

E03

ShipStation webhook received but ShipStation API unreachable for enrichment

Webhook payload received; ShipStation API returns timeout on enrichment call

Workflow uses payload data only; enrichment marked as unavailable; Gorgias ticket opened with available data; error note added to ticket

Workflow does not halt; Gorgias ticket created with partial data and enrichment_status = unavailable note; no duplicate triggered on retry

E04

Shopify API rate limit hit during reship order creation

Resolution Agent attempts to create reship order; Shopify returns 429 Too Many Requests

Exponential backoff applied: retry at 5 seconds, 20 seconds, 60 seconds; if all retries fail, Gorgias ticket updated with note and Slack alert sent

Retry sequence visible in execution log with correct intervals; Shopify order created on successful retry, or Slack alert and Gorgias note confirmed if all retries fail

E05

Manager Slack approval prompt times out with no response after 24 hours

High-value exception Slack prompt sent; no manager action taken within 24-hour window

Workflow escalates: second Slack reminder sent; Gorgias ticket priority updated to urgent; case flagged in Sheets as awaiting_approval_overdue

Second Slack message sent at 24-hour mark; Gorgias ticket priority = urgent; Sheets row shows status = awaiting_approval_overdue

E06

Exception arrives for an order that has already been refunded in Shopify

ShipStation webhook for order #2099; Shopify lookup returns order status = refunded

Workflow detects refunded status; no Shopify action taken; Gorgias ticket opened with note 'Order already refunded'; Slack alert posted; Sheets row written with resolution_type = already_resolved

No Shopify refund or reship attempted; Gorgias note and Sheets row both reflect already_resolved status; Slack alert received in test channel

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: All test runs must use an internal test email address (for example qa-test@[YourCompany.com]) as the customer notification recipient. No real customer email addresses may appear in any test order. Every Shopify test order created during Phase 3 must be voided or fully refunded immediately after the test case is verified. Any Gorgias ticket opened during Phase 3 must be closed and tagged test-qa before the end of the testing session. The FullSpec team will confirm all production records have been cleaned up before go-live is approved.

Phase 3 runs one complete end-to-end production trace per key workflow path. The FullSpec team executes each case with the Fulfilment Manager (process owner) present to observe outputs in real time. Test case IDs continue from Phase 2.

Scenario

Environment

Steps Verified

Pass Criteria

T23

Full production run: lost parcel, low-value order

Production; test order value $42; recipient = qa-test@[YourCompany.com]

ShipStation webhook received, Shopify order fetched, Triage Agent classifies as lost_parcel, Gorgias ticket opened, Gmail sent to test address, Resolution Agent creates reship in Shopify, Sheets row written, Slack summary posted

All eight steps complete within 3 minutes of webhook receipt; Gorgias ticket and Shopify reship order visible; Sheets row present with correct columns; Slack message in fulfilment channel

T24

Full production run: address issue, low-value order

Production; test order value $55; recipient = qa-test@[YourCompany.com]

Webhook received, order fetched, classified as address_issue, Gorgias ticket tagged address_issue, Gmail address-correction template sent, Resolution Agent waits for customer response simulation, Sheets row written

Correct Gmail template variant (address correction) delivered to test inbox; Gorgias ticket tag = address_issue; no premature Shopify action; Sheets row written

T25

Full production run: high-value order approval path

Production; test order value $375 (above threshold); recipient = qa-test@[YourCompany.com]

Webhook received, classified as lost_parcel, Slack prompt sent to Fulfilment Manager test channel, manager approves reship, Shopify reship order created, Sheets row written with approved_by field populated

Slack approval prompt visible in production Slack channel within 60 seconds; Shopify reship order created only after approval button clicked; Sheets approved_by field = manager Slack display name

T26

Error log and audit trail verification

Production; review execution logs from T23, T24, T25

Execution logs inspected for each run: timestamps, step durations, API response codes, any retry events, final status

Logs present for all three prior runs; no unhandled exceptions in logs; all API calls show 2xx response codes or documented retry outcomes; total end-to-end time under 3 minutes for each non-approval run

T27

Process owner review and approval

Production; Fulfilment Manager reviews all Phase 3 outputs

Fulfilment Manager reviews: Gorgias tickets, Gmail test inbox, Shopify order timeline, Sheets log rows, Slack messages, and execution logs from T23 through T26

Fulfilment Manager confirms all outputs match operational expectations and approves go-live. Approval is recorded by the FullSpec team in the FullSpec project confirmation record. No separate signoff block is required.

Once T27 is complete and the process owner approval is recorded, the FullSpec team will remove all Phase 3 test records, confirm production credentials are secured, and mark the QA phase as closed in the project record. Questions about this document or the testing process can be directed to support@gofullspec.com.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process