Back to Shipping & Tracking Notification

Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Shipping and Tracking Notification

[YourCompany.com] · Fulfilment Department · Prepared by FullSpec · [Today's Date]

This document defines the complete quality assurance process for the Shipping and Tracking Notification automation. It is written for the FullSpec build team and covers every test case that must pass before the automation is considered production-ready. Testing is structured across three sequential phases: Unit, Integration, and End-to-End. No phase begins until every case in the preceding phase has a confirmed pass status. The process owner joins Phase 3 to review and approve the final production run.

01Testing approach

All testing follows three phases in sequence. No phase begins until the previous phase fully passes. Each phase uses a distinct environment and credential set, and pass conditions are defined by explicit test-case ID ranges. The FullSpec team runs all three phases; the process owner joins Phase 3 only for the final review and approval case.

Phase 1: Unit Testing

Test each agent in isolation before any cross-agent communication is attempted

Scope

Individual agent logic: Shipment Event Listener (ShipStation trigger, Shopify fetch, tracking URL construction) and Notification Dispatch Agent (Klaviyo send, Google Sheets log, Slack post) tested independently with mocked or sandboxed inputs.

Environment

Sandbox only. No production credentials, no live customer records, no real Klaviyo sends.

Credential rule

Use sandbox API keys for ShipStation, Shopify dev store, Klaviyo test API key, a designated test Google Sheet, and a private Slack test channel. No production keys may be loaded during Phase 1.

Pass condition

All cases T01 through T14 return the expected output with zero unhandled errors.

Who runs it

FullSpec team only.

Phase 2: Integration and Edge Case Testing

Test agent handoffs, data propagation, and failure-mode handling across all connected tools

Scope

End-to-end data flow between the two agents, including the decision branch for missing customer email. Separate edge-case table covers duplicates, missing data fields, unavailable tools, and human timeout scenarios.

Environment

Sandbox for all tool connections. ShipStation staging webhook endpoint; Shopify dev store; Klaviyo test mode (suppresses real delivery); Google Sheets test tab; Slack test channel.

Credential rule

Sandbox credentials only. Edge-case inputs must be fabricated test records; no real order IDs or customer emails may be used.

Pass condition

All cases T15 through T22 pass, and all edge cases E01 through E06 behave per the defined expected output, including correct routing to the manual review path where specified.

Who runs it

FullSpec team only.

Phase 3: End-to-End Production Testing

Validate the full automation under real production conditions with a controlled test order

Scope

A single end-to-end run using production credentials and a real (but controlled) test order. Covers the full chain from ShipStation webhook through to Klaviyo send, Google Sheets log row, and Slack digest post. Includes error-log verification and process owner review.

Environment

Production. All live API keys active. Test order placed and voided after verification.

Credential rule

Production credentials are active. Test email must be an internal address. The test order record must be voided in both ShipStation and Shopify immediately after the run is confirmed complete.

Pass condition

All cases T23 through T27 pass, including the process owner sign-off recorded in the FullSpec confirmation system (T27).

Who runs it

FullSpec team executes; process owner joins for T27 review and approval.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only during Phase 1. Do not load any production API key, production order ID, or real customer email address into the test environment at this stage. All ShipStation, Shopify, Klaviyo, Google Sheets, and Slack connections must point to their respective sandbox or test equivalents.

Agent 1: Shipment Event Listener (estimated build time: 12 hours, complexity: Moderate)

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: webhook fires with valid tracking number

ShipStation webhook payload: order_id=TEST001, status=Shipped, tracking_number=1Z999AA10123456784, carrier=UPS

Structured data object: customer_email, order_number, carrier_name=UPS, tracking_url=https://wwwapps.ups.com/tracking/tracking.cgi?tracknum=1Z999AA10123456784

All four fields populated; tracking URL correctly formed for UPS carrier; no errors logged.

T02

Happy path: polling mode detects new shipment

ShipStation poll returns one new order with status=Shipped and a FedEx tracking number

Same structured data object with carrier_name=FedEx and correct FedEx tracking URL prefix

URL prefix matches https://www.fedex.com/fedextrack/?trknbr= and tracking number appended correctly.

T03

Happy path: USPS carrier URL construction

ShipStation webhook: carrier=USPS, tracking_number=9400111899223397988523

tracking_url=https://tools.usps.com/go/TrackConfirmAction?tLabels=9400111899223397988523

USPS URL prefix used; no carrier mismatch error thrown.

T04

Shopify order fetch returns full record

Valid order_id passed to Shopify API after ShipStation trigger

Shopify order object returned with customer.email, order_number, first_name, last_name

All required fields present in response; HTTP 200 returned; no null fields.

T05

Tracking number is null at trigger time

ShipStation webhook: status=Shipped, tracking_number=null

Automation halts at trigger; no Shopify fetch attempted; error logged: 'Null tracking number — event skipped'

No downstream actions fire; error entry visible in the orchestration layer error log.

T06

ShipStation returns an unrecognised carrier code

webhook: carrier=DHL_EXPRESS_ASIA (not in carrier map)

Error logged: 'Unknown carrier code — tracking URL could not be formed'; agent halts and flags for manual review

No malformed URL is passed downstream; flag visible in error log; manual review queue receives the record.

T07

Shopify order fetch returns HTTP 404

order_id=MISSING999 passed to Shopify; Shopify returns 404

Retry attempted once after 30 seconds; on second 404, error logged and record routed to manual review

Retry logic executes exactly once; final error state logged; no unhandled exception thrown.

T08

Shopify API rate limit hit (HTTP 429)

Shopify returns 429 Too Many Requests on first call

Automation pauses for the Retry-After period specified in the response header, then retries successfully

Backoff delay applied; second call succeeds; no data loss; total delay logged.

T09

Order status is Shipped but tracking number is a duplicate of a previously processed order

tracking_number=1Z999AA10123456784 (same as T01, already processed)

Duplicate detected via Google Sheets log lookup; event skipped; log entry reads 'Duplicate tracking number — already notified'

No second Klaviyo email fired; existing log row unchanged; duplicate flag written to error log.

Agent 2: Notification Dispatch Agent (estimated build time: 10 hours, complexity: Moderate)

Scenario

Input

Expected Output

Pass Criteria

T10

Happy path: Klaviyo transactional email fires successfully

Valid structured data object: customer_email=test@internal.com, order_number=ORD-1001, carrier_name=UPS, tracking_url=valid UPS URL, first_name=Test

Klaviyo API returns event_id; email queued for delivery; HTTP 200 confirmed

Klaviyo response contains a non-null event_id; no delivery error in Klaviyo activity feed.

T11

Google Sheets log row appended correctly

Same valid data object as T10 after successful Klaviyo send

New row written to designated sheet tab: order_id, customer_email, tracking_number, carrier, send_timestamp, status=Sent

Row appears in the correct tab; all six columns populated; timestamp is within 5 seconds of send time.

T12

Slack digest post fires correctly

Batch of 3 successful sends in the test session

Slack message posted to test channel: '3 tracking notifications sent in the last session. 0 failures.'

Message appears in the correct Slack channel within 60 seconds; count matches actual sends in the test run.

T13

Klaviyo API returns a 400 error (invalid template ID)

Structured data object with a misconfigured Klaviyo flow ID

Error logged: 'Klaviyo send failed — invalid flow ID'; Google Sheets row written with status=Failed; Slack message flags 1 failure

No retry on a 400; log row status=Failed; Slack failure count increments correctly.

T14

Google Sheets write fails due to permission error

Klaviyo send succeeds but Google Sheets service account lacks write permission

Error logged: 'Google Sheets write failed — permission denied'; Slack post still fires; error flagged in log

Klaviyo send is not rolled back; Slack still posts; permission error appears in error log without crashing the run.

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

The following integration tests verify that data passes correctly between the Shipment Event Listener and the Notification Dispatch Agent, and that the decision branch for missing customer email routes correctly to the manual review path. All tests use sandbox credentials.

Scenario

Input

Expected Output

Pass Criteria

T15

Full happy-path handoff: Listener to Dispatch Agent

ShipStation webhook fires with valid tracking number; Shopify returns full order record with valid customer email

Structured data object passes from Listener to Dispatch Agent intact; Klaviyo fires; Sheets row written; Slack posts

All five downstream actions complete; no field is null or mutated between agents; end-to-end latency under 30 seconds.

T16

Decision branch: customer email missing in Shopify record

Shopify order record returns customer.email=null

Automation takes the 'no' branch; order flagged in the manual review queue; Sheets row written with status=Pending Manual Review; no Klaviyo send

Klaviyo not called; manual review flag visible; Sheets row status=Pending Manual Review; Slack post counts 1 flagged order.

T17

Multiple simultaneous shipments processed in sequence

Three ShipStation webhook events fire within 10 seconds of each other, all valid

Three separate Klaviyo sends, three Sheets rows, and one aggregated Slack post listing 3 notifications sent

No events dropped; no duplicate sends; all three Sheets rows distinct; Slack count=3.

T18

Klaviyo send succeeds but Slack API is temporarily unavailable

Klaviyo returns success; Slack API returns HTTP 503 on first attempt

Retry attempted after 60 seconds; on success, Slack post fires; on continued failure after 3 retries, error logged but run marked complete

Klaviyo send and Sheets log are not affected by Slack failure; retry logic fires; final state logged correctly.

T19

Carrier URL constructed correctly for all mapped carrier codes

Four separate webhook events: UPS, FedEx, USPS, Australia Post (if mapped)

Each produces a correctly prefixed tracking URL specific to that carrier

Zero malformed URLs; each URL passes a format validation check (correct prefix, tracking number appended, no spaces).

T20

ShipStation polling fallback when webhook is unavailable

Webhook endpoint disabled; polling interval set to 5 minutes

Polling detects new shipped order within one poll cycle; processing proceeds identically to webhook path

Order processed within 5 minutes of ship event; no difference in output object structure.

T21

Google Sheets log deduplication check on re-run

Same order_id submitted twice to the Dispatch Agent (simulating a retry scenario)

Second run detects existing row in Sheets for that order_id; skips Klaviyo send; logs 'Duplicate detected — skipped'

Exactly one Klaviyo send per order_id; Sheets contains one row per order; no duplicate customer email sent.

T22

Partial data object passed between agents (tracking_url missing)

Listener produces a data object where tracking_url field is an empty string due to an unmapped carrier

Dispatch Agent detects empty tracking_url; halts send; routes to manual review; logs 'Empty tracking URL — send aborted'

No Klaviyo send with a blank link; manual review flag set; error logged with order_id and carrier code for diagnosis.

Edge case tests cover scenarios outside normal operating conditions, including data integrity failures, unavailable external services, and human-in-the-loop timeout behaviour.

Scenario

Input

Expected Output

Pass Criteria

E01

Duplicate webhook event for the same order (carrier fires event twice)

ShipStation sends two identical webhook payloads for order_id=TEST050 within 2 seconds

Second event detected as a duplicate via Sheets deduplication check; skipped; no second email sent

Customer receives exactly one tracking email; Sheets shows one row; duplicate event flagged in log.

E02

Order record missing required fields: first_name is null

Shopify returns order with customer.first_name=null; email is valid

Agent substitutes a fallback salutation ('Hi there') in the Klaviyo template; email sends successfully; log row notes 'Name field absent'

Email is delivered with fallback salutation; no hard failure; log row records the substitution.

E03

ShipStation API is completely unavailable for 10 minutes

ShipStation returns HTTP 503 on all calls for a 10-minute window

Automation retries at 2-minute intervals up to 5 attempts; after 5 failures, alert posted to Slack error channel; queue resumes when API recovers

No data lost during outage window; retry log shows 5 attempts; Slack alert fires after 5th failure; backlog processed on recovery.

E04

Klaviyo send returns a hard bounce for the test email address

Klaviyo activity feed returns bounce_type=hard for the test address

Sheets row status updated to Bounced; Slack digest increments failure count; no retry attempted (hard bounce suppressed)

No retry on a hard bounce; Sheets row correctly marked Bounced; customer is added to Klaviyo suppression list by Klaviyo natively.

E05

Manual review item not actioned within 24 hours (human timeout)

An order flagged for missing email sits in the manual review queue for 24 hours without a response from the Customer Service Rep

Automation sends a reminder Slack message to the fulfilment channel: 'Order ORD-XXXX awaiting manual email confirmation for 24 hours'

Reminder message fires exactly once at the 24-hour mark; no duplicate reminders; original flag remains open until resolved.

E06

Malformed tracking number (special characters injected)

ShipStation payload contains tracking_number='1Z999AA1<script>alert(1)</script>'

Input sanitised before URL construction; special characters stripped; if tracking number is non-numeric after sanitisation, event routed to manual review with error log entry

No unsanitised string passed to any downstream tool; no XSS vector in any URL or log field; manual review flag set.

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: The test order must be placed using an internal company email address (for example test-qa@[YourCompany.com]) that is controlled by the FullSpec team or the process owner. No real customer email address may receive a test notification. After the Phase 3 run is confirmed complete, the test order record must be voided in both ShipStation and Shopify, the Google Sheets log row must be marked 'TEST VOID', and the Klaviyo test send must be confirmed as suppressed from all customer-facing metrics. Do not proceed with Phase 3 until Phase 2 has a 100% pass record.

Scenario

Input

Expected Output

Pass Criteria

T23

Full production run: order marked shipped in live ShipStation account

A real test order is created in ShipStation and marked as Shipped with a valid UPS tracking number. Customer email field points to the internal test address.

ShipStation webhook fires within 60 seconds; Shopify order record fetched successfully; tracking URL constructed; Klaviyo email delivered to the internal test inbox within 2 minutes of ship event.

Internal test inbox receives the email; email contains the correct order number, customer first name (from Shopify), UPS tracking URL, and carrier name. Latency from webhook to email delivery is under 2 minutes.

T24

Google Sheets production log row verified

Post-send verification of the live Google Sheets fulfilment tracker

A new row is present in the tracker tab containing: order_id, internal test email address, tracking number, carrier=UPS, send_timestamp (within 5 seconds of Klaviyo confirmation), status=Sent

All six columns populated; timestamp accurate; status=Sent; no duplicate rows for this order_id.

T25

Slack dispatch summary posted to production fulfilment channel

Completion of the T23 and T24 actions

A Slack message appears in the live fulfilment Slack channel: '1 tracking notification sent. 0 failures.' Message includes the order number for reference.

Message appears in the correct channel within 60 seconds of send; count=1; failure count=0; order reference visible.

T26

Error log verification: no unexpected errors during the production run

Full error log reviewed by the FullSpec team immediately after T23 through T25 complete

Zero unhandled exceptions; zero retry events; zero null-field warnings during the production run

Error log is clean for the duration of the Phase 3 test window. Any entries present are categorised and explained before sign-off is granted.

T27

Process owner review and approval

The process owner (Fulfilment Manager or nominated delegate) reviews the internal test email received in T23, the Sheets log row from T24, the Slack post from T25, and the error log from T26.

Process owner confirms that the email content, formatting, tracking URL, and log records all meet operational requirements. Approval is recorded in the FullSpec confirmation system.

FullSpec records the process owner's approval response (name, timestamp, and confirmation statement) in the project record. The automation is not moved to the live monitoring schedule until this confirmation is received. No separate signature block is required; the FullSpec confirmation entry is the formal record of approval.

After T27 is confirmed, the FullSpec team will void the test order in ShipStation and Shopify, mark the Sheets log row as TEST VOID, and archive the Phase 3 results in the project record. The automation is then moved to active monitoring. Any issues raised during the process owner review in T27 must be resolved and re-tested before the approval status is recorded. Contact support@gofullspec.com for any questions during the QA process.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process