Back to Vendor Performance Tracking

Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality
FS-DOC-06Quality

Test and QA Plan

Vendor Performance Tracking

[YourCompany.com] · Operations Department · Prepared by FullSpec · [Today's Date]

This document defines the complete quality assurance process for the Vendor Performance Tracking automation. It is written for the FullSpec build team and covers every test case required before the workflow is promoted to production. Testing runs in three sequential phases: unit tests on individual agents, integration and edge-case tests across agent handoffs, and a final end-to-end production run that includes process owner sign-off. No phase begins until every case in the previous phase achieves a passing status. The process owner joins Phase 3 only; all earlier phases are executed entirely by the FullSpec team in a sandboxed environment.

01Testing approach

Testing runs in three phases in strict sequence. No phase begins until the previous phase has fully passed, meaning every test case in that phase carries a confirmed pass status and any failures have been resolved and re-tested. This sequencing prevents integration assumptions from being built on untested units and ensures production credentials are never used until the workflow has already demonstrated reliability in controlled conditions.

1
Phase 1: Unit Tests
Test each agent in isolation before connecting them
Scope
Each of the three agents tested independently: Vendor Data Collection Agent, Performance Scoring Agent, and Escalation and Reporting Agent. No cross-agent data flow is tested at this stage.
Environment
Sandbox only. Xero sandbox account, Airtable test base, Google Sheets test workbook, Slack test workspace, Gmail test address.
Credential rule
No production credentials may be loaded. All API keys, OAuth tokens, and service account credentials must reference sandbox or test instances exclusively.
Pass condition
All cases T01 through T14 return the expected output with no unhandled exceptions. Any failure blocks progression to Phase 2.
Who runs it
FullSpec team only. Process owner is not involved in Phase 1.
2
Phase 2: Integration and Edge-Case Tests
Test agent handoffs and failure scenarios
Scope
End-to-end data flow across all three agents, including handoff payloads between agents. Separate edge-case table covers duplicate events, missing fields, tool unavailability, and human approval timeout.
Environment
Sandbox only, with simulated tool outages and injected malformed payloads where specified.
Credential rule
Sandbox credentials only. Rate-limit headers must be monitored; no production quota must be consumed.
Pass condition
All cases T15 through T22 and E01 through E06 pass. Edge cases must either handle gracefully with correct error logging or escalate to the defined fallback path without data loss.
Who runs it
FullSpec team only. Process owner is not involved in Phase 2.
3
Phase 3: End-to-End Production Test
Full production run with process owner sign-off
Scope
One complete production run using live credentials and a real vendor event. The run is traced from Xero trigger through to Google Sheets monthly report generation. Error logs and Airtable records are verified. Process owner reviews and approves the output.
Environment
Production. Live Xero account, live Airtable base, live Google Sheets scorecard, live Slack ops channel, live Gmail account. Test record must be voided after the run.
Credential rule
Production credentials used for the first time in this phase only, following successful completion of Phases 1 and 2.
Pass condition
All cases T23 through T27 pass, including process owner sign-off recorded in the FullSpec confirmation (T27). No separate signoff block is required.
Who runs it
FullSpec team executes cases T23 through T26. Process owner joins for T27 to review outputs and confirm approval. FullSpec records the confirmation.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only. Before running any case in Phase 1, confirm that all configured API connections point to sandbox or test instances. No production Xero account, live Airtable base, or real Gmail address may be referenced. Any credential mismatch must be corrected before the first test executes.

Vendor Data Collection Agent — Complexity: Moderate. Estimated build time: 10 hours. Tests T01 through T05.

ID
Scenario
Input
Expected Output
Pass Criteria
T01
Happy path: new invoice received in Xero
Xero sandbox webhook payload: valid invoice with linked PO, vendor ID, billed amount, and delivery date populated
Structured data record written to Airtable test base containing: vendor_id, delivery_date_variance (integer days), invoice_accuracy (boolean), billed_amount, ordered_amount
All five fields present and correctly typed in Airtable record. No errors logged. Execution time under 30 seconds.
T02
Happy path: PO status changes to received
Xero sandbox poll response: PO marked received, matching invoice found, quantities match
Structured data record with delivery_date_variance = 0, invoice_accuracy = true, billed equals ordered
Record created in Airtable. delivery_date_variance = 0. invoice_accuracy = true. No error log entries.
T03
Failure: Xero returns 401 Unauthorized
Expired OAuth token injected into agent configuration
Agent halts, logs authentication error with timestamp and vendor context, does not write partial record to Airtable
Error entry present in automation platform log. No partial Airtable record created. Alert or retry mechanism triggered as configured.
T04
Failure: PO not found for invoice
Invoice payload with a PO reference number that does not exist in the Xero sandbox
Agent logs a missing-PO warning, writes a record with po_match = false and delivery_date_variance = null, does not halt
Record written to Airtable with po_match = false. delivery_date_variance field is null. Warning log entry present. Execution continues.
T05
Failure: Airtable write returns 422 Unprocessable Entity
Valid Xero payload but Airtable field schema mismatch injected (wrong field type for billed_amount)
Agent logs Airtable write error with payload snapshot, does not retry indefinitely, surfaces error to monitoring
Error logged with full payload context. No infinite retry loop. Monitoring alert generated.

Performance Scoring Agent — Complexity: Moderate. Estimated build time: 10 hours. Tests T06 through T10.

ID
Scenario
Input
Expected Output
Pass Criteria
T06
Happy path: vendor event record triggers scoring, score above threshold
New Airtable test record: delivery_date_variance = 0, invoice_accuracy = true, no quality flags. Threshold set to 70.
Vendor score calculated as >= 70, status = clear, one-paragraph performance summary written to Airtable record, Google Sheets scorecard row updated
Score >= 70 in Airtable. Status field = clear. Summary field populated (min 20 words). Google Sheets row updated within 60 seconds.
T07
Happy path: vendor event triggers scoring, score below threshold (flagged)
New Airtable test record: delivery_date_variance = 4 days late, invoice_accuracy = false, quality flag present. Threshold = 70.
Vendor score < 70, status = flagged, summary written, Airtable record updated, Google Sheets scorecard row marked as flagged
Score < 70 in Airtable. Status = flagged. Summary present. Google Sheets flag column updated. Escalation and Reporting Agent trigger condition met.
T08
Failure: scoring model receives null delivery_date_variance
Airtable record where delivery_date_variance = null (PO not matched upstream)
Agent substitutes a neutral score for the missing dimension, logs a data-quality warning, still produces a score and summary without crashing
Score produced. data_quality_warning flag set to true in record. Execution completes. No unhandled exception.
T09
Failure: Google Sheets API returns 429 Too Many Requests
Scoring completes but Google Sheets API responds with rate-limit error on write
Agent waits the retry-after interval, retries up to three times, logs outcome. Airtable record is updated regardless of Sheets status.
Airtable record updated before Sheets write is attempted. Sheets write retried up to 3 times with exponential backoff. Final status logged.
T10
Failure: scoring threshold not configured
Airtable record present but threshold environment variable is missing or empty
Agent logs a configuration error, halts scoring for that record, does not write a score, surfaces alert
No score written. Configuration error logged with variable name. Monitoring alert triggered. No silent default applied.

Escalation and Reporting Agent — Complexity: Moderate. Estimated build time: 8 hours. Tests T11 through T14.

ID
Scenario
Input
Expected Output
Pass Criteria
T11
Happy path: flagged vendor triggers Slack alert
Airtable record with status = flagged, vendor name, score, and triggering event populated
Slack message posted to test ops channel containing vendor name, current score, event description, and a link to the Airtable record
Slack message received in test channel within 60 seconds. All four required fields present in message body. No duplicate message sent.
T12
Happy path: Gmail draft created and awaits manager approval
Flagged vendor record plus ops manager approval simulation (approval = true in test environment)
Gmail draft created with vendor email address in To field, standard improvement template body populated with vendor name and specific issue, draft visible in Gmail sandbox
Draft present in Gmail sandbox. To field correct. Template fields all populated. Email not yet sent (send gated on approval).
T13
Happy path: month-end report generation triggered
Last-business-day-of-month cron trigger fires in test environment; Airtable test base contains 5 vendor records with scores and events from the simulated month
Monthly summary appended to Google Sheets test workbook: one row per vendor with score trend, flag count, and outcome notes
New rows appended to correct Google Sheets tab. All 5 vendors present. Score trend, flag count, and outcome notes populated for each.
T14
Failure: Slack API returns 503 Service Unavailable
Flagged vendor record triggers alert but Slack API is simulated as unavailable
Agent retries up to 3 times with backoff, logs the failure, falls back to logging the alert in Airtable as a missed-notification record
Retry attempts logged (max 3). Fallback Airtable record created with notification_status = failed. No unhandled exception.
Test and QA PlanPage 2 of 4
FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Phase 2 verifies that agents pass data correctly to one another and that the full chain from Xero trigger to Google Sheets output completes without loss or corruption. Integration test IDs continue from T15. Edge case tests use a separate E-series sequence covering scenarios that cross agent boundaries or involve external tool failure.

ID
Scenario
Input
Expected Output
Pass Criteria
T15
Full handoff: Data Collection Agent to Scoring Agent
Xero sandbox webhook triggers Data Collection Agent; output record written to Airtable test base
Scoring Agent detects the new Airtable record within its polling interval, reads all required fields, and produces a score without manual intervention
Scoring Agent execution log shows trigger from Airtable record creation. Score written within 2 minutes of Data Collection Agent completion. No field mapping errors.
T16
Full handoff: Scoring Agent to Escalation Agent (flagged path)
Scoring Agent writes status = flagged to Airtable test record
Escalation and Reporting Agent triggers within its polling interval, sends Slack alert to test channel, creates Gmail draft
Escalation Agent log shows trigger from flagged status. Slack message and Gmail draft both created. End-to-end elapsed time under 5 minutes.
T17
Full handoff: Scoring Agent to Google Sheets (clear path, no escalation)
Scoring Agent writes status = clear to Airtable test record
Google Sheets scorecard row updated with new score and clear status. No Slack alert sent. No Gmail draft created.
Sheets row updated. Slack and Gmail actions not triggered. Escalation Agent log confirms no-escalation branch taken.
T18
Month-end report: full chain from Airtable to Google Sheets
Cron trigger fires in sandbox; Airtable test base has 8 vendor records with mixed statuses across simulated month
Report appended to Google Sheets with correct vendor count, accurate aggregate scores, and flag summary
8 vendor rows in report. Aggregate scores match Airtable data. Flag count matches records with status = flagged. Report appended to correct tab, not overwritten.
T19
Manager approval gate: email held until approval received
Flagged vendor triggers Gmail draft creation; no approval action taken for 10 minutes in test environment
Email remains in draft state. No send action executed. Slack alert remains visible. System does not auto-send after timeout.
Gmail draft status unchanged after 10 minutes. No sent mail in test Gmail sandbox. Approval gate confirmed as blocking.
T20
Manager approval gate: email sends after approval
Flagged vendor draft exists; approval flag set to true in test environment to simulate manager action
Gmail draft promoted to sent. Vendor email address in To field. Airtable record updated with email_sent = true and timestamp.
Email present in Gmail sandbox sent items. Airtable record email_sent = true. Timestamp within 60 seconds of approval flag set.
T21
Concurrent vendor events: two invoices arrive simultaneously
Two Xero sandbox webhook payloads fired within 2 seconds for different vendor IDs
Two separate Airtable records created. Each scored independently. No field data cross-contamination between records.
Two distinct Airtable records with correct vendor IDs. Scores independent. No shared field values. Both Google Sheets rows updated.
T22
Same vendor, multiple events in one month: rolling score accumulates correctly
Three sequential Airtable records for the same vendor_id created across simulated month (scores 85, 60, 55)
Rolling score reflects weighted average of all three events. Scorecard row shows downward trend. Flag triggers on third event when score crosses threshold.
Rolling score calculated correctly per scoring model. Flag status = flagged after third event. Historical score entries preserved in Airtable performance log.

Edge-case tests — IDs E01 through E06. These cases specifically target conditions outside the normal happy path that span multiple agents or involve external system failure.

ID
Scenario
Input
Expected Output
Pass Criteria
E01
Duplicate Xero event: same invoice webhook fires twice
Identical Xero webhook payload sent twice within 5 seconds (simulating Xero retry on network error)
Data Collection Agent detects duplicate via idempotency key check; second event discarded; only one Airtable record created
Airtable contains exactly one record for the invoice ID. Duplicate-detection log entry present. No duplicate score calculated.
E02
Missing vendor name in Xero payload
Xero webhook payload where vendor_name field is null or empty string
Agent logs a data-quality warning, writes record with vendor_name = UNKNOWN, does not halt; Slack alert (if triggered) includes UNKNOWN label with record link
Record written with vendor_name = UNKNOWN. Warning log present. Scoring proceeds. Slack alert (if applicable) contains UNKNOWN label and Airtable link.
E03
Airtable API unavailable during Scoring Agent write
Scoring Agent completes calculation but Airtable returns 503 for the write attempt
Agent retries up to 3 times with exponential backoff (2s, 4s, 8s), logs each attempt, and after final failure stores the scored payload in the automation platform's dead-letter queue
3 retry attempts logged with timestamps. Dead-letter queue entry present with full scored payload. No data silently discarded.
E04
Google Sheets scorecard tab deleted or renamed
Escalation Agent attempts to write to a Google Sheets tab that no longer exists (tab deleted by a user)
Agent logs a Sheets write error with the expected tab name, does not crash, raises a configuration alert to the monitoring channel
Error log contains expected tab name. Monitoring alert sent. Airtable record still updated. No unhandled exception.
E05
Ops manager does not respond to Slack approval prompt within 24 hours
Flagged vendor alert sent to Slack test channel; no approval or rejection action taken for simulated 24-hour window
Escalation Agent sends a single reminder Slack message after 24 hours. Email draft remains unsent. No further automated action taken after reminder.
Reminder Slack message sent exactly once after 24-hour window. Gmail draft remains in draft state. No third automated message sent.
E06
Scoring threshold environment variable changed mid-month
Threshold variable updated from 70 to 60 while 3 vendor records already scored this month under the old threshold
New events scored under the new threshold immediately. Historical records retain their original scores. No retroactive rescoring occurs. Change logged.
Events after variable change scored at threshold 60. Pre-change records unaltered. Configuration change log entry present with old and new values and timestamp.
Test and QA PlanPage 3 of 4
FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: Use an internal test vendor address for any email send steps. Do not use a real supplier email address. After the production run is complete, void or delete the test vendor record in Xero, Airtable, and Google Sheets, and confirm the test Slack message is acknowledged as a test in the ops channel. Do not proceed with Phase 3 until Phase 1 and Phase 2 have fully passed and all credentials have been rotated from sandbox to production by the FullSpec team.
ID
Scenario
Input
Expected Output
Pass Criteria
T23
Full production run: Xero trigger to Airtable record creation
A real invoice is logged in the live Xero account for a test vendor entry (vendor_name = TEST-VENDOR-QA). PO reference pre-created and standardised in Xero.
Data Collection Agent fires within 60 seconds of invoice creation. Structured record created in live Airtable base with all five required fields: vendor_id, delivery_date_variance, invoice_accuracy, billed_amount, ordered_amount.
Airtable record present within 60 seconds. All five fields populated and correctly typed. No error log entries. Automation platform execution log shows successful run.
T24
Full production run: Scoring Agent fires and scorecard updates
Airtable record from T23 present; scoring threshold set to agreed production value (confirmed with ops manager prior to Phase 3)
Performance Scoring Agent scores TEST-VENDOR-QA, writes score to Airtable record, updates Google Sheets live scorecard row with score, status, and summary paragraph.
Airtable record shows numeric score, status (clear or flagged per injected data), and summary paragraph. Google Sheets live scorecard row updated. Score consistent with scoring model documented in Developer Handover Pack.
T25
Full production run: Escalation path triggered and Slack alert received
T24 score deliberately set below threshold by adjusting TEST-VENDOR-QA input data to produce a flagged outcome
Slack alert posted to live ops channel within 5 minutes of flagged status being written. Message contains vendor name (TEST-VENDOR-QA), score, triggering event description, and Airtable record link. Gmail draft created in live Gmail account.
Slack message received in live ops channel. All four required fields in message body. Gmail draft visible in live Gmail drafts folder. Email not yet sent.
T26
Error log and audit trail verification
Review automation platform execution logs, Airtable record history, and Google Sheets revision history for the T23 through T25 run
Execution logs show timestamped entries for each agent. Airtable record history shows each write with author = automation. Google Sheets revision history shows automated update. No silent failures or missing log entries.
All three log sources contain entries for the test run. No gaps in the execution chain. Airtable record history author is automation account. Sheets revision timestamp within 5 minutes of Airtable write.
T27
Process owner review and approval: final sign-off
Process owner (ops manager) is walked through the live Airtable record, Google Sheets scorecard, Slack alert, and Gmail draft produced by T23 through T25 by the FullSpec team
Process owner confirms: data matches the vendor event injected in T23; score is consistent with agreed thresholds; Slack alert contains the correct information; Gmail draft is in the correct format and gated correctly on approval. Process owner provides verbal or written confirmation that the automation is ready for live operation. FullSpec records this confirmation in the project record.
Process owner confirms all four output elements correct. FullSpec confirmation recorded with process owner name, date, and outcome (approved or approved with conditions). TEST-VENDOR-QA record voided in Xero, Airtable, and Google Sheets after sign-off. No separate signoff block required.
After T27 is complete and the FullSpec confirmation is recorded, the automation is cleared for live operation. Any conditions noted by the process owner during T27 must be resolved and re-confirmed before the first real vendor event is processed. Contact the FullSpec team at support@gofullspec.com with any questions or issues arising during or after the test phases.
Test and QA PlanPage 4 of 4

More documents for this process

Every document generated for Vendor Performance Tracking.

Launch Plan
Operations · Owner
View
ROI and Business Case
Finance · Owner
View
Process Runbook / SOP
Operations · Owner
View
Developer Handover Pack
Technical · Developer
View
Integration and API Spec
Technical · Developer
View