FS-DOC-06Quality
Test and QA Plan
Regulatory Change Monitoring
[YourCompany.com] · Legal Department · Prepared by FullSpec · [Today's Date]
This document defines the complete test and quality assurance programme for the Regulatory Change Monitoring automation. It covers all three test phases: unit testing of individual agents in sandbox, integration and edge-case testing of agent handoffs, and end-to-end production validation. The FullSpec team runs Phases 1 and 2 independently. The process owner joins Phase 3 to confirm production behaviour and record formal approval. No phase begins until the previous phase has fully passed.
01Testing approach
Testing is structured as three sequential phases. Each phase has a defined pass condition, and no phase may begin until every test case in the preceding phase has achieved a passing status. This ensures defects discovered at the unit level are resolved before integration dependencies are validated, and that production credentials are never touched until the build has passed in sandbox.
1Phase 1: Unit Testing
Each agent tested in isolation against sandbox credentials
Scope
Individual agent logic: trigger firing, data parsing, relevance scoring, summary generation, Slack formatting, and digest assembly. Each agent is tested independently with no dependency on upstream agents.
Environment
Sandbox only. Feedly development token, Airtable test base, Slack test workspace, Gmail alias, Notion test workspace.
Credential rule
No production API keys, OAuth tokens, or live Airtable bases are used in Phase 1. All credentials must be sandbox or test-tier.
Pass condition
All test cases T01 through T15 return the expected output with zero critical failures. Amber warnings may be noted but must be resolved before Phase 2.
Who runs it
FullSpec team only. No process owner involvement required.
2Phase 2: Integration and Edge-Case Testing
Agent handoffs and failure modes tested end to end in sandbox
Scope
Data flow across agent boundaries: Source Monitor to Relevance and Summary, and Relevance and Summary to Alert and Digest. Edge cases include duplicate items, missing fields, unavailable tools, and human review timeout scenarios.
Environment
Sandbox only, with intentional fault injection (network timeouts, malformed payloads, missing Airtable fields) to validate error handling.
Credential rule
Sandbox credentials only. Fault injection must not target production endpoints.
Pass condition
All integration test cases T16 through T22 pass, and all edge-case tests E01 through E07 pass or are documented with an accepted mitigation. No unhandled exceptions.
Who runs it
FullSpec team only. Results are shared with the process owner before Phase 3 begins.
3Phase 3: End-to-End Production Test
Full production run observed and approved by the process owner
Scope
A complete run of the live automation against production credentials, covering the full journey from Feedly fetch through Airtable logging, Slack routing, and Gmail digest delivery.
Environment
Production. Live Feedly account, production Airtable base, live Slack workspace, production Gmail sender. Test records are voided after each case.
Credential rule
Production credentials active. A dedicated internal test address must be used for all email cases. No external recipients receive test output.
Pass condition
All production test cases T23 through T27 pass, error logs are clean, and the process owner has reviewed and approved outputs. Owner sign-off is recorded in the FullSpec confirmation (test case T27).
Who runs it
FullSpec team executes T23 through T26. The process owner joins for T27 to review outputs and record approval.
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 — unit test cases
Sandbox credentials only: all Phase 1 tests must use sandbox API tokens, a Feedly development account, the designated Airtable test base (not the production base), a Slack test workspace, and a Gmail alias configured for testing. Do not connect any production credential at this phase. If a test requires a real feed response, replay a saved fixture rather than calling a live endpoint.
Source Monitor Agent (estimated build: 10 hours, Moderate complexity)
ID
Scenario
Input
Expected output
Pass criteria
T01
Happy path: daily schedule fires and fetches new items from Feedly
Scheduled trigger fires; Feedly sandbox returns 5 new articles from 3 configured feeds
5 deduplicated item records queued for downstream classification; Airtable intake table receives 5 rows with source URL, publication date, and raw title
All 5 rows present in Airtable test base within 30 seconds of trigger; no duplicate row IDs
T02
Deduplication: item already seen in a prior run is not reprocessed
Feedly returns 3 items; 1 item ID matches an existing Airtable record
Only 2 new rows written to Airtable; duplicate item skipped with a deduplicate log entry
Airtable row count increases by 2 only; deduplication log confirms 1 item suppressed
T03
Empty feed: Feedly returns zero new items for a source
All configured feeds return 0 new items since last poll
No rows written to Airtable; run completes with status 'no new items'
Airtable row count unchanged; run log records 'no new items' without error
T04
Feedly API timeout: source returns a 504 gateway timeout
Feedly API call times out after 10 seconds
Retry logic fires up to 3 times; after final failure the run logs an error and sends an internal alert to the FullSpec monitoring channel
Error logged with timestamp and source ID; no partial rows written; alert visible in monitoring channel
T05
Feedly API unauthorised: expired or revoked OAuth token
Feedly returns HTTP 401
Run halts immediately; error record written with code 401; FullSpec monitoring notified
No Airtable rows written; error code 401 present in run log; monitoring alert fired
Relevance and Summary Agent (estimated build: 16 hours, Complex)
ID
Scenario
Input
Expected output
Pass criteria
T06
Happy path: item scores above relevance threshold
Item with jurisdiction tag matching configured profile, topic matching keyword list, relevance score computed at 0.82 (threshold 0.65)
Plain-language summary generated (2-3 sentences), effective date extracted, suggested owner resolved via routing table, Airtable record updated with all fields
Airtable record contains non-null: summary, effective_date, assigned_owner, relevance_score >= 0.65
T07
Item scores below relevance threshold and is silently archived
Item with jurisdiction outside configured profile; computed relevance score 0.31
Item archived with status 'below_threshold'; no summary generated; no Slack alert triggered
Airtable record status field set to 'archived'; no downstream Slack or Gmail events fired
T08
Borderline item: score within configurable review band (0.55 to 0.64)
Item with ambiguous jurisdiction tag; score 0.60
Item flagged as 'borderline'; routed to human review queue in Airtable; compliance lead notified via Slack with a review prompt
Airtable status field set to 'pending_review'; Slack message delivered to compliance lead channel within 60 seconds
T09
Missing effective date in source content
Item body contains no parseable date in any standard format
effective_date field set to null; summary generated with note 'effective date not stated'; record written with warning flag
Airtable effective_date is null; warning_flag field is true; run does not error out
T10
No matching owner in routing table for item topic category
Item topic category is 'environmental_regulation'; routing table has no entry for this category
Item written to Airtable with assigned_owner set to 'UNROUTED'; fallback Slack alert sent to the default compliance channel
assigned_owner field value is 'UNROUTED'; fallback Slack alert contains item title and a note that routing failed
T11
Classification prompt returns malformed JSON from AI layer
AI layer returns a truncated or syntactically invalid JSON response
Agent catches parse error; retries once; if second attempt also fails, item is placed in error queue in Airtable and FullSpec monitoring is notified
Item present in Airtable error queue; monitoring alert fired; no partial field writes
Alert and Digest Agent (estimated build: 12 hours, Moderate complexity)
ID
Scenario
Input
Expected output
Pass criteria
T12
Happy path: Slack alert posted for high-relevance item
New Airtable record with status 'relevant', assigned_owner mapped to a Slack user ID, relevance_score 0.80
Structured Slack message posted to the correct owner channel; message includes item title, summary snippet, effective date, and a link to the Airtable record
Slack message visible in correct channel within 60 seconds; all four required fields present in message body
T13
Weekly digest email assembled and sent on Monday schedule
Monday 08:00 trigger fires; Airtable contains 7 relevant records logged in the prior 7 days
HTML digest email sent to configured recipients via Gmail; email lists all 7 items with title, summary, effective date, and assigned owner
Gmail sent folder confirms delivery; email body contains all 7 items; subject line contains the current week date range
T14
Overdue follow-up reminder: action item not acknowledged within configured window
Airtable record status remains 'pending_acknowledgement' after 5 days (configured threshold)
Reminder Slack message sent to assigned owner; reminder also logged in Airtable audit field
Slack reminder delivered; Airtable reminder_sent_at field populated with timestamp
T15
Slack API rate limit exceeded during batch alert posting
Agent attempts to post 15 Slack messages within 1 minute, hitting the Slack tier-1 rate limit
Agent applies exponential backoff; all messages delivered within 5 minutes with no message lost; rate limit warning logged
All 15 Slack messages confirmed delivered; rate_limit_warning present in run log; no messages dropped
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 — integration and edge case tests
Integration tests validate the data handoffs between agents. Each test fires a real trigger in the sandbox environment and traces the payload as it crosses agent boundaries. All sandbox credentials apply. Edge-case tests use intentional fault injection to confirm the automation degrades gracefully rather than silently failing.
ID
Scenario
Input
Expected output
Pass criteria
T16
End-to-end happy path: Source Monitor hands off to Relevance and Summary Agent
Source Monitor fetches 4 new items from Feedly sandbox; passes all 4 to classification queue
All 4 items classified within 90 seconds; 2 above threshold, 1 borderline, 1 below threshold; Airtable records updated accordingly
Airtable shows 2 'relevant', 1 'pending_review', 1 'archived'; no items lost between agents
T17
End-to-end: Relevance and Summary Agent hands off to Alert and Digest Agent
2 relevant Airtable records written by Relevance and Summary Agent with valid assigned_owner values
Alert and Digest Agent fires within 2 minutes; 2 Slack messages posted to correct channels; records updated with alerted_at timestamp
Both Slack messages delivered; alerted_at field populated; no duplicate alerts
T18
Full three-agent chain completes without manual intervention for a clear-pass item
Single high-relevance item enters at Source Monitor; no borderline flag; routing table match found
Item travels Source Monitor > Relevance and Summary > Alert and Digest; Slack alert sent; Airtable fully populated; total elapsed time under 3 minutes
Elapsed time under 180 seconds; all Airtable fields populated; Slack alert confirmed; zero human-review prompts triggered
T19
Borderline item pauses the chain at Relevance and Summary; chain resumes after human approval
Borderline item (score 0.58) enters classification; compliance lead approves via Airtable status update
Chain paused at 'pending_review'; on status change to 'approved' Alert and Digest Agent fires and sends Slack alert
Slack alert sent only after status updated to 'approved'; no alert sent during pending state; status change timestamp logged
T20
Notion write for summary document: Relevance and Summary Agent creates Notion page
Relevant item with summary text and effective date; Notion integration active in sandbox
Notion page created under the configured compliance database with title, summary body, source URL, and effective date fields populated
Notion page present in sandbox database; all four required properties populated; page URL stored in Airtable record
T21
Airtable write failure mid-chain: Airtable returns HTTP 503
Source Monitor fetches items; Airtable returns 503 on write attempt
Agent retries up to 3 times with exponential backoff; after final failure items are queued in the automation platform's internal retry buffer; FullSpec monitoring alerted
Monitoring alert fired; items present in retry buffer; no items permanently lost; Airtable write succeeds on retry once service recovers
T22
Gmail send failure: SMTP authentication error on digest send
Monday digest trigger fires; Gmail SMTP returns authentication failure
Digest send fails gracefully; error logged with SMTP error code; FullSpec monitoring notified; digest content preserved in Airtable for manual resend
Error code logged; digest content intact in Airtable; monitoring alert fired; no partial email delivered
Edge-case tests below use intentional fault injection. These are run after T16 through T22 have passed.
ID
Edge-case scenario
Fault injected
Expected behaviour
Pass criteria
E01
Duplicate item submitted twice within the same polling cycle
Same Feedly article ID injected into the queue twice in rapid succession
Deduplication logic suppresses the second instance; only one Airtable row written; deduplication counter incremented in run log
Airtable row count increases by 1 only; duplicate suppression logged
E02
Item with missing source URL field
Feedly fixture item has source_url set to null
Item written to Airtable with source_url as null and a data_quality_warning flag; not discarded; downstream agents continue processing
Record present in Airtable; data_quality_warning true; no agent crash
E03
Item body is empty string (blank publication)
Feedly fixture item has content field set to empty string
Classification agent skips AI summarisation; item written to Airtable with status 'empty_content'; compliance lead notified via Slack
Status field is 'empty_content'; Slack notification delivered; no AI prompt sent with empty content
E04
Feedly service fully unavailable for an entire polling cycle
Feedly API returns 503 for all requests in a single scheduled run
All retries exhausted; run marked as failed; FullSpec monitoring alerted; next scheduled run attempts normally without skipping
Monitoring alert fired with 'Feedly unavailable' message; next run executes on normal schedule; no items permanently skipped
E05
Slack workspace unavailable when alert is due
Slack API returns 503 during alert posting
Alert queued internally; delivery retried with exponential backoff for up to 30 minutes; if still unavailable the alert detail is emailed to the assigned owner as fallback
Fallback email delivered if Slack remains unavailable at 30-minute mark; alert eventually delivered to Slack when service recovers
E06
Human review step times out: borderline item not actioned within 48 hours
Borderline item placed in 'pending_review' status; no human action taken for 48 hours (simulated by backdating the created_at timestamp)
Escalation reminder sent to compliance lead via Slack and email; item status updated to 'escalated'; original assigned reviewer re-notified
Slack and email escalation messages delivered; status field updated to 'escalated'; timestamp of escalation logged in Airtable
E07
Routing table lookup returns multiple matching owners for a single topic category
Item topic category matches two entries in the routing table (configuration error scenario)
Agent routes to the first matched owner and logs a routing_conflict warning in Airtable; FullSpec monitoring alerted to review routing table
Slack alert sent to first matched owner only; routing_conflict warning present in Airtable record; monitoring alert fired
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 — end-to-end production test
Important before running Phase 3: use a dedicated internal test email address (not an external client or management address) for all Gmail digest cases. All Slack alerts during testing must be posted to a designated test channel, not live team channels. Any Airtable records created during Phase 3 must be tagged with 'TEST' in the record_type field and voided (deleted or archived) immediately after each test case is confirmed as passed. Do not run Phase 3 until Phase 1 and Phase 2 have fully passed and the FullSpec team has confirmed the build is production-ready in writing.
ID
Scenario
Production action
Expected output
Pass criteria
T23
Full production run: live Feedly fetch, classification, and Airtable logging
Trigger fires against production Feedly account; live feeds return real articles from configured regulatory sources
New items fetched, deduplicated, classified, and written to the production Airtable compliance register with all required fields populated; items tagged 'TEST' in record_type
Production Airtable base contains new rows with relevance_score, summary, assigned_owner, effective_date, and source_url all populated; record_type field is 'TEST'
T24
Live Slack alert posted to correct owner channel via production workspace
High-relevance item from T23 triggers Alert and Digest Agent against production Slack
Structured Slack message appears in the designated test channel (not a live team channel); message contains all required fields; owner is tagged correctly
Slack message confirmed in test channel; title, summary, effective date, and Airtable record link all present; correct owner tagged
T25
Live Gmail digest email delivered to internal test address
Digest trigger fires manually against production Gmail sender; recipients set to internal test address only
HTML digest email received at internal test address within 5 minutes; email lists all items from T23 run; subject line includes correct date range
Email received at test address; delivery confirmed in Gmail sent folder; all T23 items present in digest body
T26
Error log verification: confirm production run log is clean and all audit fields populated
Review the automation platform run log and Airtable audit fields after T23 to T25
Run log shows zero unhandled exceptions, zero missing retries, and all items accounted for; Airtable audit fields (created_at, alerted_at, record_type) fully populated
Run log contains no ERROR-level entries; all Airtable records from T23 have created_at and alerted_at populated; no orphaned items in any queue
T27
Process owner review and approval: owner reviews all Phase 3 outputs and records formal sign-off
Process owner accesses the production Airtable base, reviews the Slack test-channel messages, and inspects the digest email received at the internal test address
Process owner confirms: Airtable records are correctly structured; Slack alerts contain all required information and are routed correctly; digest email is accurately formatted; overall automation behaviour matches the agreed specification
Process owner records approval by completing the FullSpec confirmation form (provided by the FullSpec team at support@gofullspec.com); approval timestamp logged in the FullSpec project record; all TEST-tagged Airtable records voided after approval is confirmed
Once T27 is confirmed, the FullSpec team will archive all test records, remove the TEST tag filter, and set the automation schedule to its live production cadence. Contact support@gofullspec.com if any Phase 3 case requires a rerun or if the process owner needs to flag a discrepancy before final approval.
Test and QA PlanPage 4 of 4