FS-DOC-06Quality
Test and QA Plan
Client Onboarding and Engagement Letters
Process
Client Onboarding and Engagement Letters
Build option
Standard (recommended)
Volume
~15 new clients/month (approx. 188 automation runs/month)
Status
Draft, pending sign-off
01Testing approach
Testing is structured in three sequential phases. Each phase must reach its pass condition before the next phase begins. Phase 1 covers individual agent logic in isolation. Phase 2 covers integration between agents and edge-case handling. Phase 3 covers a full end-to-end production run against live credentials and real data volumes.
1Phase 1: Unit testing
Scope: each agent tested in isolation against mock data
Scope
Engagement Letter Agent, Document Collection Agent, and Onboarding Setup Agent tested independently with stubbed inputs and mocked API responses
Environment
Development environment; all external API calls replaced with recorded fixtures or sandbox endpoints (PandaDoc sandbox, Gmail test account, Karbon staging)
Tools under test
Google Docs, PandaDoc (sandbox), Gmail (test account), Karbon (staging), FullSpec Automation (dev workspace)
Data
Synthetic client records only; no live personal data
Pass condition
Every test case in Section 02 returns the expected output and no agent throws an unhandled exception
2Phase 2: Integration and edge case testing
Scope: agent-to-agent handoffs and failure paths
Scope
End-to-end agent chain tested with real API calls; edge cases including missing fields, duplicate triggers, AML flag, and reminder loops
Environment
Staging environment; live API credentials scoped to test workspaces; PandaDoc test template; Karbon staging workspace
Tools under test
Full stack: FullSpec Automation orchestration, PandaDoc, Gmail, Karbon; webhook delivery confirmed
Data
Synthetic records with deliberate gaps and error conditions injected
Pass condition
All integration paths complete or fail gracefully with correct error routing; all edge cases in Section 03 produce the expected branch or error state; no data leakage across test runs
3Phase 3: End-to-end production test
Scope: full live run on production credentials with a pilot client
Scope
One complete onboarding cycle executed on production with a real (internal or opted-in pilot) client record
Environment
Production environment; live credentials; monitoring and alerting active
Tools under test
Production Google Docs, PandaDoc, Gmail, Karbon, FullSpec Automation
Data
One real or internal pilot client record; process owner observes each stage
Pass condition
All Phase 3 test cases in Section 04 pass; engagement letter delivered and signed; documents collected; Karbon record created; total elapsed time under 4 days; zero manual interventions required outside the compliance review step
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 unit test cases
Each table below covers one agent. All tests run in the development environment against stubbed or sandbox API responses. Test IDs use the prefix U (unit).
Engagement Letter Agent: triggered when a client is marked won in the CRM. Connects to Google Docs and PandaDoc.
ID
Scenario
Input
Expected output
Pass criteria
U-01
Happy path: valid client record triggers letter generation
CRM webhook payload with client name, email, service type, and engagement date all present
Google Docs template populated; PandaDoc document created and send-for-signature request returned with document ID
HTTP 200 from PandaDoc; document ID stored in workflow state; no field left blank in the letter
U-02
Missing client email
CRM payload with client name and service type but no email address field
Workflow halts; error event logged; notification sent to practice manager with client name and missing field
Workflow does not proceed to PandaDoc; error log entry created with payload reference
U-03
Missing service type
CRM payload with email present but service_type field null
Workflow halts at template merge step; error logged with field name
No document created in PandaDoc; error state captured in FullSpec Automation run log
U-04
Duplicate trigger for same client within 10 minutes
Two identical CRM webhooks fired within 5 minutes of each other
Second webhook deduplicated; only one PandaDoc document created
PandaDoc document count for client equals 1; deduplication log entry present
U-05
PandaDoc API returns 429 (rate limit)
Valid client payload; PandaDoc sandbox returns 429 on first attempt
Workflow retries with exponential back-off (3 attempts, 30/60/120 s); succeeds on retry
Document eventually created; retry count logged; no duplicate document created
Document Collection Agent: triggered when the engagement letter is signed. Connects to Gmail and FullSpec Automation.
ID
Scenario
Input
Expected output
Pass criteria
U-06
Happy path: PandaDoc signed webhook received
PandaDoc webhook payload with status: completed and recipient email
Initial document request email sent via Gmail; checklist of required documents initialised in FullSpec Automation tracking state
Gmail send API returns 200; tracking record created with client ID and timestamp
U-07
No response after 48 hours: first reminder
Tracking record shows no document upload after 48-hour window
Reminder email sent to client from Gmail with original request attached
Second Gmail message in thread; reminder timestamp recorded
U-08
No response after 96 hours: second reminder with escalation flag
Tracking record shows no upload after 96 hours from initial send
Second reminder sent; escalation flag set to true in tracking state; practice manager notified
Escalation flag present in state; Gmail message count equals 3; practice manager notification delivered
U-09
Partial document upload (ID present, proof of address missing)
Upload event with id_verified: true, proof_of_address: false
Follow-up email requesting only the missing document; checklist updated to reflect partial completion
Gmail message references only the outstanding item; checklist state updated correctly
U-10
All documents received: agent marks collection complete
Upload events covering all required document types received
Collection status set to complete; trigger fired to Onboarding Setup Agent
FullSpec Automation state shows all items checked; downstream trigger event logged
Onboarding Setup Agent: triggered when documents and compliance checks are complete. Connects to Karbon and FullSpec Automation.
ID
Scenario
Input
Expected output
Pass criteria
U-11
Happy path: AML clear, all documents present, Karbon record created
Completion event from Document Collection Agent with aml_clear: true
New client record created in Karbon with mapped fields; contact, entity, and job records all created; team notified via FullSpec Automation
Karbon API returns 201 for each record type; no field mapping error; notification dispatched
U-12
AML flag raised: route to compliance review
Completion event with aml_clear: false
Workflow branches to manual compliance review step; practice manager alerted; Onboarding Setup Agent pauses
No Karbon record created; compliance review task created; alert delivered to practice manager
U-13
Karbon API returns 500 on contact creation
Valid completion event; Karbon staging returns 500 on POST /contacts
Retry 3 times with 60 s delay; if still failing, alert sent to developer and practice manager; workflow halted gracefully
No partial record left in Karbon; error logged with full response body; alert delivered
U-14
Duplicate Karbon record guard: client already exists
Completion event where client email already exists in Karbon
Workflow detects duplicate via GET /contacts?email check; skips creation; logs warning; notifies practice manager
No duplicate record created; warning log entry present; practice manager notification sent
U-15
Ready notification dispatched on successful setup
All Karbon records confirmed created
Ready notification sent to practice manager and onboarding admin via FullSpec Automation
Notification delivered to both recipients; Karbon record IDs included in notification body
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 integration and edge case tests
Phase 2 tests run in the staging environment with live API credentials scoped to test workspaces. Integration tests validate agent-to-agent handoffs. Edge case tests validate system behaviour under abnormal conditions. Test IDs use prefixes I (integration) and E (edge case).
Integration tests: agent-to-agent handoffs across the full automation chain.
ID
Scenario
Input
Expected output
Pass criteria
I-01
Trigger to Engagement Letter Agent handoff
CRM webhook for new client won fires in staging
Engagement Letter Agent activates; Google Docs template merged; PandaDoc staging document created and send-for-signature dispatched
PandaDoc staging document ID returned and stored; end-to-end latency under 60 seconds
I-02
PandaDoc signed webhook to Document Collection Agent handoff
PandaDoc signed event fires against staging FullSpec Automation webhook endpoint
Document Collection Agent activates within 30 seconds; initial document request email sent from Gmail test account
Gmail API confirms message sent; Document Collection Agent state initialised with correct client reference
I-03
Document Collection Agent completion to Onboarding Setup Agent handoff
All required documents marked received in FullSpec Automation tracking state
Onboarding Setup Agent activates; Karbon staging records created across contact, entity, and job types
Karbon staging API confirms all three record types created with correct field values
I-04
Compliance branch: AML flag routes to manual review and resumes
AML check returns flag; practice manager manually marks review complete in FullSpec Automation
Workflow resumes from compliance review node; Onboarding Setup Agent activates and creates Karbon records
Karbon records created only after manual approval; no bypass of compliance gate possible
I-05
Full chain: CRM trigger to ready notification in under 4 days (simulated fast clock)
All events fired in sequence with simulated timestamps compressed to minutes
All agents fire in correct order; Karbon records created; ready notification delivered
No skipped steps; all state transitions logged; notification contains correct client name and Karbon record IDs
Edge case tests: failure modes, boundary conditions, and data quality scenarios.
ID
Scenario
Input
Expected output
Pass criteria
E-01
CRM sends malformed JSON payload
Webhook payload with invalid JSON structure
Workflow rejects payload at ingestion; 400 returned to sender; error logged with raw payload excerpt
No downstream agent triggered; error log entry with timestamp and payload hash
E-02
PandaDoc webhook fires twice for same signature event
Two identical document.completed webhooks within 2 minutes
Second event deduplicated; Document Collection Agent triggered exactly once
Document collection initialised once; deduplication count incremented in state
E-03
Gmail daily send limit approached (500 messages/day threshold)
Document Collection Agent attempts to send reminder when daily Gmail send count is at 490
Warning logged; send proceeds if under hard limit; alert sent to developer if limit reached
No send failure without prior warning; developer alert triggered at 500; no message lost
E-04
Karbon API token expires mid-run
Valid completion event; Karbon OAuth access token expired before POST /contacts
Workflow attempts token refresh using refresh token; retries request; succeeds or logs failure with refresh error details
Token refresh attempted before retry; if refresh fails, error logged with OAuth error code; no silent failure
E-05
Client submits wrong document type (e.g., utility bill instead of passport)
Upload event with document_type: utility_bill where passport required
Document flagged as incorrect type; follow-up email sent requesting correct document; checklist not marked complete for ID item
Checklist ID item remains unchecked; client email references the correct document type required
E-06
Onboarding triggered for a client already fully onboarded in Karbon
CRM webhook for client whose email already exists in Karbon as a fully active contact
Duplicate guard fires; workflow halted; practice manager notified with existing Karbon record link
No records created or overwritten; notification contains existing record ID
E-07
FullSpec Automation orchestration service unavailable for 5 minutes
All inbound webhooks arrive during a 5-minute service outage window
Webhooks queued by ingestion layer; replayed in order on service recovery; no events dropped
All queued events processed in order after recovery; no duplicate processing; queue drain logged
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
04Phase 3 end-to-end production test
Phase 3 runs on production infrastructure with live credentials. One pilot client record is used, either an internal test record or an opted-in client. The process owner observes each stage in real time. All three agents must complete their actions without manual intervention except at the designated compliance review gate. Test IDs use the prefix P (production).
ID
Scenario
Steps observed
Expected outcome
Pass criteria
P-01
CRM trigger fires on production and Engagement Letter Agent activates
Practice manager marks pilot client as won in production CRM; observe FullSpec Automation run log
Engagement Letter Agent activates within 60 seconds; Google Docs letter generated using correct client name, service type, and date; PandaDoc document created and send-for-signature email delivered to pilot client inbox
Pilot client receives PandaDoc email within 2 minutes of CRM update; letter content matches client record exactly; no template merge errors
P-02
Engagement letter signed and Document Collection Agent activates
Pilot client signs document via PandaDoc link; observe FullSpec Automation webhook receipt
PandaDoc signed webhook fires to production endpoint; Document Collection Agent activates within 60 seconds; initial document request email sent from production Gmail
Gmail message received in pilot client inbox; email content lists correct required documents; FullSpec Automation tracking state initialised
P-03
Document reminder cycle runs correctly
Hold document submission for 48 hours (or simulate by advancing tracking timestamp in state); observe Gmail outbox
First reminder email sent at 48-hour mark; escalation flag set at 96-hour mark; practice manager receives escalation alert
Reminder email arrives in pilot client inbox; escalation alert delivered to practice manager; timestamps accurate to within 5 minutes
P-04
All documents submitted and AML clear path executes
Pilot client uploads all required documents; AML check returns clear result
Document Collection Agent marks collection complete; Onboarding Setup Agent activates; Karbon production contact, entity, and job records created
Karbon records visible in production workspace within 3 minutes of document completion event; all required fields populated correctly
P-05
AML flag path: compliance review gate holds and resumes correctly
Inject AML flag condition on a second test run (internal record); practice manager approves in FullSpec Automation
Workflow pauses at compliance gate; Karbon records not created until manual approval; on approval, Onboarding Setup Agent resumes and creates records
No Karbon record present before approval; record created within 3 minutes of approval action; audit trail shows manual approval event with timestamp and approver identity
P-06
Ready notification delivered to team
Observe notification delivery after Karbon setup completes on the clean AML path
Practice manager and onboarding admin both receive ready notification via FullSpec Automation; notification contains client name and Karbon record reference
Both recipients confirmed in notification delivery log; notification content accurate; delivered within 2 minutes of Karbon record creation
P-07
Full elapsed time within target
Record timestamp from CRM trigger to ready notification across the full production run (excluding 48-hour document wait)
Total automation processing time (excluding client response time) under 10 minutes; overall client-facing time to ready under 4 days under normal conditions
Processing time logged in FullSpec Automation run summary; process owner confirms experience matches expected 4-day target for client-facing steps
05Go-live pass criteria
All requirements below must be confirmed before the automation is approved for go-live with real client volume. The Status column reflects the state at time of sign-off. Any item showing amber must be resolved or formally accepted with a written risk note before launch proceeds.
Requirement
Test IDs
Status
All Phase 1 unit tests pass with no unhandled exceptions across all three agents
U-01 to U-15
To confirm
All Phase 2 integration tests pass and agent-to-agent handoffs complete without manual intervention (excluding compliance gate)
I-01 to I-05
To confirm
All Phase 2 edge case tests produce the expected error state or graceful failure with correct routing
E-01 to E-07
To confirm
All Phase 3 production tests pass; ready notification delivered for pilot client within 4-day target
P-01 to P-07
To confirm
PandaDoc deduplication confirmed: no duplicate documents generated under rapid trigger conditions
U-04, E-02
To confirm
Karbon duplicate guard confirmed: no duplicate records created for existing clients
U-14, E-06
To confirm
AML compliance gate confirmed: Karbon records cannot be created without manual approval when flag is raised
U-12, I-04, P-05
To confirm
Retry and back-off logic confirmed for PandaDoc, Gmail, and Karbon API failure scenarios
U-05, U-13, E-04
To confirm
Error notifications confirmed: developer and practice manager alerted on all critical failure states
U-02, U-03, U-08, U-13, E-03, E-06
To confirm
Production monitoring and alerting active in FullSpec Automation before first live client run
P-01 to P-07 (observed)
To confirm
Process owner has reviewed and approved the production test run output (Section 04)
P-01 to P-07
To confirm
Developer has confirmed all credentials rotated from test to production values and test data purged
All phases
To confirm
Any go-live pass criterion that cannot be confirmed before the planned launch date must be escalated to the process owner. A formal risk acceptance note signed by the process owner is required before proceeding with an open item.
06Sign-off
Both parties below must sign before the automation is considered approved for production use at full client volume. Sign-off confirms that all go-live pass criteria in Section 05 have been reviewed and either confirmed or formally accepted with a risk note.
Test and QA PlanPage 4 of 4