FS-DOC-06Quality
Test and QA Plan
Time Tracking and Billable Hours Automation
Process
Time Tracking and Billable Hours
Agents under test
Time Capture Agent, Gap Chaser Agent, Reconciliation Agent
Tools under test
Karbon, Slack, Gmail, Xero, FullSpec Automation
Test volume baseline
~30 timesheets/week, ~410 runs/month
Prepared for
[YourCompany.com]
01Testing approach
Testing is structured across three sequential phases. Each phase must reach its pass condition before the next phase begins. Phase 1 validates individual agent logic in isolation using mocked data. Phase 2 validates integrated agent-to-agent flows and edge case handling with sandbox credentials. Phase 3 executes a full end-to-end run in production against a real work period, observed by the process owner before go-live sign-off.
1Phase 1: Unit Testing
Estimated time: 2 to 3 days
Scope
Each agent tested independently: Time Capture Agent, Gap Chaser Agent, Reconciliation Agent
Environment
Isolated sandbox in FullSpec Automation with mocked Karbon, Slack, Gmail, and Xero API responses
Test data
Synthetic timesheet records with pre-defined gap patterns; 10 team member profiles
Pass condition
Every test case for each agent returns the exact expected output with zero unhandled errors; all assertions pass
2Phase 2: Integration and Edge Case Testing
Estimated time: 2 to 3 days
Scope
Agent-to-agent handoffs, API error handling, duplicate detection, and boundary inputs
Environment
Connected sandbox with live Karbon sandbox org, Slack test workspace, and Xero demo company
Test data
Synthetic period close with 15 timesheets; 5 intentional edge case records injected
Pass condition
All integration flows complete without data loss; all edge cases resolve to the defined fallback behaviour; no false nudges sent
3Phase 3: End-to-End Production Test
Estimated time: 1 day (observed run)
Scope
Full pipeline from period-close trigger through to Xero export and partner approval prompt
Environment
Production credentials; real Karbon org; real Slack channels; Xero production org (draft invoices only)
Test data
Live work period with actual team timesheets (~30 records); partner observes the approval step
Pass condition
Timesheet completion rate reaches 98% or above; zero unintended messages sent; Xero draft matches Karbon reconciliation; partner approves the run
Test and QA PlanPage 1 of 4
FS-DOC-06Quality
02Phase 1 unit test cases
Each table below covers one agent. Test IDs are prefixed by agent: TC (Time Capture), GC (Gap Chaser), RA (Reconciliation).
Time Capture Agent
ID
Scenario
Input
Expected output
Pass criteria
TC-01
Standard period open with full calendar data
Karbon work period open event; 10 team members with calendar entries
10 draft timesheets created, each with entries pre-populated from calendar
All 10 drafts created; no entries missing where calendar data existed
TC-02
Team member has no calendar entries for the period
Karbon period open; 1 member with zero calendar events
1 draft timesheet created with all fields empty and a gap flag set to true
Draft created; gap flag is true; no error thrown
TC-03
Duplicate calendar event for same task
Two identical calendar entries for the same client/task slot
Single timesheet entry created; duplicate discarded with a deduplicate log entry
Only one entry per task slot; deduplicate log records the discard
TC-04
Karbon API returns a 429 rate-limit error
Mocked Karbon API 429 response on first call
Agent retries after 30 seconds; succeeds on second attempt; draft created
Retry logic fires; no workflow failure; draft created on retry
TC-05
Period open trigger fires twice in quick succession
Two period-open webhook events within 5 seconds
Only one set of draft timesheets created; second trigger deduplicated
Idempotency key prevents duplicate timesheets
Gap Chaser Agent
ID
Scenario
Input
Expected output
Pass criteria
GC-01
One team member has a gap; others are complete
Draft timesheets: 9 complete, 1 with missing Monday entries
Slack DM sent to the 1 member only, referencing the specific missing day
Exactly 1 Slack message sent; content includes the correct member name and missing day
GC-02
All timesheets are complete, no gaps
Draft timesheets: 10 complete, 0 gaps
No Slack or Gmail messages sent
Zero outbound messages; agent marks the period as gap-free
GC-03
Team member has a very thin entry (under 1 hour logged)
1 timesheet with a 0.25-hour entry for a full workday
Slack nudge sent referencing the thin entry and the specific date
Message sent; content references the date and the low hours logged
GC-04
Slack API returns a 503 error on message send
Mocked Slack 503 on first attempt
Agent falls back to Gmail nudge; email sent to member's address from Karbon record
Gmail message delivered; Slack failure logged; no silent drop
GC-05
Multiple gaps across multiple members
5 members each with at least one missing entry
5 separate Slack DMs, each personalised to the specific gap
5 messages sent; each references only that member's gap, not other members' data
Reconciliation Agent
ID
Scenario
Input
Expected output
Pass criteria
RA-01
All timesheets complete; all entries match a Karbon job
10 filled timesheets; all client/task codes present in Karbon jobs
Reconciled billable hours record created; all entries matched; ready-to-bill flag set
Zero unmatched entries; flag set to ready; Xero draft payload generated
RA-02
One entry has an unrecognised job code
1 timesheet entry with a job code not found in Karbon
Entry flagged as unmatched; partner notified for manual review; remainder reconciled
Unmatched entry isolated; remainder proceeds; review notification sent
RA-03
Xero API token expired
Mocked Xero 401 response
Agent triggers OAuth token refresh; retries export; export succeeds
Token refresh completes; export succeeds on retry; no data loss
RA-04
Timesheets submitted before gap chaser has completed
Reconciliation trigger fires while gap chaser is still active
Agent waits for gap chaser completion signal before proceeding
No premature reconciliation; correct sequencing enforced
RA-05
Zero billable hours recorded for the period
All timesheet entries marked as internal/non-billable
Reconciliation completes with a $0 billable total; partner notified with a summary
Zero-dollar record created; partner notification sent; no Xero draft with $0 line items
Test and QA PlanPage 2 of 4
FS-DOC-06Quality
03Phase 2 integration and edge case tests
Integration tests
ID
Scenario
Agents involved
Expected behaviour
Pass criteria
INT-01
Time Capture to Gap Chaser handoff
Time Capture Agent, Gap Chaser Agent
Gap Chaser receives the draft timesheet payload from Time Capture Agent and begins gap analysis within 60 seconds
Handoff payload validated; Gap Chaser starts with correct data; no manual intervention required
INT-02
Gap Chaser to Reconciliation handoff after gaps resolved
Gap Chaser Agent, Reconciliation Agent
Once all gaps are marked resolved, Reconciliation Agent triggers automatically
Reconciliation starts only after gap resolution; completion event correctly passed
INT-03
Reconciliation to Xero export end state
Reconciliation Agent, Xero
Reconciled hours are written to a Xero draft invoice line by line, mapped to the correct contact and account code
All line items present; contact matches Karbon client; account code is billable services
INT-04
Full pipeline with a gap resolved mid-run
All three agents
Gap Chaser sends nudge; member updates timesheet in Karbon; Reconciliation Agent picks up the updated record
Updated record reflected in reconciliation; original gap entry not duplicated
INT-05
FullSpec Automation workflow execution log
All three agents
Every agent step is logged with timestamp, status, and payload reference in FullSpec Automation
Execution log contains an entry per step; no silent failures; log accessible via FullSpec dashboard
Edge case tests
ID
Scenario
Input condition
Expected behaviour
Pass criteria
EC-01
Team member on leave for the full period
Karbon record shows leave for all days in period
No gap nudge sent; timesheet marked as leave-exempt; reconciliation skips the member
Zero messages sent to leave member; exempt flag visible in run log
EC-02
More than 30 timesheets in a single period (volume spike)
35 timesheet records submitted simultaneously
All 35 processed without timeout; batch processing engaged if applicable
All 35 records processed; no timeout errors; completion within 5 minutes
EC-03
Karbon webhook payload missing required fields
Period-open event with null team_member_id
Agent rejects the record, logs a structured error, and alerts the developer via FullSpec
No partial record created; error log entry present; alert fired
EC-04
Slack DM delivery fails silently (no error, no delivery)
Slack API returns 200 but message not delivered (mocked)
Agent checks delivery confirmation; falls back to Gmail after one failed confirmation check
Gmail fallback triggered; delivery confirmed via Gmail; incident logged
EC-05
Xero invoice contact does not exist
Karbon client has no matching Xero contact
Reconciliation Agent creates a draft with a placeholder contact and flags for manual review
Draft created; flag raised; no export blocked; review task created
04Phase 3 end-to-end production test
Phase 3 runs once against a live work period in production. The process owner (Practice Manager) and Partner observe the run. All agents execute in sequence. No synthetic data is used. The Partner performs the approval step manually as designed.
ID
Test case
Steps to execute
Expected result
Pass criteria
E2E-01
Full pipeline trigger to Xero draft
Allow period-close trigger to fire naturally; do not intervene
All three agents run in sequence; Xero draft invoice created; partner approval prompt delivered via Slack
No manual intervention required from trigger to approval prompt; Xero draft present
E2E-02
Timesheet completion rate measurement
Count timesheets with all entries complete before and after the run
Completion rate at or above 98% after Gap Chaser run
Completion rate 98% or above; confirmed against Karbon records by Practice Manager
E2E-03
Nudge targeting accuracy
Review Slack message log for the run
Only team members with genuine gaps received a Slack DM; no messages to complete members
Zero messages sent to members with no gaps; message content matches the actual gap
E2E-04
Partner approval flow
Partner receives approval prompt and approves via Slack or FullSpec interface
Approval recorded; Reconciliation Agent proceeds to finalise the Xero export
Approval step completed by partner; export status updated to approved in FullSpec log
E2E-05
Xero export accuracy
Compare Xero draft line items against Karbon reconciliation report
Every billable hour in Karbon appears as a line item in Xero with the correct client, hours, and rate
Xero line item count matches Karbon billable entry count; zero missing or duplicated lines
E2E-06
Total run time measurement
Record timestamp from period-close trigger to Xero draft created
Full pipeline completes within 15 minutes for up to 30 timesheets
Elapsed time 15 minutes or less; confirmed in FullSpec execution log
E2E-07
Execution log completeness
Review FullSpec Automation execution log after the run
Every agent step logged with status, duration, and output reference
Log is complete; no missing steps; all statuses show success or a handled exception
Test and QA PlanPage 3 of 4
FS-DOC-06Quality
05Go-live pass criteria
All requirements in the table below must be met before the automation is marked live. Any row with a status of To Confirm must be resolved and re-tested before sign-off. The developer and process owner both confirm against this table at the sign-off meeting.
Requirement
Covered by test IDs
Status
Time Capture Agent creates draft timesheets for all team members at period open
TC-01, TC-02, TC-03, TC-05, E2E-01
To confirm
Duplicate trigger events produce no duplicate timesheets
TC-05, INT-05
To confirm
Gap Chaser Agent sends nudges only to members with genuine gaps
GC-01, GC-02, GC-03, GC-05, E2E-03
To confirm
Slack fallback to Gmail operates correctly when Slack is unavailable
GC-04, EC-04
To confirm
Reconciliation Agent matches all entries to Karbon jobs before export
RA-01, RA-02, INT-03, E2E-05
To confirm
Xero OAuth token refresh operates without workflow interruption
RA-03
To confirm
Agent sequencing is enforced: Reconciliation does not start before gaps are resolved
RA-04, INT-02
To confirm
Leave-exempt members are excluded from all nudge and reconciliation logic
EC-01
To confirm
Pipeline handles a volume of 35 timesheets without timeout
EC-02
To confirm
Partner approval step fires correctly and gates the Xero export
E2E-04
To confirm
Timesheet completion rate reaches 98% or above in production
E2E-02
To confirm
Full pipeline completes within 15 minutes for up to 30 timesheets
E2E-06
To confirm
FullSpec Automation execution log is complete with no silent failures for any run
TC-04, INT-05, E2E-07
To confirm
Xero draft line items match Karbon billable entries with zero discrepancy
RA-01, E2E-05
To confirm
All API credentials stored as environment variables; none hardcoded in workflow
FS-DOC-04 reference; verified by developer
To confirm
A go-live date must not be set until every row above has been tested and the status updated to Passed by the developer. The process owner confirms the production run results independently before co-signing.
06Sign-off
By signing below, each party confirms that the test cases assigned to their scope have been executed, results reviewed, and all go-live pass criteria in Section 05 have been met. The automation is approved to operate in production.
Test and QA PlanPage 4 of 4