Back to SLA Monitoring & Breach Alerting

Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

SLA Monitoring & Breach Alerting

[YourCompany.com] · Customer Support Department · Prepared by FullSpec · [Today's Date]

This document defines the complete test and quality assurance plan for the SLA Monitoring and Breach Alerting automation. It covers three sequential phases: unit testing of individual agents, integration and edge-case testing across agent handoffs and failure scenarios, and a final end-to-end production run with process owner sign-off. The FullSpec team runs Phases 1 and 2 in full; the process owner joins Phase 3 to validate production behaviour and record approval. No phase begins until the previous phase achieves a full pass across all test cases.

01Testing approach

Testing is structured across three phases that must be completed in strict sequence. No phase begins until every test case in the preceding phase has achieved its defined pass condition. This gates unstable builds from reaching production and ensures each layer of the automation is independently verified before being tested together.

Phase 1: Unit Testing

Each agent tested in isolation against sandbox data

Scope

Individual agent logic: SLA Monitor Agent (Zendesk poll, HubSpot tier lookup, business-hours calculation, ticket classification) and Alert Dispatch Agent (Slack alert, PagerDuty escalation, Zendesk note, HubSpot breach log, Gmail summary) tested independently.

Environment

Sandbox only. All connections point to Zendesk sandbox, HubSpot sandbox, Slack test workspace, PagerDuty dev account, and Gmail test account.

Credential rule

No production API credentials may be used. All tokens and keys must be sandbox-issued. Credentials are stored in the automation platform's secret manager under the sandbox namespace.

Pass condition

All cases T01 through T14 return expected outputs with zero failures. Any failure blocks progression to Phase 2.

Who runs it

FullSpec team only. Process owner is not required for Phase 1.

Phase 2: Integration and Edge-Case Testing

Agent handoffs and failure modes tested end-to-end in sandbox

Scope

Full agent-to-agent handoff from SLA Monitor Agent to Alert Dispatch Agent, covering the classified ticket list payload and all downstream routing decisions. Edge cases include duplicate tickets, missing HubSpot client IDs, tool unavailability, and human escalation timeout scenarios.

Environment

Sandbox only. The integration environment mirrors production field naming and SLA tier configuration but uses synthetic ticket data only.

Credential rule

Sandbox credentials continue throughout Phase 2. No production tokens are introduced at this stage.

Pass condition

All integration cases T15 through T20 and all edge-case cases E01 through E06 pass. Any failure or unexpected routing must be resolved before Phase 3 is scheduled.

Who runs it

FullSpec team only. Process owner may observe but is not required.

Phase 3: End-to-End Production Test

Live environment validation with process owner present

Scope

Full production run covering the complete automation flow from scheduled Zendesk poll through to Slack alert, PagerDuty escalation, Zendesk internal note, HubSpot breach log, and Gmail weekly summary. Error log verification and final process owner approval included.

Environment

Production. All credentials are production-issued. Test tickets are created using an internal test address and voided after each case.

Credential rule

Production API credentials are used for the first time in Phase 3. All sandbox credentials must be rotated out before Phase 3 begins. FullSpec confirms credential swap before the first test case runs.

Pass condition

All cases T21 through T25 pass, including the final process owner review case (T25). Owner approval is recorded in the FullSpec confirmation record.

Who runs it

FullSpec team leads execution. Process owner (Support Lead and Support Manager) must be present and available to confirm alerts received, review error logs, and record final approval.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only. All Phase 1 tests must be run against the Zendesk sandbox, HubSpot sandbox, Slack test workspace, PagerDuty dev account, and Gmail test account. Do not connect or test against any production endpoint during this phase. If a production credential is accidentally used, invalidate it immediately and reissue a sandbox token before continuing.

SLA Monitor Agent: Polls Zendesk on a 15-minute schedule, fetches SLA tier data from HubSpot per client ID, calculates business-hours time remaining, and classifies each ticket as safe, at-risk, or critical.

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: ticket at 50% of SLA window

Open Zendesk ticket, creation timestamp 4 hrs ago, SLA window 8 business hrs, client ID maps to Tier 2 in HubSpot

Ticket classified as 'safe'; no alert triggered; internal classification payload generated

Output payload contains status='safe', time_remaining_pct=50, no downstream alert action queued

T02

Happy path: ticket crosses 75% threshold

Open ticket, 6.1 business hrs elapsed, 8-hr SLA window, Tier 2 client

Ticket classified as 'at-risk'; Alert Dispatch Agent receives payload with alert_type='warning'

Output payload contains status='at-risk', time_remaining_pct<=25, alert_type='warning' queued for Slack

T03

Happy path: ticket crosses 95% threshold

Open ticket, 7.7 business hrs elapsed, 8-hr SLA window, Tier 1 client

Ticket classified as 'critical'; Alert Dispatch Agent receives payload with alert_type='escalation'

Output payload contains status='critical', time_remaining_pct<=5, alert_type='escalation' queued for PagerDuty

T04

Multiple open tickets, mixed classification

10 synthetic open tickets across Tier 1, Tier 2, and Tier 3 with varying elapsed times

Each ticket independently classified; safe, at-risk, and critical tickets present in output batch

All 10 tickets appear in output; each carries correct status and SLA tier; no tickets dropped from payload

T05

Business-hours boundary: ticket created outside business hours

Ticket created at 11:45 PM, current time 9:15 AM next business day, 4-hr SLA window

Elapsed business-hours time calculated as 15 minutes; ticket classified as safe

time_elapsed_business_hrs=0.25; status='safe'; no alert triggered

T06

Failure: HubSpot client ID not found

Open Zendesk ticket with a client identifier that has no matching record in HubSpot sandbox

SLA tier fetch returns a null result; ticket flagged with status='tier_lookup_failed'; error logged

Ticket does not proceed to classification; error_code='HUBSPOT_ID_NOT_FOUND' present in error log; no silent pass-through

T07

Failure: Zendesk API returns 429 rate-limit error during poll

Sandbox Zendesk API configured to return HTTP 429 on the second page of results

Agent pauses, retries after 60-second backoff, completes poll on retry

Retry attempt logged; second poll attempt returns 200; all tickets eventually processed; no tickets permanently skipped

T08

Failure: business-hours calendar not configured for SLA tier

Ticket assigned to a Tier 3 client where no business-hours calendar has been mapped

Agent falls back to 24-hour calendar and flags the ticket with a configuration warning

status reflects 24-hr calculation; warning_code='BH_CALENDAR_MISSING' present; ticket not silently dropped

Alert Dispatch Agent: Receives the classified ticket list from the SLA Monitor Agent and routes alerts to Slack, PagerDuty, Zendesk, HubSpot, and Gmail based on ticket status and threshold.

Scenario

Input

Expected Output

Pass Criteria

T09

Happy path: Slack warning sent for at-risk ticket

Classified payload with status='at-risk', assigned_agent='agent_id_001', ticket_id='ZD-1042', client='Acme', minutes_remaining=28

Slack direct message sent to agent_id_001 with ticket number, client name, SLA tier, and minutes remaining

Slack API returns 200; message body contains ticket_id, client name, minutes_remaining, and SLA tier label

T10

Happy path: PagerDuty escalation fired for critical ticket

Classified payload with status='critical', ticket_id='ZD-1043', client='Beta Corp', time_remaining_pct=3

PagerDuty incident created; on-call manager receives page; incident includes ticket ID and client name

PagerDuty API returns incident ID; incident payload includes ticket_id='ZD-1043'; on-call schedule resolves correctly in dev account

T11

Happy path: Zendesk internal note posted

status='at-risk', alert sent, ticket_id='ZD-1042'

Internal note appended to ZD-1042 with timestamp, alert type, and recipient

Zendesk API returns 201; note body contains alert_type, recipient identifier, and ISO 8601 timestamp

T12

Happy path: HubSpot breach record logged

Ticket confirmed breached: ticket_id='ZD-1044', client_id='HS-992', sla_tier='Tier 1', minutes_overrun=14

Activity record created on HubSpot client HS-992 with all four breach fields populated

HubSpot API returns 201; activity record contains ticket_id, sla_tier, minutes_overrun, and breach_timestamp

T13

Happy path: Gmail weekly summary sent

Aggregated prior-week data: 3 breaches, 7 near-breaches, tier breakdown available

Formatted summary email sent via Gmail to support manager address; subject line includes week date range

Gmail API returns 200; email delivered to test manager address; body contains breach count, near-breach count, and tier breakdown table

T14

Failure: Slack API unavailable when warning queued

status='at-risk' payload received; Slack sandbox endpoint returns HTTP 503

Agent retries up to 3 times with exponential backoff; failure logged after third attempt; Zendesk internal note still posted

Retry count=3 logged; error_code='SLACK_UNAVAILABLE' in error log; Zendesk note posted regardless; no unhandled exception thrown

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Phase 2 verifies that the SLA Monitor Agent and Alert Dispatch Agent work correctly together across the full handoff, and that the combined system handles realistic failure conditions without data loss or silent errors. Integration tests continue the T sequence from T15. Edge-case tests use a separate E sequence starting at E01.

Scenario

Input

Expected Output

Pass Criteria

T15

Full handoff: safe ticket does not trigger any alert actions

SLA Monitor Agent output: batch of 5 tickets, all status='safe'

Alert Dispatch Agent receives batch; no Slack messages, PagerDuty incidents, or breach records created; Zendesk notes posted as poll confirmation only

Zero Slack API calls for alert messages; zero PagerDuty incidents; HubSpot has no new breach records; poll confirmation note present on each ticket

T16

Full handoff: mixed batch triggers correct split routing

Batch: 2 safe, 1 at-risk (Tier 2), 1 critical (Tier 1), 1 at-risk (Tier 3)

Slack warnings sent for both at-risk tickets; PagerDuty escalation fired for critical ticket only; all tickets receive internal Zendesk note

Exactly 2 Slack messages sent; exactly 1 PagerDuty incident created; 5 Zendesk internal notes posted; no cross-routing between tiers

T17

Full handoff: breach event creates HubSpot record and Zendesk note in same poll cycle

1 ticket confirmed breached during poll cycle

HubSpot breach activity record created; Zendesk internal note updated with breach confirmation; Gmail summary queue updated

HubSpot record exists with correct ticket_id and minutes_overrun; Zendesk note contains breach_confirmed=true; Gmail queue entry added for Monday summary

T18

Handoff timing: 15-minute poll completes before next poll cycle begins

Full batch of 200 synthetic open tickets fed through both agents

Both agents complete processing within 14 minutes; no overlap with next scheduled poll

End-to-end processing time logged under 14 minutes; next poll trigger does not fire while previous cycle is still running

T19

Repeat poll: ticket already alerted does not generate duplicate Slack message

Same at-risk ticket present in two consecutive poll cycles; Slack alert already sent in cycle 1

Cycle 2 does not resend Slack message; Zendesk note updated with re-check confirmation only

Slack API call count for that ticket_id = 1 across both cycles; idempotency check passes; Zendesk note reflects re-check timestamp

T20

PagerDuty escalation routes to correct on-call manager based on schedule

Critical ticket; PagerDuty dev account configured with two rotation schedules: weekday and weekend

Incident routed to the on-call person active at the time of test; off-rotation person does not receive page

PagerDuty incident assignment matches the active on-call entry in the dev schedule; no duplicate pages

Edge-case tests cover conditions that fall outside the happy path but are realistic in a live support environment: duplicate records, missing data, tool outages, and human non-response scenarios.

Scenario

Input

Expected Output

Pass Criteria

E01

Duplicate ticket IDs in Zendesk poll response

Poll returns the same ticket_id='ZD-2001' twice in the same batch due to a Zendesk pagination overlap

Agent deduplicates on ticket_id; processes the record once; logs a deduplication warning

Only one classification entry for ZD-2001 in output batch; warning_code='DUPLICATE_TICKET_ID' in log; no duplicate Slack message or breach record

E02

HubSpot SLA tier field is blank on client record

Ticket retrieved for client_id='HS-447'; HubSpot record exists but sla_tier field is empty

Ticket flagged as tier_lookup_incomplete; default safe classification applied; alert sent to FullSpec monitoring endpoint; no alert sent to agent or manager

status='safe_default_applied'; warning_code='SLA_TIER_FIELD_EMPTY' logged; no Slack or PagerDuty action triggered; monitoring notification fired

E03

Zendesk API times out mid-poll (gateway timeout)

Poll starts for 200 tickets; API returns HTTP 504 after returning 120 records

Agent logs the partial poll; retries the incomplete page range; completes the full batch on retry without reprocessing the first 120 tickets

Final batch count = 200 unique tickets; no ticket processed twice; timeout logged with partial_count=120; retry succeeds within one additional attempt

E04

PagerDuty unavailable when critical ticket escalation fires

status='critical' payload; PagerDuty API returns HTTP 503 on escalation attempt

Agent retries PagerDuty 3 times; falls back to sending a direct Slack message to the support manager channel; error logged

PagerDuty retry count=3 in log; fallback Slack message delivered to manager channel; error_code='PAGERDUTY_UNAVAILABLE'; Zendesk note reflects fallback escalation method used

E05

Ticket breaches SLA before any 75% warning was dispatched (rapid breach)

Ticket created close to end of business hours; next poll cycle occurs after SLA window has already expired

Agent detects breach directly; skips at-risk alert; fires critical escalation and breach logging immediately

No at-risk Slack message sent; PagerDuty incident created; HubSpot breach record created; Zendesk note contains breach_detected_without_prior_warning=true

E06

Manager does not acknowledge PagerDuty incident within 15 minutes

PagerDuty incident created; no acknowledgement from on-call manager after 15 minutes

PagerDuty escalation policy re-pages the secondary on-call contact per the configured escalation policy; no additional automation action required

PagerDuty escalation policy fires to secondary contact at 15-minute mark as configured; FullSpec verifies policy is active in dev account before Phase 3 is approved

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: All test tickets must be created using an internal test email address and a designated test client record that is clearly labelled in both Zendesk and HubSpot (for example: client_name='FULLSPEC_TEST_DO_NOT_BILL'). Every test ticket must be voided, closed, and its HubSpot activity record deleted immediately after each test case completes. Do not use a real client record. Do not leave a test PagerDuty incident open after testing. FullSpec confirms all cleanup steps before signing off on each case.

Phase 3 runs on the live production environment with real credentials, real tool connections, and real alert routing. The FullSpec team leads each case. The process owner (Support Lead and Support Manager) must be present to confirm receipt of alerts, review error logs, and record final approval. Test cases continue the T sequence from T21.

Scenario

Steps Traced

Expected Production Output

Pass Criteria

T21

Full production run: at-risk ticket triggers Slack warning end to end

Scheduled poll fires on production Zendesk; SLA Monitor Agent fetches Tier 2 SLA from production HubSpot; calculates 76% elapsed; Alert Dispatch Agent sends Slack warning; Zendesk internal note posted

Assigned agent receives Slack DM within 2 minutes of poll completion; Zendesk note posted with alert timestamp; no HubSpot breach record created

Slack message received by assigned agent within 2 minutes; message contains correct ticket_id, client name, SLA tier, and minutes remaining; Zendesk note confirmed in ticket audit trail; HubSpot breach record count unchanged

T22

Full production run: critical ticket triggers PagerDuty escalation end to end

Poll detects ticket at 96% elapsed; Alert Dispatch Agent fires PagerDuty incident; Zendesk note posted; support manager receives page

PagerDuty incident created and routed to production on-call manager within 3 minutes of poll; Zendesk note confirms escalation dispatched

Manager confirms receipt of PagerDuty page; incident payload contains correct ticket_id and client name; Zendesk note present; incident voided immediately after confirmation

T23

Full production run: breach event logged end to end in HubSpot and Zendesk

Poll detects breached ticket; Alert Dispatch Agent creates HubSpot breach activity record on test client; Zendesk note updated with breach confirmation

HubSpot activity record created on test client with ticket_id, sla_tier, minutes_overrun, and breach_timestamp; Zendesk note contains breach_confirmed=true

HubSpot record verified by process owner in production CRM; all four breach fields populated correctly; Zendesk note confirmed; test records deleted post-verification

T24

Error log verification: confirm all production errors surface correctly

FullSpec team reviews automation platform error log after T21, T22, and T23 runs; checks for any unhandled exceptions, silent failures, or missing retry entries

Error log shows only expected informational entries; no unhandled exceptions; retry logic entries present where applicable; no silent data drops recorded

Process owner reviews error log alongside FullSpec; zero unhandled exceptions confirmed; log format readable and timestamped in ISO 8601; monitoring alert channel receives no unexpected error notifications during the three test runs

T25

Process owner review and approval

Process owner (Support Lead and Support Manager) reviews all Phase 3 results: Slack alerts received, PagerDuty routing confirmed, HubSpot records verified, Zendesk notes confirmed, error log reviewed, Gmail summary format previewed

Process owner confirms the automation behaves as specified across all production cases and records approval in the FullSpec confirmation record

Process owner confirms approval in writing via the FullSpec confirmation record. FullSpec logs the approval with the owner's name, role, and timestamp. No separate signature block is required. Approval in the FullSpec system constitutes formal sign-off and unlocks the go-live switch for continuous production polling.

Once T25 is recorded as approved in the FullSpec confirmation system, the automation is cleared for continuous production operation. The scheduled 15-minute poll will activate on the production Zendesk instance and remain live. Any issues after go-live should be reported to support@gofullspec.com with the process template ID: customer-support-sla-monitoring-and-breach-alerting.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process