Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Software Deployment Workflow

[YourCompany.com] · IT Department · Prepared by FullSpec · [Today's Date]

This document defines the full test and quality assurance programme for the Software Deployment Workflow automation. It is written for the FullSpec engineering team and covers every scripted test case across three sequential phases: unit testing of individual agents, integration and edge-case testing of agent handoffs, and end-to-end validation in production. No phase begins until the previous phase has fully passed. The process owner joins the FullSpec team for Phase 3 sign-off only.

01Testing approach

Testing is structured across three phases that run in strict sequence. Phase 1 validates each agent in isolation using sandbox credentials and mock payloads. Phase 2 verifies that agents hand off correctly to one another and that the workflow handles edge cases gracefully. Phase 3 executes a full end-to-end run against the production environment with real credentials and a real (but voided) deployment record. No phase begins until every test case in the preceding phase returns a pass result and the FullSpec team has confirmed the outcome in writing.

Phase 1: Unit Testing

Each agent tested in isolation before integration begins

Scope

Individual agent logic: trigger detection, API calls, output payloads. No cross-agent handoffs tested here.

Environment

Sandbox only. GitHub test repository, Jira sandbox project, Slack test workspace, Confluence sandbox space, Datadog sandbox monitors, PagerDuty developer account.

Credential rule

Sandbox credentials only. No production tokens, API keys, or secrets may be loaded into the test runner during Phase 1.

Pass condition

All cases T01 through T15 return expected outputs with no unhandled errors. Zero failures permitted before proceeding to Phase 2.

Who runs it

FullSpec team only. Process owner is not required to attend.

Phase 2: Integration and Edge-Case Testing

Agent handoffs and failure-mode coverage validated end to end in sandbox

Scope

Cross-agent handoffs, data propagation between agents, and all identified edge cases including duplicates, missing data, unavailable tools, and approval timeout.

Environment

Sandbox only. Same credential set as Phase 1. No production systems accessed.

Credential rule

Sandbox credentials only. Phase 2 does not require production access. Any credential rotation must be completed before Phase 3 begins.

Pass condition

All integration cases T16 through T22 and all edge-case cases E01 through E06 return expected outputs. Any amber result requires a documented resolution before Phase 3 is unlocked.

Who runs it

FullSpec team only. Process owner is not required to attend.

Phase 3: End-to-End Production Test

Full live run with real credentials; process owner review and sign-off required

Scope

Complete deployment workflow from GitHub PR merge trigger through to Jira ticket closure and Slack team notification, using production credentials and a designated test deployment record.

Environment

Production. Real GitHub repository (test branch), real Jira project, real Slack workspace, real Datadog monitors, real PagerDuty service. All test records voided after the run.

Credential rule

Production credentials loaded via the secrets manager. FullSpec rotates sandbox credentials out and loads production secrets immediately before Phase 3 commences.

Pass condition

All cases T23 through T27 return expected outputs, including the process owner review case (T27). FullSpec records the owner approval outcome in the project confirmation log.

Who runs it

FullSpec team executes T23 through T26. Process owner joins for T27 (review and approval). Owner confirmation is recorded by FullSpec; no separate signature block is required.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only. Before running any case in this phase, confirm that no production API tokens, GitHub secrets, Jira project keys, Datadog API keys, or PagerDuty integration keys are present in the test runner environment. Use the designated sandbox accounts for every service. Any test that inadvertently touches a production resource must be halted immediately and reported to support@gofullspec.com.

Deployment Gate Agent — tests T01 through T05

Scenario

Input

Expected output

Pass criteria

T01

Happy path: PR merged, all CI checks passed

GitHub webhook: PR merged to main; all GitHub Actions check runs status = success

Jira ticket created with correct version, environment, and PR link; Slack Block Kit approval message posted to IT lead with Approve and Reject buttons

Jira ticket ID returned in response; Slack message ts field populated; no errors in agent log

T02

CI checks partially failed

GitHub webhook: PR merged to main; one GitHub Actions check run status = failure

Slack alert sent to developer channel stating failing check name; Jira ticket not created; deployment pipeline not triggered

Slack alert contains check run name and failure reason; agent exits at CI gate branch; zero downstream actions fired

T03

CI check status pending (timeout scenario)

GitHub webhook: PR merged; check run status = in_progress after 10-minute poll window

Agent logs timeout event; Slack alert sent to developer with pending check detail; no further actions taken

Timeout logged at correct interval; Slack message delivered; agent does not proceed to Jira creation

T04

GitHub API returns 401 Unauthorized

GitHub webhook fired; GitHub API responds with HTTP 401 on check run query

Agent logs authentication error; Slack alert sent to FullSpec error channel; agent halts gracefully

Error log entry contains status code 401 and endpoint URL; no partial Jira ticket created; Slack error message delivered

T05

Jira ticket creation fails (403 Forbidden)

CI checks passed; Jira API responds with HTTP 403 on ticket creation call

Agent logs Jira permission error; Slack alert sent to IT ops channel; no Slack approval message sent to IT lead

Error log contains Jira project key and HTTP 403 response; approval message not posted; agent halts at Jira step

Environment and Deploy Agent — tests T06 through T10

Scenario

Input

Expected output

Pass criteria

T06

Happy path: IT lead clicks Approve in Slack

Slack interactive payload: action_id = approve_deployment; associated Jira ticket ID present

Confluence config page read successfully; environment variables applied to target environment; GitHub Actions deployment workflow triggered with correct inputs

GitHub Actions run ID returned and logged; Confluence page version confirmed in log; Slack acknowledgement message sent to IT lead

T07

IT lead clicks Reject in Slack

Slack interactive payload: action_id = reject_deployment

Jira ticket status updated to Rejected; Slack message sent to developer channel with IT lead rejection noted; deployment pipeline not triggered

Jira ticket transitions to Rejected status; GitHub Actions not invoked; developer channel message contains ticket ID and rejection timestamp

T08

Confluence config page not found (404)

Slack approval received; Confluence API returns HTTP 404 for configured page ID

Agent logs config fetch error; Slack alert sent to IT ops channel; GitHub Actions not triggered

Error log contains Confluence page ID and HTTP 404; no deployment triggered; IT ops Slack alert delivered

T09

GitHub Actions API returns 422 Unprocessable Entity on workflow dispatch

Approval received; Confluence config read successfully; GitHub Actions dispatch endpoint returns HTTP 422

Agent logs dispatch error with response body; Slack alert sent to developer; Jira ticket flagged with deployment error comment

Error log contains workflow ID and HTTP 422 response body; Jira comment added within 30 seconds; Slack alert delivered

T10

Environment variable missing from Confluence config table

Confluence page found; required key DATABASE_URL absent from config table

Agent logs missing key error; Slack alert sent to IT ops channel naming the missing variable; deployment not triggered

Error log names missing key; Slack alert contains variable name and Confluence page URL; no GitHub Actions run created

Post-Deploy Monitor Agent — tests T11 through T15

Scenario

Input

Expected output

Pass criteria

T11

Happy path: deployment completes, all health checks within threshold

GitHub Actions run status = completed (success); Datadog monitors all in OK state after 5-minute poll

Slack deployment summary posted to team channel with success status, environment, and timestamp; Jira ticket transitioned to Done with deployment timestamp comment

Slack message ts populated; Jira ticket in Done status; no PagerDuty incident created; all fields (environment, version, duration) present in Slack message

T12

Deployment completes but Datadog error rate breaches threshold

GitHub Actions run status = completed (success); Datadog monitor error_rate in ALERT state

PagerDuty incident created with deployment context; Slack alert posted to team channel flagging health check failure; Jira ticket updated with rollback review flag

PagerDuty incident ID logged; Jira ticket contains rollback flag comment; Slack alert includes Datadog monitor name and breach value

T13

GitHub Actions deployment run fails (conclusion = failure)

GitHub Actions run status = completed (failure)

Slack failure alert posted to team channel; Jira ticket transitioned to Deployment Failed status with error log link; PagerDuty incident not created (pipeline failure, not health breach)

Jira ticket in Deployment Failed status; Slack alert contains GitHub Actions run URL; no PagerDuty incident raised

T14

Datadog API unavailable during health poll

Deployment completes; Datadog API returns HTTP 503 on monitor status query

Agent retries Datadog query three times at 60-second intervals; on continued failure logs degraded monitoring event; Slack alert sent to IT ops noting monitoring unavailability

Retry attempts (3) logged with timestamps; Slack alert delivered after third failure; Jira ticket comment notes monitoring gap

T15

Jira ticket update fails after successful deployment

Deployment and health checks successful; Jira API returns HTTP 500 on status transition call

Agent logs Jira update failure; Slack alert sent to developer channel requesting manual Jira update; Slack team summary still posted

Slack team summary posted successfully; Jira error logged with ticket ID and HTTP 500; developer channel alert requests manual resolution

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Integration tests verify that data flows correctly between agents across handoff boundaries. The Jira ticket ID created by the Deployment Gate Agent must be visible and referenced correctly by both downstream agents. The Slack interactive payload must carry sufficient context for the Environment and Deploy Agent to proceed without additional API lookups. The GitHub Actions run ID must be logged and accessible to the Post-Deploy Monitor Agent for health check correlation.

Scenario

Input

Expected output

Pass criteria

T16

Deployment Gate Agent to Environment and Deploy Agent: Jira ticket ID propagated correctly

Full run through Deployment Gate Agent producing a Jira ticket ID; IT lead approves in Slack

Environment and Deploy Agent reads the Jira ticket ID from the Slack payload context and includes it in the GitHub Actions workflow dispatch inputs

GitHub Actions run inputs contain matching Jira ticket ID; agent log shows ID read from Slack payload without additional Jira API call

T17

Environment and Deploy Agent to Post-Deploy Monitor Agent: GitHub Actions run ID handed off

Environment and Deploy Agent triggers GitHub Actions successfully; run ID returned in dispatch response

Post-Deploy Monitor Agent uses the GitHub Actions run ID to poll run status and correlate the completed run with the correct Jira ticket

Datadog poll initiated after correct run ID is confirmed complete; Jira ticket updated using the same ticket ID from the original handoff chain

T18

Full three-agent happy-path chain in sandbox

Sandbox PR merged; CI passes; IT lead approves; Confluence config complete; deployment succeeds; Datadog monitors OK

Jira ticket moves from Open to In Progress to Done; two Slack messages sent (approval request and team summary); GitHub Actions run completed; no PagerDuty incident

All state transitions confirmed in Jira audit log; both Slack message timestamps logged; GitHub run ID consistent across all three agent logs; elapsed time under 10 minutes for sandbox run

T19

Approval rejected mid-chain: downstream agents do not fire

Deployment Gate Agent completes; IT lead clicks Reject in Slack

Environment and Deploy Agent receives rejection signal and halts; Post-Deploy Monitor Agent is never invoked; Jira ticket moves to Rejected

No GitHub Actions run created; no Datadog poll initiated; no PagerDuty incident; Jira ticket in Rejected status; developer channel alerted

T20

CI failure mid-chain: Jira and Slack approval not triggered

GitHub webhook fires; CI check returns failure status

Deployment Gate Agent halts at CI gate; no Jira ticket created; no Slack approval message sent; developer alerted

Jira ticket count unchanged from pre-test value; no Slack Block Kit message delivered to IT lead; developer alert contains failing check name

T21

Post-deploy health breach: PagerDuty and Jira both updated

Deployment completes in GitHub Actions; Datadog monitor returns ALERT state on first poll

PagerDuty incident created with deployment metadata; Jira ticket updated with rollback review comment; Slack team channel receives failure summary

PagerDuty incident dedup_key matches deployment run ID; Jira comment timestamp within 60 seconds of GitHub run completion; Slack message contains PagerDuty incident URL

T22

Slack interactive message payload malformed (missing action value)

Slack sends interactive callback with empty actions array

Environment and Deploy Agent logs payload parse error; Slack error message sent to IT lead asking them to reopen the approval request; no deployment triggered

Error log contains raw payload excerpt and parse failure reason; IT lead receives Slack follow-up within 30 seconds; no GitHub Actions run created

Edge-case tests cover failure scenarios that are unlikely in normal operation but must be handled without data loss, duplicate actions, or silent failures.

Scenario

Input

Expected output

Pass criteria

E01

Duplicate PR merge webhook received (GitHub sends event twice)

Two identical GitHub webhook payloads for the same PR merge within 5 seconds

Idempotency check prevents a second Jira ticket being created; second approval message not sent to IT lead; single pipeline run initiated

Jira ticket count for the PR is exactly 1; IT lead receives exactly 1 Slack approval message; agent log records duplicate suppression event

E02

Jira ticket ID missing from Slack payload at Environment and Deploy Agent

Slack interactive payload lacks the jira_ticket_id context field

Agent logs missing field error; Slack alert sent to IT ops; deployment halted until context is manually re-supplied

No deployment triggered; error log names the missing field; IT ops Slack alert delivered with instructions to re-trigger

E03

Confluence page last-modified timestamp older than 30 days (stale config warning)

Confluence API returns page with last_modified date more than 30 days prior

Agent posts a warning to the IT ops Slack channel noting stale config; deployment proceeds if all required variables are present

Slack warning contains Confluence page URL and last-modified date; deployment is not blocked; warning logged in agent run record

E04

PagerDuty API unavailable when health check breach is detected

Datadog monitor in ALERT state; PagerDuty Events API returns HTTP 503

Agent retries PagerDuty call twice at 30-second intervals; on continued failure logs incident creation failure and sends escalation Slack alert to IT lead directly

Retry attempts (2) logged; IT lead receives direct Slack escalation message naming the Datadog monitor and breach value; Jira ticket updated with PagerDuty failure note

E05

IT lead approval times out (no action within 60 minutes)

Slack approval message sent; no interactive callback received after 60 minutes

Deployment Gate Agent posts a timeout alert to the IT ops channel and the developer channel; Jira ticket updated with Awaiting Approval - Timed Out status; pipeline not triggered

Timeout fires at 60-minute mark within a 2-minute tolerance; both Slack channels alerted; Jira ticket status updated; no deployment initiated

E06

GitHub Actions run enters a queued state indefinitely (no completion event after 30 minutes)

GitHub Actions workflow dispatched; run status remains queued for 30 minutes without transitioning

Post-Deploy Monitor Agent logs stuck run event; Slack alert sent to developer and IT ops channels with run URL; Jira ticket flagged with Deployment Stalled comment

Stuck-run alert fires at 30-minute mark within a 2-minute tolerance; Slack alert contains GitHub Actions run URL; Jira comment references run ID

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3. Use a designated internal test branch and a pre-agreed test Jira project key for all production runs in this phase. The IT lead must use a named test Slack channel (not the live deployment channel) for the approval click during T23 and T24. Every Jira ticket, PagerDuty incident, and Slack message created during Phase 3 must be voided or deleted by the FullSpec team immediately after the test run completes. Do not merge the test branch to main. Do not apply test environment variables to any live service. Contact support@gofullspec.com before starting this phase if any production system access is unclear.

Scenario

Input

Expected output

Pass criteria

T23

Full production happy-path run: PR merged, approved, deployed, health checks pass

Test PR merged to designated test branch in production GitHub repo; CI checks pass; IT lead clicks Approve in test Slack channel; Confluence production config page read; GitHub Actions test workflow dispatched; Datadog production monitors in OK state

Jira test ticket created and progressed to Done; two Slack messages delivered (approval request and team summary); GitHub Actions run completed successfully; no PagerDuty incident created; all timestamps logged in agent run record

End-to-end elapsed time under 15 minutes; Jira ticket audit trail shows all status transitions; Slack messages contain correct environment name, version, and PR link; GitHub Actions run ID consistent across all three agent logs

T24

Production run with simulated health check breach

Test PR merged; CI passes; IT lead approves; GitHub Actions test workflow dispatched and completed; Datadog test monitor manually set to ALERT state immediately after run

PagerDuty incident created in production PagerDuty account against the pre-agreed test service; Slack team channel receives failure summary; Jira test ticket updated with rollback review comment

PagerDuty incident ID logged; Jira comment timestamp within 60 seconds of GitHub run completion; Slack failure message delivered to correct channel; incident voided after test completes

T25

Error log verification: confirm all agent errors are written to the centralised log

Simulate a Jira API 403 error during the production run by temporarily revoking the Jira token mid-test, then restore it

Agent halts at Jira step; error log entry written with HTTP 403, endpoint URL, Jira project key, and timestamp; Slack IT ops alert delivered

Error log entry present in the centralised logging destination within 30 seconds of the error occurring; Slack alert delivered; log entry contains all required fields (status code, endpoint, timestamp, agent name)

T26

Verify all test records are voided after production run

Phase 3 runs T23 through T25 completed; FullSpec team executes post-test cleanup checklist

Test Jira ticket deleted or moved to void project; PagerDuty test incident resolved and deleted; test Slack messages deleted from channels; test GitHub Actions run archived; Confluence config page reverted if edited

Zero test artefacts remaining in production systems after cleanup; FullSpec team confirms cleanup in project log with timestamps for each deleted or voided record

T27

Process owner review and approval of Phase 3 results

FullSpec team presents Phase 3 test results summary to process owner (IT Lead), including agent logs, Slack message screenshots, Jira audit trail, and PagerDuty incident record for T24

Process owner reviews all Phase 3 outputs, confirms the automation behaves as specified, and provides verbal or written approval to proceed to go-live. FullSpec records the approval in the project confirmation log with the owner name, date, and outcome.

Owner approval recorded in FullSpec confirmation log. No separate signature block required. Go-live is blocked until this case is marked Pass by FullSpec.

Once T27 is marked Pass in the FullSpec project log, the automation is cleared for go-live activation on the production repository. Any case marked Fail at any phase resets that phase; all cases in the failed phase must be re-run in full after the fix is applied. Contact support@gofullspec.com to report a failure or to request a phase reset.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process