Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Cybersecurity Incident Response Automation

[YourCompany.com] · IT Department · Prepared by FullSpec · [Today's Date]

This document defines the full quality assurance programme for the Cybersecurity Incident Response automation, covering three sequential test phases: unit testing of individual agents, integration and edge-case testing of agent handoffs and failure modes, and end-to-end production validation. The FullSpec team runs all phases; the process owner (IT Manager) joins Phase 3 to confirm acceptance. No phase begins until the previous phase has fully passed. Test case IDs are sequential across all phases; edge-case IDs use the E prefix.

01Testing approach

Testing runs in three phases in sequence. No phase begins until every test case in the previous phase carries a confirmed pass status. Phase 1 validates each agent in isolation against sandbox credentials. Phase 2 validates cross-agent handoffs and edge-case resilience. Phase 3 validates the full workflow in the production environment with real credentials, witnessed by the process owner.

Phase 1: Unit Testing

Each agent tested in isolation against sandbox data

Scope

Individual agent behaviour: Triage Agent, Notification Agent, Incident Documentation Agent. Each agent is tested independently with mocked or sandbox inputs.

Environment

Sandbox only. No production tenants, live Jira projects, real PagerDuty policies, or active Slack channels.

Credential rule

Sandbox API keys and test tokens only. Production credentials must not be present in the environment during Phase 1.

Pass condition

All cases T01 through T15 return the expected output with zero unexpected errors. Any failure blocks progression to Phase 2.

Who runs it

FullSpec team only.

Phase 2: Integration and Edge-Case Testing

Agent handoffs and failure modes validated

Scope

End-to-end agent handoff chain (Triage to Notification to Documentation) and a dedicated edge-case battery covering duplicates, missing data, unavailable tools, and human timeout scenarios.

Environment

Sandbox for all edge-case tests. Integration handoff tests use sandbox tenants with pre-seeded test data mirroring production schema.

Credential rule

Sandbox credentials only. A separate sandbox Jira project, Notion workspace, Slack channel, and PagerDuty service must be provisioned and named with a 'TEST-' prefix.

Pass condition

All cases T16 through T24 and E01 through E08 resolve with the expected output or the correct error-handling path. No silent failures permitted.

Who runs it

FullSpec team only.

Phase 3: End-to-End Production Test

Full workflow validated in the live environment with process owner sign-off

Scope

One complete incident lifecycle in the production environment: alert ingestion, triage, Jira ticket creation, Slack notification, PagerDuty escalation (where applicable), IT Manager review step, Notion report generation, and post-incident Slack summary.

Environment

Production. Live Microsoft Defender tenant, live Jira project, live Slack workspace, live PagerDuty account, live Notion workspace. Test alert injected via the Defender API; all artefacts voided after confirmation.

Credential rule

Production credentials active. A dedicated test alert type must be agreed with the IT Manager before injection so live on-call engineers are not disturbed unnecessarily.

Pass condition

All cases T25 through T28 pass. The final case (T28) is process owner review and approval, recorded in the FullSpec confirmation log.

Who runs it

FullSpec team executes; process owner (IT Manager) joins for T28 approval.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

All Phase 1 tests must be executed against sandbox credentials only. Do not connect production Microsoft Defender, Microsoft 365, Jira, Slack, PagerDuty, or Notion accounts at this stage. Any test that inadvertently touches a production system is immediately invalidated and must be re-run in the correct sandbox environment before the phase can be marked as passed.

Triage Agent — Microsoft Defender, Microsoft 365, Jira

Scenario

Input

Expected Output

Pass Criteria

T01

Happy path: Critical alert received

Defender webhook payload; alert type: malware execution; affected asset: domain admin account; risk score: 95

Jira ticket created with severity tag CRITICAL, alert summary, and attached log excerpts from Microsoft 365 audit log

Ticket exists in sandbox Jira project within 90 seconds; severity field = CRITICAL; log attachment present and non-empty

T02

Happy path: Medium alert received

Defender webhook payload; alert type: suspicious sign-in; affected asset: standard user account; risk score: 45

Jira ticket created with severity tag MEDIUM, alert summary, log excerpts attached

Ticket exists within 90 seconds; severity = MEDIUM; no PagerDuty escalation triggered

T03

Happy path: Low alert received

Defender webhook payload; alert type: informational policy match; risk score: 10

Jira ticket created with severity tag LOW; minimal log excerpt attached

Ticket exists within 90 seconds; severity = LOW; no Slack notification or PagerDuty call triggered by this agent

T04

Failure: Defender API returns 401 Unauthorised

Invalid or expired sandbox OAuth token in Defender connection

Automation platform logs an authentication error; no Jira ticket created; error surfaced to monitoring log

No orphaned ticket created; error entry visible in platform execution log with status 401 and timestamp

T05

Failure: Microsoft 365 log retrieval returns empty result

Valid Defender alert; Microsoft 365 returns 0 audit events for the affected user in the query window

Jira ticket created with severity classification intact; log attachment field contains a structured note: 'No audit logs returned for the query window'

Ticket exists; severity populated; attachment field is not blank; agent does not halt or throw unhandled exception

T06

Failure: Jira API returns 429 Rate Limit Exceeded

Valid Defender alert and Microsoft 365 log data; Jira sandbox rate limit simulated

Automation platform retries with exponential back-off (3 attempts, max 60 seconds); if still failing, logs a critical error and sends a fallback Slack DM to the IT Manager

Retry attempts logged; fallback Slack DM delivered to sandbox test user within 120 seconds of initial failure

T07

Severity boundary: High alert classification

Defender payload; alert type: lateral movement; affected asset: privileged service account; risk score: 78

Jira ticket created with severity tag HIGH; log excerpts attached

Severity = HIGH; ticket present; downstream Notification Agent trigger condition (Critical or High) will be satisfied in Phase 2

Notification Agent — Slack, PagerDuty

Scenario

Input

Expected Output

Pass Criteria

T08

Happy path: Critical ticket triggers PagerDuty escalation and Slack message

Sandbox Jira ticket with severity = CRITICAL created by mock Triage Agent output

Formatted Slack message posted to sandbox #it-incidents channel with severity, summary, and Jira link; PagerDuty incident opened on sandbox on-call policy

Slack message present within 60 seconds; PagerDuty incident status = triggered; message body contains severity label, one-line summary, and valid Jira URL

T09

Happy path: High ticket triggers PagerDuty escalation and Slack message

Sandbox Jira ticket with severity = HIGH

Slack message posted to #it-incidents; PagerDuty incident opened

Same pass criteria as T08; PagerDuty incident policy = high-severity sandbox policy

T10

Happy path: Medium ticket triggers Slack message only, no PagerDuty

Sandbox Jira ticket with severity = MEDIUM

Slack message posted to #it-incidents; no PagerDuty incident created

Slack message present within 60 seconds; PagerDuty sandbox shows zero new incidents

T11

Happy path: Low ticket triggers Slack message only

Sandbox Jira ticket with severity = LOW

Slack message posted; no PagerDuty call

Slack message present; PagerDuty sandbox unchanged

T12

Failure: Slack API returns 503 Service Unavailable

Valid Jira ticket; Slack sandbox endpoint returning 503

Platform retries up to 3 times; logs error; sends fallback notification via PagerDuty Events API if severity is Critical or High

Retry log present; fallback PagerDuty alert sent for Critical/High; error recorded with timestamp and HTTP status

T13

Failure: PagerDuty API returns 400 Bad Request (malformed routing key)

Valid Critical Jira ticket; PagerDuty sandbox routing key deliberately malformed

Error logged; Slack message still delivered; error entry created in platform log referencing the bad routing key

Slack message delivered despite PagerDuty failure; error log entry visible; no silent failure

Incident Documentation Agent — Notion, Jira, Slack

Scenario

Input

Expected Output

Pass Criteria

T14

Happy path: Jira ticket resolved triggers Notion report and Slack summary

Sandbox Jira ticket updated to status RESOLVED with resolution notes and action log

Structured Notion page created with timeline, log excerpts, actions taken, and resolution status; post-incident Slack summary posted to #it-incidents with link to Notion page

Notion page exists within 120 seconds of ticket resolution; all mandatory fields populated (timeline, affected asset, severity, actions, resolution); Slack message contains Notion URL

T15

Failure: Notion API returns 401 Unauthorised

Valid resolved Jira ticket; Notion sandbox integration token expired

Error logged; Slack summary still posted with a note that the Notion report could not be created; platform error log updated

Slack summary delivered; error message in summary body states Notion report creation failed; error log entry present with 401 status

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Phase 2 validates the full handoff chain between agents (T16 through T24) and then runs a dedicated edge-case battery (E01 through E08) covering real-world failure conditions that span multiple agents or involve unexpected data states.

Integration handoff tests

Scenario

Input

Expected Output

Pass Criteria

T16

Full chain: Critical alert — Triage to Notification to Documentation

Defender sandbox webhook; risk score 92; domain admin account affected

Triage Agent creates CRITICAL Jira ticket with logs; Notification Agent posts Slack message and opens PagerDuty incident; after ticket resolution, Documentation Agent creates Notion report and posts Slack summary

All three agents fire in sequence; Jira ticket, PagerDuty incident, Notion page, and two Slack messages all present; total elapsed time under 3 minutes from webhook receipt to Slack alert

T17

Full chain: Medium alert — Triage to Notification to Documentation

Defender sandbox webhook; risk score 48; standard user account affected

MEDIUM Jira ticket created; Slack alert posted; no PagerDuty; Notion report and Slack summary on resolution

No PagerDuty incident; all other artefacts present; elapsed time under 3 minutes

T18

Handoff delay: Triage Agent completes; Notification Agent delayed by 45 seconds

Valid CRITICAL Jira ticket; artificial 45-second delay injected before Notification Agent trigger poll

Notification Agent fires on next poll cycle; Slack message and PagerDuty incident created despite delay

Slack message delivered within 120 seconds of Jira ticket creation; no duplicate notifications; delay logged

T19

IT Manager review step: manual approval recorded in Jira

CRITICAL ticket in Jira; IT Manager sandbox user adds a comment 'Approved: isolate device' and transitions ticket to IN PROGRESS

Workflow continues; Documentation Agent monitors ticket and will fire on RESOLVED transition

Jira audit log contains IT Manager comment; ticket status = IN PROGRESS; no premature Documentation Agent trigger

T20

IT Manager review step: override with manual action recorded

HIGH ticket; IT Manager sandbox user adds override comment 'False positive: no action required' and closes ticket

Documentation Agent fires on RESOLVED; Notion report reflects override note and false-positive determination

Notion report contains the override comment text; resolution status = false positive; Slack summary reflects same

T21

Notification Agent: correct Slack channel routing by severity

CRITICAL ticket routes to #it-critical-incidents; MEDIUM ticket routes to #it-incidents

Each severity level delivers to the correct sandbox channel

Message present in the correct channel only; no cross-posting

T22

Documentation Agent: Jira ticket with multiple updates produces single Notion report

Jira ticket updated 4 times before resolution (status changes and comment additions)

Single Notion page created on resolution; page body reflects the most recent ticket state; no duplicate pages

Exactly one Notion page per Jira ticket; content reflects final ticket state

T23

Triage Agent: alert with no affected user field populated

Defender payload where the affected user field is null; asset field populated

Triage Agent creates Jira ticket with asset information; affected user field in Jira set to 'Unknown'; Microsoft 365 log query scoped to asset instead

Ticket created; affected user = 'Unknown'; log query uses asset identifier; no unhandled null reference exception

T24

Concurrent alerts: two Defender alerts arrive within 5 seconds of each other

Two distinct Defender sandbox webhook payloads, both with HIGH severity, arriving 5 seconds apart

Two separate Jira tickets created; two Slack notifications sent; two PagerDuty incidents opened; no data from one alert bleeding into the other ticket

Two distinct Jira tickets with unique IDs; ticket bodies contain only their respective alert data; two Slack messages posted; two PagerDuty incidents visible

Edge-case tests

Scenario

Input

Expected Output

Pass Criteria

E01

Duplicate alert: same Defender alert ID received twice within 60 seconds

Identical Defender webhook payload sent twice; same alert ID

Automation platform detects duplicate alert ID; second payload is suppressed; no second Jira ticket created; deduplication log entry recorded

Exactly one Jira ticket per unique alert ID; deduplication log entry present with suppressed payload timestamp

E02

Missing data: Defender payload arrives with empty alert body

Defender webhook fires; alert body is an empty JSON object

Triage Agent logs a structured parse error; no Jira ticket created; fallback Slack DM sent to IT Manager alerting that an alert arrived with no parseable data

No Jira ticket created; parse error in platform log; Slack DM delivered to sandbox IT Manager user within 90 seconds

E03

Unavailable tool: Jira is unreachable for 3 minutes (simulated outage)

Valid Critical Defender alert; Jira sandbox endpoint returns 503 for 3 minutes

Platform retries with back-off; if Jira remains unavailable after 3 minutes, alert details are written to a fallback queue and a Slack DM is sent to the IT Manager with full alert detail

Fallback queue entry present; Slack DM contains full alert summary; Jira ticket created automatically when Jira recovers (within 5 minutes of service restoration)

E04

Unavailable tool: PagerDuty is unreachable during Critical alert escalation

CRITICAL Jira ticket created; PagerDuty sandbox returns 500 for the escalation call

Notification Agent logs PagerDuty failure; Slack message still delivered; a secondary Slack DM is sent directly to the on-call contact defined in the runbook as a fallback escalation path

Slack message to #it-critical-incidents delivered; secondary DM to on-call contact present; PagerDuty failure logged with HTTP 500 and timestamp

E05

Unavailable tool: Notion is unreachable when Documentation Agent fires

Resolved Jira ticket; Notion sandbox returns 503

Documentation Agent logs error; Slack post-incident summary still delivered with note that the Notion report is pending; retry queued for 10 minutes

Slack summary delivered with pending-report note; retry queue entry present; Notion page created on successful retry

E06

Human timeout: IT Manager does not action the Jira review within 30 minutes

CRITICAL Jira ticket in OPEN state; no IT Manager comment or status change for 30 minutes

Automation platform sends a reminder Slack DM to the IT Manager; after a further 15 minutes with no action, a second escalation DM is sent to the Operations Lead (Priya Nair)

First reminder DM at T+30 min; second escalation DM at T+45 min; both visible in sandbox Slack; no automated containment action taken without human approval

E07

Severity misclassification: alert scores at the MEDIUM/HIGH boundary (risk score 65 to 70)

Defender payload; risk score = 67; affected asset: shared mailbox

Triage Agent classifies as MEDIUM based on asset type weighting in the rubric; Jira ticket tagged MEDIUM; no PagerDuty escalation

Severity = MEDIUM in Jira; no PagerDuty incident; classification log entry shows rubric scoring breakdown for the boundary decision

E08

Missing data: Microsoft 365 log query times out (30-second timeout threshold)

Valid Defender alert; Microsoft 365 sandbox log endpoint does not respond within 30 seconds

Triage Agent creates Jira ticket with severity classification based on available Defender data only; log attachment field set to 'Log retrieval timed out — manual investigation required'; no agent halt

Jira ticket created; severity populated; log field contains timeout note; no unhandled exception; platform log records timeout event with timestamp

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: Use a pre-agreed internal test alert type in Microsoft Defender that is identifiable as a test event (for example, a labelled test policy match rather than a genuine threat signal). Use a dedicated test Jira ticket prefix (e.g. TEST-IR-001) so the record is easily identified and voided. Notify the on-call PagerDuty engineer before the test run so they are not alarmed by an incoming page. Void all test artefacts (Jira ticket, Notion page, PagerDuty incident, Slack messages) immediately after the process owner confirms the test result. Do not leave test records in the production incident log.

Scenario

Input

Expected Output

Pass Criteria

T25

Full production run: Critical alert lifecycle from ingestion to Slack summary

Pre-agreed test alert injected via the Microsoft Defender API into the live tenant; severity expected to classify as CRITICAL based on alert type and affected asset agreed with IT Manager in advance

Triage Agent classifies alert as CRITICAL within 90 seconds; Jira ticket TEST-IR-001 created with severity, summary, and Microsoft 365 log excerpts attached; Notification Agent posts formatted Slack message to #it-incidents with Jira link; PagerDuty incident opened on the production on-call policy (on-call engineer pre-notified); IT Manager receives Slack review prompt; after IT Manager approves (T26), Documentation Agent creates Notion report; post-incident Slack summary posted with Notion link

Each step fires in the correct sequence; all artefacts (Jira ticket, Slack alert, PagerDuty incident, Notion report, Slack summary) present and correctly populated; mean time from alert injection to Slack notification is under 2 minutes; no manual intervention required before the IT Manager review step

T26

Production IT Manager review step: approval and containment confirmation

IT Manager (Jordan Mercer) reviews the Jira ticket TEST-IR-001 and the Slack notification in the live environment; adds an approval comment and transitions the ticket to IN PROGRESS

Workflow continues from the human review step; Documentation Agent begins monitoring the ticket for a RESOLVED transition; no automated containment action is taken until this human approval is recorded

IT Manager comment recorded in Jira audit log; ticket status transitions to IN PROGRESS; no containment action fired before this step; Documentation Agent does not trigger prematurely

T27

Production error-log verification

Review the automation platform execution log and the logs generated during T25 and T26

Platform execution log shows a clean run with no unhandled exceptions, no silent failures, and no unexpected retries; all step durations are within acceptable thresholds (Triage Agent under 90 seconds, Notification Agent under 60 seconds, Documentation Agent under 120 seconds after ticket resolution)

Zero unhandled exceptions in the log; zero silent failures; step durations within thresholds; all API calls returned 2xx status codes; any retries that occurred are documented with reason and resolution

T28

Process owner review and approval: go-live acceptance

IT Manager (Jordan Mercer) reviews the complete test artefact set: Jira ticket TEST-IR-001, Slack notifications, PagerDuty incident record, Notion report page, and post-incident Slack summary; FullSpec team presents the execution log alongside the artefacts

IT Manager confirms that all outputs meet the agreed specification: severity classification is accurate, notifications routed correctly, Notion report is complete and structured, Slack summary is clear, and the human review step functioned as designed; approval is recorded in the FullSpec confirmation log; all test artefacts are voided from production systems

Written or verbal approval from the IT Manager captured by the FullSpec team and recorded in the FullSpec confirmation log; all test artefacts (Jira ticket, PagerDuty incident, Notion page) deleted or marked void; Slack test messages archived; the FullSpec confirmation log entry is the authoritative go-live acceptance record for this automation

Questions or issues during testing: contact the FullSpec team at support@gofullspec.com. Reference the process name 'Cybersecurity Incident Response' and the test case ID in your message to ensure the fastest response.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process