Test and QA Plan

How the build is validated before go-live: three sequential test phases, edge cases, and production verification.

4 pagesPDF · Quality

FS-DOC-06Quality

Test and QA Plan

Data Quality Monitoring

[YourCompany.com] · IT Department · Prepared by FullSpec · [Today's Date]

This document defines the full testing programme for the Data Quality Monitoring automation. It is written for the FullSpec build team and covers all unit, integration, edge-case, and end-to-end production tests required before go-live is approved. The FullSpec team runs Phases 1 and 2 entirely in sandbox environments. Phase 3 involves a controlled production run and requires the process owner (IT Manager, Sam Okafor) to review results and confirm acceptance before manual checks are decommissioned. No phase begins until the previous phase has fully passed.

01Testing approach

Testing is structured across three sequential phases: Unit, Integration, and End-to-end. Each phase must reach a full pass before the next phase is allowed to begin. No exceptions or partial waivers apply. Any failing test case must be remediated and re-run before progression. The FullSpec team owns execution for all three phases; the process owner joins Phase 3 to confirm real-world output meets operational expectations.

Phase 1: Unit Testing

Each agent tested in isolation against sandbox data

Scope

Each agent tested individually: Data Check Agent, Issue Routing and Notification Agent, Reporting and Metrics Agent. No cross-agent calls.

Environment

Sandbox only. No connection to production PostgreSQL, live Jira projects, or the production Slack channel.

Credential rule

Sandbox credentials exclusively. Production API keys, database connection strings, and Datadog API keys must not be present in the test environment at any point during Phase 1.

Pass condition

All test cases T01 through T14 return the expected output with no unhandled exceptions. Any failure halts phase progression.

Who runs it

FullSpec team only.

Phase 2: Integration and Edge Case Testing

Agent handoffs and failure scenarios validated

Scope

End-to-end agent handoffs (Data Check to Issue Routing to Reporting), plus all defined edge cases including duplicates, missing data, tool unavailability, and human timeout scenarios.

Environment

Sandbox environment with stubbed tool responses for Jira, Slack, Airtable, and Datadog. PostgreSQL sandbox schema populated with synthetic test records.

Credential rule

Sandbox credentials only. Stubbed webhook endpoints replace live Slack and Datadog destinations. Jira sandbox project used; no production board touched.

Pass condition

All integration test cases T15 through T22 and edge-case tests E01 through E07 pass without data loss, silent failures, or unrouted errors.

Who runs it

FullSpec team only.

Phase 3: End-to-End Production Test

Full production run with process owner sign-off

Scope

A single complete production cycle: scheduled trigger fires, all three agents execute against live data sources in read-only mode, Jira tickets are created in a designated QA project, Slack alert posts to a test channel, and Datadog dashboard receives a test push.

Environment

Production credentials active. PostgreSQL read-only service account connected. Jira QA project and dedicated test Slack channel used. Datadog live but writing to a staging dashboard widget.

Credential rule

Production API keys active for read operations. Write operations (Jira ticket creation, Airtable log rows, Datadog push) directed to QA or staging destinations. All test records voided after the run.

Pass condition

All production test cases T23 through T27 pass, including the final process owner review case T27. IT Manager confirms output accuracy before sign-off is recorded in the FullSpec confirmation.

Who runs it

FullSpec team executes; IT Manager (Sam Okafor) and IT Analyst (Jordan Reyes) join for result review and acceptance.

Test and QA PlanPage 1 of 4

FS-DOC-06Quality

02Phase 1 — unit test cases

Sandbox credentials only. Before executing any test in this phase, confirm that no production connection strings, API keys, or OAuth tokens are present in the test environment. Use only the designated sandbox PostgreSQL schema, the Airtable sandbox base, and stubbed endpoints for Jira, Slack, and Datadog. Any accidental production write during Phase 1 must be reported to support@gofullspec.com immediately.

Data Check Agent — complexity: Moderate. Estimated build time: 20 hours.

Scenario

Input

Expected output

Pass criteria

T01

Happy path: scheduled trigger fires, clean dataset

PostgreSQL sandbox table with 500 rows, all fields valid. Airtable sandbox base up to date.

Agent completes run, returns structured issue list with zero issues found. Execution log shows all rule checks ran.

Issue list length = 0; all 12 rules logged as executed; no exceptions thrown; run duration recorded.

T02

Null value detection in required field

100 rows in sandbox; 8 rows have NULL in the required 'customer_email' field.

Issue list contains exactly 8 entries for null_required_field on customer_email, severity scored as critical.

issue_count = 8; error_type = 'null_required_field'; field = 'customer_email'; severity = 'critical'.

T03

Email format validation failure

50 rows; 5 rows have malformed email strings (missing '@' or domain).

5 issues returned, type = 'format_mismatch', field = 'customer_email', severity = 'high'.

issue_count = 5; error_type = 'format_mismatch'; regex pattern applied matches configured rule.

T04

Duplicate key detection

200 rows; 3 pairs of duplicate 'order_id' values (6 affected rows).

3 duplicate issues returned, affected_row_count = 2 per issue, severity = 'high'.

issue_count = 3; error_type = 'duplicate_key'; field = 'order_id'; affected_row_count = 2 each.

T05

Numeric range violation

80 rows; 4 rows have 'quantity' value of -1 or 0, below the configured minimum of 1.

4 issues returned, type = 'out_of_range', field = 'quantity', severity = 'moderate'.

issue_count = 4; error_type = 'out_of_range'; severity = 'moderate'; min_threshold = 1 reflected in output.

T06

Import event trigger (non-scheduled path)

Synthetic import event payload sent to the trigger endpoint. Dataset contains 2 critical issues.

Agent fires immediately on event receipt, not waiting for scheduled time. Issues detected and list produced.

Trigger source = 'import_event' in run log; issue_count = 2; run completes within 60 seconds of event receipt.

T07

Database connection failure

PostgreSQL sandbox connection string deliberately misconfigured.

Agent catches the connection error, logs a 'source_unavailable' error entry, and does not crash the workflow.

No unhandled exception; error_type = 'source_unavailable' in error log; no partial issue list emitted.

T08

Empty dataset returned by query

PostgreSQL sandbox table truncated to zero rows. Query returns empty result set.

Agent logs a 'no_records_returned' warning, produces an issue list with zero entries, and completes without error.

issue_count = 0; warning logged = 'no_records_returned'; run status = 'completed_with_warning'.

T09

Ruleset config missing or malformed

Quality ruleset config file in Google Sheets sandbox deliberately emptied.

Agent halts run, logs 'ruleset_load_failed' error, and does not proceed to evaluation.

Run status = 'failed'; error = 'ruleset_load_failed'; no issue list emitted; error surfaced in run log.

Issue Routing and Notification Agent — complexity: Moderate. Estimated build time: 16 hours.

Scenario

Input

Expected output

Pass criteria

T10

Happy path: critical and high issues routed correctly

Issue list with 2 critical and 3 high severity entries from Data Check Agent output.

2 Jira tickets created (one per critical issue); all 5 issues written to Airtable log; Slack message posted with severity summary and Jira links.

Jira ticket count = 2; Airtable rows added = 5; Slack message contains issue count breakdown and at least 2 ticket URLs.

T11

Moderate and low issues: no Jira ticket, log only

Issue list with 4 moderate and 6 low severity entries only.

Zero Jira tickets created; 10 Airtable rows written; Slack message posted noting no critical issues.

Jira ticket count = 0; Airtable rows = 10; Slack message indicates 'no critical issues this run'.

T12

Jira API unavailable

Issue list with 1 critical issue. Jira sandbox endpoint returns 503.

Agent logs 'jira_unavailable' error; Airtable row still written; Slack alert still posted with note that ticket creation failed.

Airtable row written = 1; Slack message includes 'ticket creation failed' notice; error log entry = 'jira_unavailable'; no unhandled exception.

T13

Slack API unavailable

Issue list with 2 high severity issues. Slack sandbox webhook returns 500.

Agent logs 'slack_notification_failed'; Jira tickets created successfully; Airtable rows written. Alert failure recorded but run completes.

Jira tickets = 2; Airtable rows = 2; error log = 'slack_notification_failed'; run status = 'completed_with_errors'.

T14

Empty issue list received

Issue list from Data Check Agent contains zero issues.

No Jira tickets created; no Airtable rows added for this run (or a zero-issue run row is written per config); Slack message confirms clean run.

Jira ticket count = 0; Slack message = 'No issues detected in this run'; run completes without error.

Reporting and Metrics Agent — complexity: Simple. Estimated build time: 12 hours.

Scenario

Input

Expected output

Pass criteria

T15

Happy path: weekly aggregation and Datadog push

Airtable sandbox log contains 14 issue rows across the week: 3 critical, 5 high, 4 moderate, 2 low. 9 rows marked resolved.

Datadog sandbox receives a metric push with total_issues = 14, critical = 3, high = 5, moderate = 4, low = 2, resolved = 9, resolution_rate = 64%.

All five metric fields present in Datadog payload; values match Airtable source data; push timestamp within 5 minutes of weekly trigger.

T16

Week with zero issues logged

Airtable sandbox log contains zero rows for the current week.

Datadog push sent with all counts = 0 and resolution_rate = 100% (or N/A per config). Run completes without error.

Datadog payload present; total_issues = 0; no exception; run status = 'completed'.

T17

Datadog API key invalid or endpoint unreachable

Datadog sandbox endpoint returns 403.

Agent logs 'datadog_push_failed'; aggregation data retained in memory or written to a fallback Airtable row for retry. Run does not crash.

Error log = 'datadog_push_failed'; run status = 'completed_with_errors'; no data silently discarded.

Test and QA PlanPage 2 of 4

FS-DOC-06Quality

03Phase 2 — integration and edge case tests

Integration tests verify that agents hand off data correctly in sequence and that the full pipeline from trigger to Slack alert operates as designed. Test IDs continue from the Phase 1 sequence.

Scenario

Input

Expected output

Pass criteria

T18

Full handoff: Data Check Agent to Issue Routing Agent

Scheduled trigger fires. Sandbox PostgreSQL contains 3 critical, 2 high, 4 moderate issues across 2 tables.

Data Check Agent emits structured issue list; Issue Routing Agent receives it without transformation loss; 3 Jira tickets created; 9 Airtable rows written; Slack alert sent.

No fields dropped between agents; Jira count = 3; Airtable count = 9; Slack message received on test channel.

T19

Full handoff: Issue Routing Agent to Reporting Agent (end-of-week path)

Weekly trigger fires after T18 data is in Airtable sandbox. 9 issue rows present, 5 marked resolved.

Reporting Agent reads Airtable, aggregates correctly, pushes to Datadog sandbox. Metrics match the 9-row dataset.

Datadog payload: total = 9, resolved = 5, resolution_rate = 56%; timestamp present; no data from prior weeks included.

T20

Import event overrides scheduled timing

Import event fires 3 hours before the next scheduled run. Dataset contains 1 critical issue.

Data Check Agent fires immediately on event; does not wait for schedule. Full pipeline completes. Next scheduled run fires at its normal time and processes updated data.

Run log shows trigger_source = 'import_event'; second run at scheduled time completes independently; no duplicate issue rows in Airtable.

T21

Multiple data sources in single run

Sandbox configured with both a PostgreSQL table and an Airtable base as watched sources. Each contains distinct issues.

Issue list combines findings from both sources. Source field in each issue row correctly identifies origin (PostgreSQL or Airtable).

Issues from both sources present; source field populated correctly for every row; no cross-source data mixing.

T22

Partial run: one source fails mid-pipeline

PostgreSQL sandbox returns data normally. Airtable sandbox API returns 503 mid-run.

Agent logs 'airtable_source_unavailable' for the failed source; processes PostgreSQL data successfully; Airtable issues section of output flagged as incomplete.

PostgreSQL issues present in output; Airtable section flagged as 'source_error'; run status = 'partial'; no silent data loss.

Edge case tests validate system behaviour under abnormal or adversarial conditions. These use the E prefix and are independent of the T sequence.

Scenario

Input

Expected output

Pass criteria

E01

Duplicate import event: same event fires twice within 60 seconds

Import event payload sent twice with identical event_id and timestamp.

Deduplication logic prevents a second run from starting. Only one issue list produced. One set of Airtable rows written.

Run log shows second event rejected with 'duplicate_event_id'; Airtable contains no duplicate rows for the run.

E02

Missing required fields in incoming issue list

Data Check Agent emits a malformed issue list with 'severity' field absent from 3 entries.

Issue Routing Agent catches missing severity, applies a default severity of 'low', logs a 'missing_severity_field' warning, and continues processing.

3 rows in Airtable show severity = 'low' (default); warning logged; no rows dropped; no exception thrown.

E03

Extremely large dataset: 50,000 rows in PostgreSQL

Sandbox PostgreSQL table loaded with 50,000 rows, 200 containing issues.

Data Check Agent completes run within 5 minutes. Issue list contains exactly 200 entries. No timeout or memory error.

Run completes; issue_count = 200; run_duration <= 5 minutes; no timeout errors in log.

E04

All tools unavailable simultaneously

Jira, Slack, and Airtable sandbox endpoints all return 503.

Agent logs individual failures for each tool; queues or retains output for retry; does not enter an infinite retry loop; sends a failure summary to the configured fallback alert address.

Three separate error entries logged; retry count does not exceed configured max (3); fallback notification sent; run status = 'failed'.

E05

Ruleset contains a rule with an invalid regex pattern

Quality ruleset config includes one rule with a syntactically invalid regex for email validation.

Data Check Agent skips the invalid rule, logs 'invalid_rule_config' for that rule ID, and continues executing all other rules.

Rule skipped; warning logged with rule_id; all other rules execute; no unhandled exception; issue list produced for remaining rules.

E06

Airtable log base at row capacity (approaching plan limit)

Airtable sandbox configured to simulate a base approaching its record limit. Write attempt returns a 422 capacity error.

Agent catches the Airtable write error, logs 'airtable_capacity_warning', and falls back to writing issue data to a configured Google Sheets fallback tab.

Airtable write fails gracefully; Google Sheets fallback row written; warning logged; Slack alert includes capacity warning notice.

E07

Weekly trigger fires but Airtable log contains no rows for the week

Reporting Agent weekly trigger fires. Airtable sandbox has zero rows for the current ISO week.

Reporting Agent produces a zero-issue summary and pushes to Datadog. Does not error or skip the push entirely.

Datadog payload present with all counts = 0; run status = 'completed'; no exception; weekly cadence maintained.

Test and QA PlanPage 3 of 4

FS-DOC-06Quality

04Phase 3 — end-to-end production test

Important before running Phase 3: Use the designated QA test Slack channel and Jira QA project for all write operations. Do not post to the production IT alert channel or create tickets in the live Jira board. After the test run completes, void all test-generated Airtable rows by setting their record_type field to 'QA_TEST' and archiving them. Confirm with the FullSpec team that the Datadog staging widget has been reset before the first live production push. Any test records not voided within 24 hours of the run must be flagged to support@gofullspec.com.

Scenario

Input

Expected output

Pass criteria

T23

Full production cycle: scheduled trigger through to Slack alert

Production PostgreSQL database queried via read-only service account. Live data. Jira QA project and test Slack channel as write targets.

Data Check Agent completes run against live data. Issue list produced with accurate severity scores. Issue Routing Agent creates Jira tickets in QA project, writes Airtable rows, and posts Slack summary to test channel. All three steps complete without error.

Run completes end-to-end; issue list produced; at least 1 Airtable row written (or zero-issue run confirmed); Jira QA ticket created for any critical finding; Slack test channel receives formatted message; no exceptions in production run log.

T24

Severity accuracy spot-check against known production issues

FullSpec team pre-seeds a known critical issue (NULL in a required field) in the QA test schema or confirms a known live issue with the IT Analyst before the run.

The known issue appears in the issue list with severity = 'critical'. Affected row count matches the pre-confirmed count. Jira QA ticket is created for it.

Known issue present in output; severity = 'critical'; row count within 5% of pre-confirmed value; Jira ticket created.

T25

Datadog staging push: weekly metrics accuracy

Weekly trigger fires (or is manually invoked) after T23 data is in Airtable. Staging Datadog dashboard configured.

Reporting Agent aggregates the QA run data and pushes to the Datadog staging widget. Metric values are visible in the dashboard within 10 minutes.

Datadog staging widget shows updated metrics; total_issues reflects T23 run output; push timestamp is current; no authentication errors.

T26

Error log verification: confirm all run events are captured

Review automation platform run log after T23 and T24 complete.

Run log contains an entry for every step: trigger received, PostgreSQL query executed, rules evaluated, issue list emitted, Airtable write confirmed, Jira write confirmed, Slack send confirmed. Timestamps present for each step.

All 7 pipeline steps have corresponding log entries; no step has a gap of more than 2 minutes without a log event; any warning-level entries are reviewed and accepted by FullSpec team.

T27

Process owner review and go-live acceptance

IT Manager (Sam Okafor) reviews: the Jira QA tickets from T23, the Airtable issue log rows, the Slack test channel message, and the Datadog staging dashboard from T25.

IT Manager confirms that detected issues match known data quality problems in the live system, that severity scores are accurate, that Jira ticket content is complete and actionable, and that the Slack alert format is clear. Sign-off recorded in the FullSpec confirmation.

IT Manager provides written confirmation via the FullSpec confirmation record that all output meets operational expectations. No outstanding objections. FullSpec team marks Phase 3 as passed and schedules go-live cutover. Process owner approval is the final acceptance gate.

Once T27 is confirmed, the FullSpec team will coordinate the go-live cutover: production credentials activated for all write destinations, the Jira QA project replaced with the live IT project, the test Slack channel replaced with the production IT alert channel, and the Datadog staging widget replaced with the live dashboard. The IT Analyst will be notified before and after the switch. Manual data checks are decommissioned only after the first fully automated production run completes without error.

Test and QA PlanPage 4 of 4

Test and QA Plan

Test and QA Plan

01Testing approach

02Phase 1 — unit test cases

03Phase 2 — integration and edge case tests

04Phase 3 — end-to-end production test

More documents for this process