Back to SLA Monitoring & Breach Alerting

Developer Handover Pack

Everything needed to build the automation from scratch: current state, full agent specs, data flow, and the build stack.

4 pagesPDF · Technical
FS-DOC-04Technical

Developer Handover Pack

SLA Monitoring & Breach Alerting

[YourCompany.com] · Customer Support Department · Prepared by FullSpec · [Today's Date]

This document provides the complete technical specification for building the SLA Monitoring and Breach Alerting automation. It covers the current-state process map with bottleneck annotations, full agent specifications with IO contracts, the end-to-end data flow across both agents, and the recommended build stack with estimated effort. The FullSpec team uses this document to guide every build and configuration decision. No external developer or agency is involved.

01Process snapshot

Step
Name
Description
1
Open Ticket Queue Review
Support Lead opens the Zendesk dashboard and filters for all open tickets, visually scanning for tickets close to their SLA deadline. (15 min/cycle)
2
Identify Client SLA Tier
For each flagged ticket the lead cross-references the HubSpot client record or a spreadsheet to determine which SLA tier applies, as windows differ by contract. (10 min/cycle)
3
Calculate Time Remaining
The lead manually calculates hours remaining against the SLA window using the ticket creation timestamp and contracted deadline. Business-hours-only SLAs require extra arithmetic. (12 min/cycle)
4
Prioritise Breach-Risk Tickets
At-risk tickets are sorted by urgency and assigned a priority label or internal note in Zendesk. Tickets within 30 minutes of breach are flagged critical. (8 min/cycle)
5
Notify Assigned Agent
The lead sends a direct Slack message or email to the responsible agent including ticket number, client name, and minutes remaining. (10 min/cycle)
6
Escalate Unresolved Critical Tickets
If a ticket breaches or the assigned agent does not respond within 10 minutes, the lead escalates to the support manager via Slack or phone. Relies entirely on the lead being present. (8 min/cycle)
7
Update Ticket Status and Notes
The lead posts an internal note in Zendesk recording when the warning was issued and who was contacted. (6 min/cycle)
8
Log Breach Events
Any breached ticket is recorded in a tracking spreadsheet with client name, ticket ID, SLA tier, and minutes overrun, feeding the weekly report. (10 min/cycle)
9
Compile Weekly SLA Report
The lead pulls breach data from the spreadsheet and formats a summary email for the support manager covering breach count, affected clients, and SLA tier breakdown. (25 min/cycle)
Time cost summary: Each full monitoring cycle totals 104 minutes of manual work. At approximately 6 cycles per week (across daily checks and end-of-week reporting) this equals 6 hours of manual effort per week and roughly 26 hours per 30-day period. The automation replaces steps 1 through 9 entirely. The one remaining human action is the support manager's review and response to a PagerDuty escalation (step 6 equivalent), which requires client relationship judgement that the automation deliberately does not replace.
Developer Handover PackPage 1 of 4
FS-DOC-04Technical

02Agent specifications

SLA Monitor Agent

Fires on a fixed 15-minute schedule and polls Zendesk for every open ticket. For each ticket it retrieves the linked client identifier and queries HubSpot to determine the contracted SLA tier, response window, and resolution window. It then calculates exact business-hours time elapsed since ticket creation and the time remaining against the contracted deadline, applying per-tier timezone-aware business-hours calendars. Each ticket exits this agent classified as safe (under 75% of window consumed), at-risk (75% to 94%), or critical (95% and above, or already breached). Estimated build time: 14 hours. Complexity: Moderate.

Trigger
Scheduled poll every 15 minutes via the automation platform's built-in scheduler. No webhook; schedule-driven only.
Tools
Zendesk (ticket list retrieval, status and field read), HubSpot (company record lookup by client ID, SLA tier field read)
Replaces steps
Steps 1, 2, 3, 4 — queue scan, SLA tier lookup, business-hours calculation, and urgency classification
Estimated build
14 hours — Moderate
// Input
zendesk.tickets.list({ status: 'open' })
  -> ticket_id: string
  -> subject: string
  -> created_at: ISO8601 timestamp
  -> assignee_id: integer
  -> organization_id: integer  // used as client_id for HubSpot lookup
  -> zendesk_custom_field: sla_tier_override (optional, string)

hubspot.companies.get({ id: client_id })
  -> hs_object_id: string
  -> sla_tier: string  // e.g. 'Tier1', 'Tier2', 'Tier3'
  -> sla_response_hours: integer
  -> sla_resolution_hours: integer
  -> business_hours_calendar: string  // named calendar ID
  -> client_timezone: string  // IANA timezone e.g. 'America/New_York'

// Output
classified_ticket_list: array of {
  ticket_id: string,
  subject: string,
  assignee_id: integer,
  client_id: string,
  client_name: string,
  sla_tier: string,
  sla_response_hours: integer,
  sla_resolution_hours: integer,
  business_hours_elapsed: float,
  business_hours_remaining: float,
  sla_pct_consumed: float,  // 0.0 to 1.0+
  urgency_flag: enum('safe', 'at_risk', 'critical', 'breached'),
  alert_sent_75: boolean,  // dedupe flag, read from Zendesk tag or state store
  alert_sent_95: boolean   // dedupe flag
}
  • Zendesk API tier must support organization field reads and custom field retrieval. Confirm the account is on Team plan or above before build starts; the Essential plan does not expose all custom fields via REST.
  • The HubSpot SLA tier property must be named exactly 'sla_tier' on the Company object. Confirm property name and enumeration values with the client before building the lookup step. If the field is missing or blank the agent must fail gracefully and route the ticket to a fallback classification of 'at_risk' with a warning flag, never silently drop it.
  • Business-hours calendar logic must account for public holidays per client timezone. Build a configurable holiday list as a static lookup table within the automation platform rather than relying on an external calendar API, to reduce external dependencies.
  • Deduplification for the alert_sent_75 and alert_sent_95 flags must persist across poll cycles. Use a lightweight state store (a Supabase table or equivalent) keyed on ticket_id. Without this, the agent will resend alerts every 15 minutes for the same ticket.
  • If the Zendesk poll returns more than 100 tickets, pagination must be handled explicitly using the Zendesk cursor-based pagination API. The agent must loop until next_page is null.
  • Confirm the Zendesk organization_id field is consistently populated on all tickets. Tickets with a null organization_id cannot be matched to a HubSpot company and must be routed to the fallback path, not discarded.
Alert Dispatch Agent

Receives the classified ticket list from the SLA Monitor Agent after every poll cycle and routes graded outputs to five downstream systems based on each ticket's urgency flag and current alert state. For at-risk tickets (75% threshold crossed, alert not yet sent) it posts a structured Slack warning to the assigned agent's channel. For critical or breached tickets (95% threshold, escalation not yet fired) it raises a PagerDuty incident to the on-call support manager. For all tickets that triggered any alert it posts a timestamped internal note to the Zendesk ticket. For confirmed breaches it creates a breach activity record on the HubSpot company record. Every Monday morning at 08:00 it compiles the prior week's breach and near-breach data and delivers a formatted summary email via Gmail. Estimated build time: 14 hours. Complexity: Moderate.

Trigger
Receives classified_ticket_list payload from SLA Monitor Agent after each 15-minute poll cycle. Weekly Gmail report is triggered by a separate Monday 08:00 schedule branch within the same workflow.
Tools
Slack (agent warning DM or channel post), PagerDuty (incident creation via Events API v2), Zendesk (internal note POST), HubSpot (breach activity record creation), Gmail (weekly summary send via OAuth)
Replaces steps
Steps 5, 6, 7, 8, 9 — agent notification, critical escalation, Zendesk note, breach logging, and weekly report compilation
Estimated build
14 hours — Moderate
// Input
classified_ticket_list: array (from SLA Monitor Agent output, see above)
  filtered to tickets where urgency_flag IN ('at_risk', 'critical', 'breached')
  and (alert_sent_75 == false OR alert_sent_95 == false)

// Output — per ticket, routed conditionally

// Slack (urgency_flag == 'at_risk' AND alert_sent_75 == false)
slack.chat.postMessage({
  channel: assignee_slack_id,  // resolved via Zendesk assignee_id -> Slack user map
  text: '[SLA WARNING] Ticket #{{ticket_id}} — {{client_name}} ({{sla_tier}}) is at {{sla_pct_consumed}}% of SLA window. {{business_hours_remaining}} business hours remaining.',
  blocks: structured_card_with_ticket_link
})

// PagerDuty (urgency_flag IN ('critical', 'breached') AND alert_sent_95 == false)
pagerduty.events.send({
  routing_key: PAGERDUTY_INTEGRATION_KEY,
  event_action: 'trigger',
  dedup_key: 'sla-{{ticket_id}}-95',
  payload: {
    summary: 'SLA critical: Ticket #{{ticket_id}} for {{client_name}} at {{sla_pct_consumed}}%',
    severity: 'critical',
    source: 'SLA Monitor Automation',
    custom_details: { ticket_id, client_id, sla_tier, business_hours_remaining }
  }
})

// Zendesk internal note (all alerted tickets)
zendesk.tickets.update({
  id: ticket_id,
  comment: {
    body: '[AUTOMATED] SLA alert sent at {{sla_pct_consumed}}% threshold. Notified: {{recipient}}. Timestamp: {{iso_timestamp}}.',
    public: false
  }
})

// HubSpot breach record (urgency_flag == 'breached')
hubspot.crm.objects.notes.create({
  associations: [{ to: { id: hs_object_id }, type: 'company_to_note' }],
  properties: {
    hs_note_body: 'SLA breach: Ticket #{{ticket_id}}, Tier {{sla_tier}}, {{minutes_overrun}} min overrun. Breached at {{breach_timestamp}}.',
    hs_timestamp: breach_timestamp
  }
})

// Gmail weekly summary (Monday 08:00 schedule branch)
gmail.send({
  to: MANAGER_EMAIL,
  subject: 'Weekly SLA Report — week ending {{week_end_date}}',
  body: compiled_html_summary  // breach count, near-breach count, tier breakdown, ticket list
})
  • PagerDuty escalation policy and on-call schedule must be created and tested in PagerDuty before this agent is built. The PAGERDUTY_INTEGRATION_KEY must correspond to the correct service. Confirm the on-call rotation includes at least one override for holidays.
  • Slack user ID mapping from Zendesk assignee_id requires a maintained lookup table (Zendesk agent ID to Slack user ID). Build this as a configurable map in the credential or config store. If a Slack ID is not found for an assignee, fall back to posting to a designated support-alerts channel rather than failing silently.
  • The PagerDuty dedup_key pattern 'sla-{{ticket_id}}-95' ensures that repeated poll cycles do not create duplicate PagerDuty incidents for the same ticket. Confirm PagerDuty deduplication behaviour in the target account tier (Professional or above required for full dedup support).
  • Gmail send must use OAuth 2.0 with the https://www.googleapis.com/auth/gmail.send scope only. Do not request broader Gmail scopes. The sending account must be a shared service account, not a personal inbox.
  • The weekly Gmail report data source is the Supabase breach log table populated during the week, not a live Zendesk query. Confirm the table schema is finalised before building this branch.
  • HubSpot breach notes must be associated to the Company object, not the Contact object. Confirm the correct object type with the client's HubSpot admin before build. If the client uses Deals to track SLA contracts, an additional association may be required.
  • Confirm PagerDuty account tier supports the Events API v2 endpoint. Some legacy PagerDuty accounts are on v1 only and require a different payload structure.
Developer Handover PackPage 2 of 4
FS-DOC-04Technical

03End-to-end data flow

Full trigger-to-output data flow for SLA Monitoring and Breach Alerting. Field names are exact and must match Zendesk and HubSpot property names confirmed during the data audit.
// ============================================================
// SLA MONITORING & BREACH ALERTING — END-TO-END DATA FLOW
// ============================================================

// TRIGGER: Automation platform scheduler fires every 15 minutes
// ____________________________________________________________

SCHEDULER.fire({ interval: '15m', timezone: 'UTC' })
  -> initiates: SLA_MONITOR_AGENT run

// ============================================================
// AGENT 1: SLA MONITOR AGENT
// ============================================================

// Step 1: Fetch all open tickets from Zendesk
ZENDESK.GET /api/v2/tickets.json?status=open&per_page=100
  -> paginate via next_page cursor until null
  -> extract per ticket: {
       ticket_id,          // string, Zendesk ticket ID
       subject,            // string
       created_at,         // ISO8601 UTC timestamp
       assignee_id,        // integer, Zendesk agent ID
       organization_id,    // integer, maps to HubSpot company
       custom_field[sla_tier_override]  // string or null
     }

// Step 2: For each ticket, fetch SLA tier from HubSpot
// Keyed on: organization_id -> HubSpot company hs_object_id
HUBSPOT.GET /crm/v3/objects/companies/{hs_object_id}
  ?properties=sla_tier,sla_response_hours,sla_resolution_hours,
              business_hours_calendar,client_timezone
  -> extract: {
       sla_tier,               // string: 'Tier1' | 'Tier2' | 'Tier3'
       sla_response_hours,     // integer: e.g. 4 | 8 | 24
       sla_resolution_hours,   // integer: e.g. 8 | 24 | 72
       business_hours_calendar,// string: named calendar ID
       client_timezone         // string: IANA e.g. 'America/Chicago'
     }
  // FALLBACK: if organization_id is null or HubSpot returns 404,
  // classify ticket as urgency_flag='at_risk', log warning to Supabase.

// Step 3: Calculate business-hours time remaining
// Inputs: created_at, sla_response_hours, business_hours_calendar, client_timezone
CALC.businessHoursElapsed({
  start: created_at,
  now: SCHEDULER.fired_at,
  calendar: business_hours_calendar,  // weekday start/end, holiday list
  timezone: client_timezone
})
  -> business_hours_elapsed: float   // e.g. 3.5
  -> business_hours_remaining: float // sla_response_hours - elapsed
  -> sla_pct_consumed: float         // elapsed / sla_response_hours

// Step 4: Classify urgency
IF sla_pct_consumed < 0.75  -> urgency_flag = 'safe'
IF sla_pct_consumed >= 0.75 AND < 0.95 -> urgency_flag = 'at_risk'
IF sla_pct_consumed >= 0.95 AND <= 1.0 -> urgency_flag = 'critical'
IF sla_pct_consumed > 1.0   -> urgency_flag = 'breached'

// Step 5: Read dedupe flags from Supabase state store
SUPABASE.SELECT alert_sent_75, alert_sent_95
  FROM sla_alert_state WHERE ticket_id = ticket_id
  -> alert_sent_75: boolean
  -> alert_sent_95: boolean

// AGENT HANDOFF: SLA Monitor Agent -> Alert Dispatch Agent
// Payload: classified_ticket_list (full array, safe tickets excluded)
// Only tickets where urgency_flag != 'safe' are forwarded

// ============================================================
// AGENT 2: ALERT DISPATCH AGENT
// ============================================================

// Branch A: Slack warning (at_risk, alert_sent_75 == false)
SLACK.POST /api/chat.postMessage
  -> channel: assignee_slack_id  // from Zendesk_to_Slack lookup map
  -> text: '[SLA WARNING] Ticket #{{ticket_id}} — {{client_name}}'
           + ' ({{sla_tier}}) at {{sla_pct_consumed}}%.'
           + ' {{business_hours_remaining}} business hours remaining.'
  -> blocks: structured card with ticket deep link
  // On success: SUPABASE.UPDATE sla_alert_state SET alert_sent_75=true
  //             WHERE ticket_id = ticket_id
  // Fallback: if assignee_slack_id not found, post to #support-alerts channel

// Branch B: PagerDuty escalation (critical or breached, alert_sent_95 == false)
PAGERDUTY.POST /v2/enqueue
  -> routing_key: PAGERDUTY_INTEGRATION_KEY
  -> event_action: 'trigger'
  -> dedup_key: 'sla-{{ticket_id}}-95'
  -> payload.summary: 'SLA critical: Ticket #{{ticket_id}} {{sla_pct_consumed}}%'
  -> payload.severity: 'critical'
  -> payload.custom_details: { ticket_id, client_id, sla_tier,
                               business_hours_remaining, assignee_id }
  // On success: SUPABASE.UPDATE sla_alert_state SET alert_sent_95=true
  //             WHERE ticket_id = ticket_id

// Branch C: Zendesk internal note (all alerted tickets)
ZENDESK.PUT /api/v2/tickets/{{ticket_id}}.json
  -> comment.body: '[AUTOMATED] SLA alert fired at {{sla_pct_consumed}}%.'
                  + ' Notified: {{recipient}}. At: {{iso_timestamp}}.'
  -> comment.public: false

// Branch D: HubSpot breach record (urgency_flag == 'breached')
HUBSPOT.POST /crm/v3/objects/notes
  -> properties.hs_note_body: 'SLA breach: Ticket #{{ticket_id}},'
                             + ' Tier {{sla_tier}},'
                             + ' {{minutes_overrun}} min overrun.'
  -> properties.hs_timestamp: breach_timestamp
  -> associations: [{ to: hs_object_id, type: 'company_to_note' }]

// Branch E: Gmail weekly summary (Monday 08:00 UTC schedule branch)
// Data source: Supabase sla_breach_log table, prior 7 days
SUPABASE.SELECT ticket_id, client_name, sla_tier, minutes_overrun,
                breach_timestamp, near_breach_flag
  FROM sla_breach_log
  WHERE breach_timestamp >= NOW() - INTERVAL '7 days'
  -> compile: breach_count, near_breach_count, tier_breakdown, ticket_list
GMAIL.POST /gmail/v1/users/me/messages/send
  -> to: MANAGER_EMAIL
  -> subject: 'Weekly SLA Report — week ending {{week_end_date}}'
  -> body: compiled_html_summary (breach_count, near_breach_count,
           tier_breakdown table, full ticket list with client names)

// ============================================================
// END OF FLOW — cycle repeats on next 15-minute scheduler tick
// ============================================================
Developer Handover PackPage 3 of 4
FS-DOC-04Technical

04Recommended build stack

Orchestration layer
A workflow automation platform (not yet selected) configured with one workflow per agent: 'SLA Monitor Agent' workflow and 'Alert Dispatch Agent' workflow. Both workflows share a single credential store. The Alert Dispatch Agent workflow is triggered by the output of the SLA Monitor Agent workflow within the same execution run, not by a separate webhook. A third workflow handles the Monday 08:00 Gmail summary on its own schedule branch.
Webhook configuration
No inbound webhooks are required for the core 15-minute polling loop; the scheduler is the only trigger. An optional inbound webhook endpoint may be configured for manual test triggers during QA. PagerDuty uses an outbound Events API v2 POST (not a webhook receiver). Slack uses the Web API chat.postMessage method, not incoming webhooks, to support per-user DM routing.
Templating approach
All alert message bodies (Slack cards, Zendesk internal notes, HubSpot note text, Gmail summary HTML) are stored as named templates in the automation platform's template store. Template variables use double-brace syntax: {{ticket_id}}, {{client_name}}, {{sla_tier}}, {{sla_pct_consumed}}, {{business_hours_remaining}}, {{iso_timestamp}}. The Gmail weekly summary uses a dedicated HTML template with an inline table for the tier breakdown. Templates are version-controlled alongside the workflow definitions.
Error logging
All agent execution errors (Zendesk pagination failures, HubSpot 404s, Slack user-not-found, PagerDuty send failures, Gmail auth errors) are written to a Supabase table named 'sla_automation_errors' with columns: error_id (uuid), agent_name (text), ticket_id (text, nullable), error_type (text), error_message (text), occurred_at (timestamptz). A Supabase database webhook fires a Slack alert to the #automation-alerts channel when any row is inserted, ensuring the FullSpec team is notified of runtime failures without polling.
State store (dedupe)
Supabase table 'sla_alert_state' with columns: ticket_id (text, primary key), alert_sent_75 (boolean, default false), alert_sent_95 (boolean, default false), last_seen_at (timestamptz), resolved_at (timestamptz, nullable). Rows are created on first ticket encounter and updated on each alert dispatch. Rows for resolved or closed tickets are soft-deleted by setting resolved_at. A nightly cleanup job removes rows older than 30 days.
Testing approach
All build and integration work is completed against sandbox environments first: Zendesk sandbox instance, HubSpot sandbox portal, Slack test workspace, PagerDuty staging service, and a Gmail test account. Production credentials are not loaded until end-to-end QA is signed off. Historical ticket data from Zendesk is used to replay past cycles and verify classification accuracy across all SLA tiers. Threshold triggers at 75% and 95% are tested by seeding tickets with backdated creation timestamps in the sandbox Zendesk instance.
Estimated total build time
SLA Monitor Agent: 14 hours. Alert Dispatch Agent: 14 hours. End-to-end integration testing and QA (cross-system, threshold validation, PagerDuty routing, Gmail delivery): 6 hours. Total: 34 hours across 4 weeks. Note: the template snapshot records 28 hours for core agent build effort; the 6-hour QA allowance is additional and consistent with the 4-week delivery schedule.
Pre-build blockers: The following must be confirmed before any build work begins. (1) HubSpot Company property 'sla_tier' exists and is populated for all active clients. (2) Zendesk organization_id is consistently set on all open tickets. (3) Business-hours calendar definitions are documented per SLA tier, including timezone and public holiday rules. (4) PagerDuty escalation policy and on-call schedule are active and verified. (5) Zendesk-to-Slack agent ID mapping is available or can be produced by the support team. None of these can be resolved during build without delaying delivery.
Support contact: For any questions about this specification or to flag a data gap before build begins, contact the FullSpec team at support@gofullspec.com.
Developer Handover PackPage 4 of 4

More documents for this process

Every document generated for SLA Monitoring & Breach Alerting.

Launch Plan
Operations · Owner
View
ROI and Business Case
Finance · Owner
View
Process Runbook / SOP
Operations · Owner
View
Integration and API Spec
Technical · Developer
View
Test and QA Plan
Quality · Developer
View