Offboarding Operator Runbook

Operational reference for SREs and on-call engineers responding to issues in the Offboarding module (spec 43) — the request lifecycle, the five scheduled worker scans, the access-revocation queue, and the exit-interview portal.

Audience. SRE / on-call. Assumes shell access to the production server (serge@clear) and a working psql session against the production database. For end-user docs see /offboarding and /settings/offboarding-* in-app.

---

Quick reference — service map

Component	Container	Notes
API	production-api-1 / staging-api-1	Routes under /api/v1/offboarding/* + /api/v1/exit-interview/*
Worker (BullMQ)	production-worker-1 / staging-worker-1	Runs the orchestrate consumer + the five scheduled scanners
Postgres	production-postgres-1	Tables: offboarding_requests, offboarding_tasks, offboarding_exit_interviews, offboarding_access_revocations, offboarding_asset_returns, offboarding_kt_templates*, offboarding_revocation_configs, departure_reason_codes, employee_alumni_profiles, alumni_rehire_invitations, perf_hub_attrition_signal_labels. Plus the shared worker_cron_completions ledger.
Redis	production-redis-1	BullMQ queues + the queue:offboarding-orchestrate raw-list channel the API enqueues on approval

Scheduled cron jobs

All times UTC. None of these scanners have an upstream chain gate — they read offboarding state directly and write idempotent marker columns.

Job (queue_name in worker_cron_completions)	Cadence	Spec	Marker column on row
offboarding-sla-scanner	0 8 * * * (08:00)	§7A.5	sla_warning_sent_at, sla_breach_sent_at on offboarding_requests
offboarding-final-day-countdown	0 9 * * * (09:00)	§7A.4	t_minus_7_sent_at / _3_ / _1_ on offboarding_requests
offboarding-contract-end-scanner	0 5 * * * (05:00)	§7D	contract_end_cr_seeded_at on employees
offboarding-access-revocation-execute	0 2 * * * (02:00)	§7C.4	due_notified_at on offboarding_access_revocations
offboarding-kt-escalation	0 10 * * * (10:00)	§10.5	escalation_amber_sent_at / _red_ on offboarding_tasks

Event-driven worker

Job	Trigger	Spec	Idempotency
offboarding-orchestrate	LPUSH on queue:offboarding-orchestrate from the API at request approval	§7A.1	offboarding_tasks COUNT check at step 0 + new (mig 214) tenant kill-switch check at step 0a

---

The status panel

HR admins (and anyone with offboarding.analytics.read) can hit GET /api/v1/offboarding/operational-status — your first stop on a triage call.

It returns:

• tenant — the kill-switch state (offboarding_enabled from tenant_configs, mig 214). When false, all API mutations 503 and the orchestrate worker skips the tenant; reads keep working.
• workers — one row per scheduled cron (always all 5; missing entries surface as status: "never_ran"), each with last_completed_at, age_hours, a band (fresh ≤ 26h / stale ≤ 48h / critical / never_ran), and the last_run_metadata blob.
• in_flight — current backlog: open_requests, open_requests_past_sla (sla_breach already fired and the request is still open), pending_manual_revocations (SCHEDULED MANUAL on an open request), failed_revocations (automated WEBHOOK revocations in FAILED on an open request — these need IT to revoke manually; see §7C.4), pending_exit_interviews, kt_tasks_overdue (past due date, not COMPLETED/WAIVED/CANCELLED).
• thresholds — pinned to the spec-27 panel bands for muscle memory.

curl -s -H "Authorization: Bearer $TOK" \
  https://api.frontlinehq.io/api/v1/offboarding/operational-status | jq

If workers[].status is critical or never_ran, see the per-worker diagnostics below. If in_flight.open_requests_past_sla > 0, escalate to HR — the request is in genuine breach. If in_flight.failed_revocations > 0, an automated deprovision failed — fix the tenant's webhook_url/headers and reset the row (see the offboarding-access-revocation-execute section).

---

Per-worker diagnostics

offboarding-sla-scanner (§7A.5)

What it does. Daily 08:00 UTC. Reads offboarding_requests where status = 'approved' and final_day is in the past; emits sla_warning at (offboarding_sla_days − 2) business days, sla_breach at offboarding_sla_days business days. Idempotent via sla_warning_sent_at / sla_breach_sent_at columns.

Diagnostic SQL:

-- requests that SHOULD have a warning but don't (the scanner hasn't
-- processed them yet, or the row was inserted after today's cron run)
SELECT obr.id, e.work_email, obr.final_day, obr.sla_warning_sent_at, obr.sla_breach_sent_at
  FROM offboarding_requests obr
  JOIN employees e ON e.id = obr.employee_id
 WHERE obr.tenant_id = $1
   AND obr.status = 'approved'
   AND obr.final_day < CURRENT_DATE - 3
   AND obr.sla_warning_sent_at IS NULL
 ORDER BY obr.final_day;

Recovery — replay a single request:

-- nullify the marker so the next scan reprocesses it
UPDATE offboarding_requests
   SET sla_warning_sent_at = NULL, sla_breach_sent_at = NULL
 WHERE id = '<request-id>' AND tenant_id = $1;

Then either wait for the 08:00 UTC tick or kick the worker on-demand (docker exec production-worker-1 node ./dist/scripts/run-sla-scanner.js if a manual driver exists; otherwise rely on the cron).

offboarding-final-day-countdown (§7A.4)

What it does. Daily 09:00 UTC. T-7 / T-3 / T-1 reminders to HR + the direct manager. Idempotent via three t_minus_*_sent_at columns.

Diagnostic SQL:

SELECT id, employee_id, final_day,
       t_minus_7_sent_at, t_minus_3_sent_at, t_minus_1_sent_at
  FROM offboarding_requests
 WHERE tenant_id = $1
   AND status = 'approved'
   AND final_day BETWEEN CURRENT_DATE AND CURRENT_DATE + 7
 ORDER BY final_day;

Recovery. Same pattern — null the appropriate t_minus_*_sent_at to re-fire that band.

offboarding-contract-end-scanner (§7D)

What it does. Daily 05:00 UTC. For employees with a contract_end_date within tenant_configs.contract_end_cr_lead_days (default 30), auto-creates a change_type = 'termination' CR. Idempotent via contract_end_cr_seeded_at on employees.

Diagnostic SQL:

SELECT e.id, e.first_name, e.last_name, e.contract_end_date,
       e.contract_end_cr_seeded_at,
       (SELECT cr.id FROM employee_change_requests cr
          WHERE cr.employee_id = e.id AND cr.change_type = 'termination'
            AND cr.proposed_changes ->> 'separation_reason_code' = 'CONTRACT_END'
          ORDER BY cr.created_at DESC LIMIT 1) AS latest_cr_id
  FROM employees e
 WHERE e.tenant_id = $1
   AND e.employment_type IN ('FIXED_TERM', 'CONTRACTOR_T4A', 'CONTRACTOR_1099')
   AND e.status = 'active'
   AND e.contract_end_date IS NOT NULL
   AND e.contract_end_date <= CURRENT_DATE +
       (SELECT contract_end_cr_lead_days FROM tenant_configs WHERE tenant_id = $1)::int
 ORDER BY e.contract_end_date;

Common false alarm. If the scanner shows fresh but no CR appears, check whether HR already created one manually — contract_end_cr_seeded_at is ALSO stamped when a manual CR exists for the same employee, to prevent double-seeding.

offboarding-access-revocation-execute (§7C.4)

What it does. Daily 02:00 UTC. Dispatches due SCHEDULED revocations by revocation_method:

• MANUAL — notifies it_manager to revoke + confirm in the UI; idempotent via due_notified_at. The row stays SCHEDULED until IT confirms (→ CONFIRMED).
• WEBHOOK — POSTs to the tenant's config_json.webhook_url through the SSRF-hardened sender. 2xx → status='EXECUTED', executed_at set. Any failure → status='FAILED', error_detail populated, it_manager notified to revoke manually (no auto-retry). Either terminal status removes the row from the daily scan.
• SCIM — still deferred (needs the spec-19 IdP integration); logged and due_notified_at set so it isn't re-logged.

Security — outbound webhook egress. The WEBHOOK path is the only place the worker makes an outbound HTTP call to a tenant-supplied URL. The in-process guard (worker/src/lib/safe-webhook.ts) blocks private / loopback / link-local / metadata targets and refuses non-https URLs, but the recommended belt-and-suspenders control is a network egress policy on the worker container that allowlists only known partner webhook hosts. Two env escape hatches exist for dev/test only and must NEVER be set in production: OFFBOARDING_WEBHOOK_ALLOW_INSECURE (permits http://) and OFFBOARDING_WEBHOOK_ALLOW_PRIVATE (skips the private-range blocklist). Verify they are unset:

docker exec production-worker-1 printenv | grep -i OFFBOARDING_WEBHOOK_ALLOW || echo "OK — no allow hatches set"

Investigate a failed webhook revocation:

SELECT id, system_name, status, error_detail, executed_at, scheduled_for
  FROM offboarding_access_revocations
 WHERE tenant_id = $1 AND status = 'FAILED'
 ORDER BY scheduled_for DESC;
-- Fix the tenant's webhook_url/headers in offboarding_revocation_configs,
-- then reset the row to re-attempt on the next 02:00 sweep:
--   UPDATE offboarding_access_revocations
--     SET status='SCHEDULED', error_detail=NULL, due_notified_at=NULL
--   WHERE id = $2 AND tenant_id = $1;

Diagnostic SQL:

SELECT oar.id, oar.system_name, oar.method, oar.scheduled_for,
       oar.status, oar.due_notified_at,
       obr.id AS request_id, e.work_email
  FROM offboarding_access_revocations oar
  JOIN offboarding_requests obr ON obr.id = oar.offboarding_request_id
  JOIN employees e ON e.id = obr.employee_id
 WHERE oar.tenant_id = $1
   AND oar.status = 'SCHEDULED'
   AND oar.scheduled_for <= CURRENT_DATE
 ORDER BY oar.scheduled_for;

offboarding-kt-escalation (§10.5)

What it does. Daily 10:00 UTC. KT-deliverable tasks with due_date approaching get an amber "closing soon" reminder to the assignee; tasks past due_date get a red "overdue" escalation to the assignee + HR. Idempotent via escalation_amber_sent_at / escalation_red_sent_at.

Diagnostic SQL:

SELECT t.id, t.title, t.due_date, t.status,
       t.escalation_amber_sent_at, t.escalation_red_sent_at,
       e.first_name AS assignee_first, e.last_name AS assignee_last
  FROM offboarding_tasks t
  LEFT JOIN employees e ON e.id = t.assigned_to_employee_id
 WHERE t.tenant_id = $1
   AND t.category = 'KT'
   AND t.status NOT IN ('COMPLETED','WAIVED','CANCELLED')
   AND t.due_date IS NOT NULL
   AND t.due_date <= CURRENT_DATE + 2
 ORDER BY t.due_date;

offboarding-orchestrate (event-driven, §7A.1)

What it does. Consumes queue:offboarding-orchestrate via Redis BRPOP. For each approved offboarding request, seeds tasks, exit interview, asset returns, and revocations per the per-tenant template catalog. Idempotent via task-count check (step 0). Skips the tenant entirely when tenant_configs.offboarding_enabled = false (step 0a, mig 214).

Stuck request recovery.

-- find approved requests with zero tasks (orchestration never ran or
-- was rolled back). Re-enqueue manually via redis-cli:
SELECT id, tenant_id, employee_id, status,
       (SELECT COUNT(*) FROM offboarding_tasks WHERE offboarding_request_id = obr.id) AS task_count
  FROM offboarding_requests obr
 WHERE status = 'approved'
   AND (SELECT COUNT(*) FROM offboarding_tasks WHERE offboarding_request_id = obr.id) = 0;

# re-enqueue (one-liner; tenant_id and request_id from the row above)
docker exec production-redis-1 redis-cli LPUSH queue:offboarding-orchestrate \
  '{"tenant_id":"<TENANT>","offboarding_request_id":"<REQUEST>"}'

---

The tenant kill switch (mig 214)

tenant_configs.offboarding_enabled BOOLEAN DEFAULT true — flip it false to disable the module for one tenant without a code deploy. Use case: a regression hits a single tenant, you need to pause new offboardings while the team investigates.

-- DISABLE for one tenant
UPDATE tenant_configs SET offboarding_enabled = false WHERE tenant_id = $1;
-- RE-ENABLE
UPDATE tenant_configs SET offboarding_enabled = true  WHERE tenant_id = $1;

What changes when disabled:

• All POST / PATCH / DELETE on /api/v1/offboarding/* and /api/v1/exit-interview/* return 503 RESOURCE_DISABLED with the message "Offboarding is disabled for this tenant. A tenant_admin can re-enable it from the offboarding settings page."
• The offboarding-orchestrate worker logs at WARN and skips the tenant when a queued job lands.
• Reads continue to work — the historical record stays accessible.
• The five scheduled scanners still run tenant-agnostically; they only act on existing rows so they can't make things worse. Their writes (notifications) still fire for previously-approved requests.

The kill switch is a tool of last resort. Default to fixing the bug.

---

Common incidents

"HR says SLA warnings aren't going out"

1. Check the panel: is offboarding-sla-scanner fresh? If critical, see "Worker hasn't run in 48h" below.
2. Run the §7A.5 diagnostic SQL. If rows come back with NULL sla_warning_sent_at but should have fired, the scanner found them but couldn't notify. Check notify-helpers logs in the worker (docker logs production-worker-1 | grep notify-helpers).
3. If notify-helpers succeeded but HR didn't receive, it's a notification preferences issue — check spec-18 delivery policies for the offboarding.sla_warning event_type.

"An offboarding request is stuck — no tasks generated"

Run the "Stuck request recovery" SQL above. If the kill switch is on, re-enabling will unstick it on the next orchestrate enqueue. Otherwise manually LPUSH the job back onto the queue.

"Worker hasn't run in 48h" (critical band)

1. docker ps — is production-worker-1 running?
2. docker logs production-worker-1 --tail 200 — recent errors?
3. BullMQ stuck? Inspect Redis: docker exec production-redis-1 redis-cli KEYS 'bull:offboarding-*' | head.
4. If the worker is healthy but worker_cron_completions hasn't updated, verify the cron registration in worker.ts — a typo or skipped registration is the usual culprit.

"Whole offboarding module needs to stop NOW for one tenant"

Flip the kill switch (see above). 503s are immediate; no app restart needed.

---

Related references

• Spec 43 Offboarding Module Spec — specs/43_Offboarding_Module_Spec.md
• Spec 27 Performance Hub operator runbook — docs/docs/performance/operator-runbook.md (the template for this doc)
• Deployment runbook — docs/docs/offboarding/deploy-runbook.md
• API reference (overview) — docs/docs/offboarding/overview.md