Scheduling Operator Runbook

Operational reference for SREs and on-call engineers responding to alerts and incidents in the Scheduling subsystem (specs 23, 23A–23I).

This runbook covers what to monitor, what alerts mean, and how to recover from the most common failure modes — informed by the hardening pass (PRs #100–#109) that added the underlying instrumentation.

Audience. Anthropic-style SRE runbook — assumes shell access to the production server (serge@clear) and a working psql session against the production database. For end-user docs see Schedule, Operations, and Adherence.

Quick reference — service map

Component	Container	Port (prod / staging)	Notes
API	`production-api-1` / `staging-api-1`	3100 / 3200	Schedule routes under `/api/v1/workforce/*`
Worker (BullMQ)	`production-worker-1` / `staging-worker-1`	n/a	Optimization runs, copy-from jobs, exception webhooks
Web	`production-web-1` / `staging-web-1`	3082 / 3086	`/schedule`, `/my-schedule`, `/wallboard`
Postgres	`production-postgres-1`	internal	RLS-scoped `workforce_*` tables
Redis	`production-redis-1`	internal	BullMQ queues, idempotency cache, reaper lock bucket

Alerts and what to do

`scheduling.optimization_run.stuck`

Trigger. An optimization_runs row has been in running state for

10 minutes. A stuck run blocks subsequent runs for the same period.

Why this exists. The optimization run reaper (PR #104) reclaims stuck runs every 5 minutes. The alert fires only if the reaper itself is also failing, or if a run started <5 min ago is genuinely runaway.

Diagnose.

-- Find stuck runs
SELECT id, period_id, started_at, NOW() - started_at AS age, last_heartbeat_at
FROM workforce_optimization_runs
WHERE status = 'running'
  AND started_at < NOW() - INTERVAL '10 minutes'
ORDER BY started_at;

# Check reaper is heartbeating
docker logs production-api-1 --since 10m 2>&1 | grep -i "optimization.reaper"

Recover. Trigger the reaper manually:

# Inside the API container
docker exec -it production-api-1 node -e "
  require('./dist/jobs/optimization-reaper').runReaperOnce()
    .then(r => { console.log(JSON.stringify(r)); process.exit(0); })
    .catch(e => { console.error(e); process.exit(1); });
"

If runs remain stuck after manual reap, mark them failed directly — they will not auto-recover and a planner needs to retry from the UI:

UPDATE workforce_optimization_runs
SET status = 'failed',
    failure_reason = 'reaped: stuck >10min',
    completed_at = NOW()
WHERE id = '<run-id>' AND status = 'running';

Notify the planner (their tenant's wfm_planner role) so they can retry.

`scheduling.swap_backlog.high`

Trigger. > 50 swap requests in submitted or accepted state for any single tenant for > 24 hours. Indicates planner inattention or a UI bug.

Diagnose.

SELECT t.slug, s.state, COUNT(*) AS count, MIN(s.created_at) AS oldest
FROM workforce_swap_requests s
JOIN tenants t ON t.id = s.tenant_id
WHERE s.state IN ('submitted', 'accepted')
  AND s.created_at < NOW() - INTERVAL '24 hours'
GROUP BY t.slug, s.state
ORDER BY count DESC;

Decide.

submitted backlog: planners aren't approving — page the tenant's CSM, not engineering.
accepted backlog: target agents accepted but planners aren't applying. Same — CSM escalation.
If swap submission is broken (no rows at all and tenant complains), see the swap submission failures section.

`scheduling.acknowledgement_rate.low`

Trigger. A published period older than 48h has <80% acknowledgement rate across all assigned agents.

Diagnose.

WITH per AS (
  SELECT id, name, tenant_id, published_at
  FROM workforce_schedule_periods
  WHERE state = 'published' AND published_at < NOW() - INTERVAL '48 hours'
)
SELECT per.name,
       COUNT(DISTINCT s.user_id) AS assigned,
       COUNT(DISTINCT a.user_id) FILTER (WHERE a.acknowledged_at IS NOT NULL) AS acked,
       ROUND(100.0 * COUNT(DISTINCT a.user_id) FILTER (WHERE a.acknowledged_at IS NOT NULL)
             / NULLIF(COUNT(DISTINCT s.user_id), 0), 1) AS pct
FROM per
JOIN workforce_shifts s ON s.period_id = per.id
LEFT JOIN workforce_period_acknowledgements a
  ON a.period_id = per.id AND a.user_id = s.user_id
GROUP BY per.id, per.name
HAVING COUNT(DISTINCT s.user_id) > 0;

Decide. This is almost always a product/people problem, not infra. Page the CSM and verify with the planner that notifications fired (check notification_events for the period publish event). Only escalate to engineering if notifications are missing entirely — see publish notifications missing.

Common failure modes

Failed publish

A planner clicks Publish and the call returns 500 or hangs.

Most common cause. A shift in the period violates the labour compliance check (spec 23C) — the publish transaction rolls back. The UI should show the violation list; if it shows a generic error instead, check the API log:

docker logs production-api-1 --since 5m 2>&1 | grep -E "publish|compliance" | tail -50

Recover. No DB cleanup required — the transaction rolled back. The planner edits the offending shift(s) and republishes.

If the period is partially published (some assignments saved but state still draft), inspect:

SELECT id, name, state, published_at FROM workforce_schedule_periods WHERE id = '<period-id>';

If state = 'published' but published_at is NULL, that's a corrupted state — never observed in production but recoverable by:

UPDATE workforce_schedule_periods SET state = 'draft' WHERE id = '<period-id>' AND published_at IS NULL;

Swap submission fails

Agent gets 422 on swap creation. Was historically caused by missing employee_id ↔ user_id mapping (see PR #100, the email-bridge audit).

Diagnose.

-- Confirm the agent exists in both tables and the email-bridge resolves
SELECT u.id AS user_id, u.email, e.id AS employee_id, e.work_email
FROM users u
LEFT JOIN employees e
  ON e.tenant_id = u.tenant_id AND e.work_email = u.email
WHERE u.id = '<reported-user-id>';

If employee_id is NULL → the user has no matching employee row. They were created via SSO without an HR record. Onboarding owns this fix.

Publish notifications missing

Planner publishes; agents report no notification.

Diagnose.

SELECT * FROM notification_events
WHERE event_type = 'workforce.period.published'
  AND created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC LIMIT 20;

-- Then check delivery
SELECT * FROM notification_deliveries
WHERE event_id = '<event-id>'
ORDER BY created_at;

If no event row → publish handler didn't enqueue. Check API log around publish timestamp for errors. If event exists but no deliveries → BullMQ worker is down or queue is wedged:

docker logs production-worker-1 --since 30m 2>&1 | grep -iE "error|fatal" | tail -50

Wallboard shows stale data

The wallboard (/wallboard) is WebSocket-driven from chat-gateway. If staffing tiles freeze:

docker logs production-chat-gateway-1 --since 5m 2>&1 | tail -100
docker exec production-redis-1 redis-cli LLEN wallboard:broadcast:queue

Restart chat-gateway if the worker is alive but the gateway is stuck:

docker compose -p production restart chat-gateway

Browser clients auto-reconnect within 5s.

Keyset pagination slow

If a list endpoint (/swaps, /exceptions, /shifts) takes >1s on a large tenant, the composite index from migration 139 (PR #103) may be missing. Verify:

\d+ workforce_swap_requests
-- Look for: workforce_swap_requests_tenant_created_id_idx

If absent, re-apply migration 139 or check why it skipped during deploy. Missing indexes silently degrade to seq-scans.

Manual maintenance procedures

Re-run the optimization reaper

See scheduling.optimization_run.stuck above.

Force-archive an abandoned draft period

UPDATE workforce_schedule_periods
SET state = 'archived', archived_at = NOW(), archived_reason = 'ops: abandoned draft'
WHERE id = '<period-id>' AND state = 'draft';

Audit row is not auto-emitted for direct DB updates — note the action in the incident log.

Cancel a pending swap on behalf of a user

UPDATE workforce_swap_requests
SET state = 'cancelled', cancelled_at = NOW(), cancelled_reason = 'ops: <reason>'
WHERE id = '<swap-id>' AND state IN ('draft', 'submitted', 'accepted');

Notify both the requester and the target agent.

Replay a stuck BullMQ job

docker exec -it production-redis-1 redis-cli
> KEYS bull:*:failed
> LRANGE bull:workforce:failed 0 10

Use the BullMQ web UI (if installed) or:

docker exec production-worker-1 node -e "
  const { Queue } = require('bullmq');
  const q = new Queue('workforce', { connection: { host: 'redis', port: 6379 } });
  q.getFailed(0, 10).then(jobs => jobs.forEach(j => j.retry())).then(() => process.exit(0));
"

Escalation

Symptom	First responder	Escalate to
All scheduling 5xx	On-call SRE	Backend team lead after 15min
Single-tenant data corruption	On-call SRE	Backend + CSM together
Reaper itself failing	On-call SRE	Backend team lead immediately
Compliance-engine bug (false positive blocks publish)	Backend team lead	Spec owner (WFM team)
Wallboard down	On-call SRE	Realtime team lead
Self-schedule cache empty / agents see no available slots	On-call SRE	Backend team lead — see §Self-scheduling below

Self-scheduling (spec 23K)

The self-scheduling subsystem adds three worker handlers and 19 new API endpoints. SREs investigating issues here:

`selfschedule.availability_recompute.empty_cache`

Trigger. Agents report /my-schedule/self-schedule shows "No pickable intervals" for any LOB despite scheduled coverage existing.

Diagnose.

-- Are there any cache rows at all?
SELECT COUNT(*), MIN(computed_at), MAX(computed_at)
FROM self_schedule_available_blocks;

-- Are any LOBs participating? (LOCKED tier doesn't recompute)
SELECT client_lob_id, guardrail_tier
FROM self_schedule_lob_configs
WHERE guardrail_tier IN ('OPEN', 'RESTRICTED');

# Worker logs for the recompute job (every 30 min)
docker logs production-worker-1 --since 1h 2>&1 | grep self-schedule-availability-recompute

Recover. Trigger manually inside the worker container:

docker exec -it production-worker-1 node -e "
  require('./dist/handlers/self-schedule-availability-recompute')
    .recomputeAvailableBlocks(require('./dist/db').db, console)
    .then(r => console.log(JSON.stringify(r))).catch(console.error);
"

If lobsScanned = 0: tenant has no participating LOBs (planner needs to set tier in /settings/self-schedule). If upsertedCount = 0 but LOBs are participating: no shifts overlap the next 14 days for those LOBs (planner needs to publish a period covering future dates).

`selfschedule.sla_reaper.backlog`

Trigger. PENDING_APPROVAL self-schedule picks accumulating past their LOB's planner_review_sla_hours.

Diagnose.

SELECT t.slug, COUNT(*) AS backlog,
       MIN(NOW() - pk.submitted_at) AS oldest
FROM self_schedule_picks pk
JOIN tenants t ON t.id = pk.tenant_id
WHERE pk.state = 'PENDING_APPROVAL'
GROUP BY t.slug
ORDER BY backlog DESC;

If oldest is past SLA, reaper should have caught it. Check it's running:

docker logs production-worker-1 --since 30m 2>&1 | grep self-schedule-sla-reaper

The reaper auto-rejects with AUTO_REJECTED_SLA; planner inattention beyond that is a CSM escalation, not engineering.

`selfschedule.trade_expiry_reaper.stuck_open`

OPEN trades past their expires_at should auto-transition to EXPIRED within 5 minutes. If they don't:

SELECT id, expires_at, NOW() - expires_at AS overdue
FROM self_schedule_trades
WHERE state = 'OPEN' AND expires_at < NOW() - INTERVAL '10 minutes'
LIMIT 10;

Same recovery pattern as the other reapers — check worker logs, restart if stuck.

Common self-schedule failure modes

Pick fails with SELF_SCHEDULE_COMPLIANCE_BLOCK 409. The agent's proposed shift violates a labour rule (overtime, min rest, weekly cap). Check the rule code in error.details.rule_code. Not an operations issue — the agent is being protected from scheduling themselves into a violation. Surface to planner if they want a case-by-case override (planner approve will RE-RUN the same compliance check at approval time).

Pick fails with SELF_SCHEDULE_PATTERN_MISALIGNED 400. The proposed shift doesn't match any active workforce_shift_patterns row for the LOB on the target day-of-week. Either an agent picked a non-standard time, or the pattern library is incomplete. Check:

SELECT name, days_of_week_mask, start_time, end_time
FROM workforce_shift_patterns
WHERE tenant_id = '<tenant>' AND active = true
  AND (client_lob_id = '<lob>' OR client_lob_id IS NULL);

Pick fails with SELF_SCHEDULE_AMBIGUOUS_PERIOD 409. Multiple published periods overlap the target_date for that LOB. Either archive the stale period or add a schedule_period_id to the pick submission (UI doesn't expose this today — operations rare).

Multi-site, business continuity, and shift patterns (specs 23E / 23H / 23I)

Three operator surfaces shipped on 2026-05-03 (PRs #151, #152, #153) that filled in UI for previously-API-only modules:

/workforce/shift-patterns — shift pattern library
/workforce/incidents — incident command center
/workforce/borrow-requests — cross-site borrow with dual-manager approval

Backend was live well before this; the operational signals below are what you need to know now that operators have a real UI.

`/workforce/shift-patterns` — common questions

Q: An operator says "the optimizer rejected my schedule because no shift patterns matched." Spec 23E §4.1 — the optimizer (and 23K self-schedule pick alignment) consults workforce_shift_patterns for each LOB. If a tenant has zero active patterns for the LOB, the optimizer falls back to "any reasonable shift" and the spec 23K pick flow soft-allows. If a tenant has SOME patterns but the proposed shape matches none, the call rejects.

-- Are there active patterns for this LOB?
SELECT id, name, days_of_week_mask, start_time, end_time, active
FROM workforce_shift_patterns
WHERE tenant_id = '<tid>' AND (client_lob_id = '<lob>' OR client_lob_id IS NULL) AND active;

Q: Why is overnight (22:00 → 06:00) blocked? v1 doesn't support cross-midnight patterns — the optimizer math interprets end_time < start_time as a negative-duration shift. Operators wanting overnight coverage today split into two patterns (22:00–24:00 + 00:00–06:00). Tracked as a planned enhancement.

Q: Bitmask convention. Mon=bit0 (=1), Tue=bit1 (=2), ..., Sun=bit6 (=64). Mon-Fri = 1+2+4+8+16 = 31. The 23K pick-alignment SQL at self-schedule.ts:706 uses the same convention; do NOT use JS getDay() (Sun=0, Mon=1) or PG EXTRACT(DOW) (Sun=0, Sat=6) without translating.

`/workforce/incidents` — common questions

Q: How are affected shifts identified? On POST /incidents, the declare handler (routes/workforce/incidents.ts:79) joins workforce_shift_assignments → users → employees → employee_site_assignments filtered by is_primary = true AND site_id = <incident.site_id> for the look-ahead window (default 24h, configurable per incident). One incident_shift_mappings row is inserted per affected shift, all with action = 'UNCHANGED'. Reallocation actions are deferred to the planner workflow on /schedule.

Q: The "1 affected shifts" / "1 affected shift" plural quirk. Fixed in PR #152 — the API was returning COUNT(*) as a bigint, the JS pg driver delivered it as a string, and the UI's count === 1 ternary always rendered plural. SQL now casts to ::int. If you see plural for count=1 in production, you're on a stale image.

Q: How does Cancel differ from Resolve? Cancel = "this declaration was a mistake or no longer applies." Resolve = "the incident is genuinely over." Both terminate the lifecycle; Cancel does NOT prompt for a resolution note. Audit events differ accordingly.

`/workforce/borrow-requests` — common questions

Q: A planner says "I approved the borrow but the request is still pending." The dual-manager gate (spec 23H §6.5) requires BOTH source-side and destination-side approval. The list view shows status PENDING until both have decided.

-- See decision state for a request
SELECT id, status, source_manager_decision, destination_manager_decision,
       agent_count, approved_agent_count
FROM cross_site_borrow_requests
WHERE id = '<req-id>';

Q: Status transitions.

Source REJECTS → status REJECTED, destination side never decides
Source APPROVES with approved_count == agent_count → destination ACCEPT → APPROVED
Source APPROVES with approved_count < agent_count → destination ACCEPT → PARTIALLY_APPROVED
Source APPROVES, destination REJECTS → REJECTED
Anyone CANCELs while PENDING → CANCELLED

Q: Why same-site borrows are blocked. Server enforces requesting_site_id != source_site_id (route line 57). The UI also gates the Submit button when both selectors point at the same site. Adding a borrow against your own site doesn't make operational sense and would loop the dual-manager check.

Q: Escalation timer. escalation_at is set to created_at + 24h on creation. Today the field exists for future use (intended to drive "borrow request unanswered for 24h" alerts) but no worker consumes it yet — flagged for a follow-up.

Change log

2026-05-03 v1.2 — added ## Multi-site, business continuity, and shift patterns (specs 23E / 23H / 23I) section covering the three operator surfaces shipped in PRs #151 / #152 / #153.
2026-05-02 v1.1 — added ## Self-scheduling (spec 23K) section covering the three new worker handlers and common self-schedule failure modes.
2026-05-02 v1.0 — initial runbook, derived from the hardening pass (PRs #100–#109).

Quick reference — service map​

Alerts and what to do​

scheduling.optimization_run.stuck​

scheduling.swap_backlog.high​

scheduling.acknowledgement_rate.low​

Common failure modes​

Failed publish​

Swap submission fails​

Publish notifications missing​

Wallboard shows stale data​

Keyset pagination slow​

Manual maintenance procedures​

Re-run the optimization reaper​

Force-archive an abandoned draft period​

Cancel a pending swap on behalf of a user​

Replay a stuck BullMQ job​

Escalation​

Self-scheduling (spec 23K)​

selfschedule.availability_recompute.empty_cache​

selfschedule.sla_reaper.backlog​

selfschedule.trade_expiry_reaper.stuck_open​

Common self-schedule failure modes​

Multi-site, business continuity, and shift patterns (specs 23E / 23H / 23I)​

/workforce/shift-patterns — common questions​

/workforce/incidents — common questions​

/workforce/borrow-requests — common questions​

Change log​

Quick reference — service map

Alerts and what to do

`scheduling.optimization_run.stuck`

`scheduling.swap_backlog.high`

`scheduling.acknowledgement_rate.low`

Common failure modes

Failed publish

Swap submission fails

Publish notifications missing

Wallboard shows stale data

Keyset pagination slow

Manual maintenance procedures

Re-run the optimization reaper

Force-archive an abandoned draft period

Cancel a pending swap on behalf of a user

Replay a stuck BullMQ job

Escalation

Self-scheduling (spec 23K)

`selfschedule.availability_recompute.empty_cache`

`selfschedule.sla_reaper.backlog`

`selfschedule.trade_expiry_reaper.stuck_open`

Common self-schedule failure modes

Multi-site, business continuity, and shift patterns (specs 23E / 23H / 23I)

`/workforce/shift-patterns` — common questions

`/workforce/incidents` — common questions

`/workforce/borrow-requests` — common questions

Change log