Skip to main content

Scheduling Operator Runbook

Operational reference for SREs and on-call engineers responding to alerts and incidents in the Scheduling subsystem (specs 23, 23A–23I).

This runbook covers what to monitor, what alerts mean, and how to recover from the most common failure modes — informed by the hardening pass (PRs #100–#109) that added the underlying instrumentation.

Audience. Anthropic-style SRE runbook — assumes shell access to the production server (serge@clear) and a working psql session against the production database. For end-user docs see Schedule, Operations, and Adherence.

Quick reference — service map​

ComponentContainerPort (prod / staging)Notes
APIproduction-api-1 / staging-api-13100 / 3200Schedule routes under /api/v1/workforce/*
Worker (BullMQ)production-worker-1 / staging-worker-1n/aOptimization runs, copy-from jobs, exception webhooks
Webproduction-web-1 / staging-web-13082 / 3086/schedule, /my-schedule, /wallboard
Postgresproduction-postgres-1internalRLS-scoped workforce_* tables
Redisproduction-redis-1internalBullMQ queues, idempotency cache, reaper lock bucket

Alerts and what to do​

scheduling.optimization_run.stuck​

Trigger. An optimization_runs row has been in running state for

10 minutes. A stuck run blocks subsequent runs for the same period.

Why this exists. The optimization run reaper (PR #104) reclaims stuck runs every 5 minutes. The alert fires only if the reaper itself is also failing, or if a run started <5 min ago is genuinely runaway.

Diagnose.

-- Find stuck runs
SELECT id, period_id, started_at, NOW() - started_at AS age, last_heartbeat_at
FROM workforce_optimization_runs
WHERE status = 'running'
  AND started_at < NOW() - INTERVAL '10 minutes'
ORDER BY started_at;
# Check reaper is heartbeating
docker logs production-api-1 --since 10m 2>&1 | grep -i "optimization.reaper"

Recover. Trigger the reaper manually:

# Inside the API container
docker exec -it production-api-1 node -e "
  require('./dist/jobs/optimization-reaper').runReaperOnce()
    .then(r => { console.log(JSON.stringify(r)); process.exit(0); })
    .catch(e => { console.error(e); process.exit(1); });
"

If runs remain stuck after manual reap, mark them failed directly — they will not auto-recover and a planner needs to retry from the UI:

UPDATE workforce_optimization_runs
SET status = 'failed',
    failure_reason = 'reaped: stuck >10min',
    completed_at = NOW()
WHERE id = '<run-id>' AND status = 'running';

Notify the planner (their tenant's wfm_planner role) so they can retry.

scheduling.swap_backlog.high​

Trigger. > 50 swap requests in submitted or accepted state for any single tenant for > 24 hours. Indicates planner inattention or a UI bug.

Diagnose.

SELECT t.slug, s.state, COUNT(*) AS count, MIN(s.created_at) AS oldest
FROM workforce_swap_requests s
JOIN tenants t ON t.id = s.tenant_id
WHERE s.state IN ('submitted', 'accepted')
  AND s.created_at < NOW() - INTERVAL '24 hours'
GROUP BY t.slug, s.state
ORDER BY count DESC;

Decide.

  • submitted backlog: planners aren't approving — page the tenant's CSM, not engineering.
  • accepted backlog: target agents accepted but planners aren't applying. Same — CSM escalation.
  • If swap submission is broken (no rows at all and tenant complains), see the swap submission failures section.

scheduling.acknowledgement_rate.low​

Trigger. A published period older than 48h has <80% acknowledgement rate across all assigned agents.

Diagnose.

WITH per AS (
  SELECT id, name, tenant_id, published_at
  FROM workforce_schedule_periods
  WHERE state = 'published' AND published_at < NOW() - INTERVAL '48 hours'
)
SELECT per.name,
       COUNT(DISTINCT s.user_id) AS assigned,
       COUNT(DISTINCT a.user_id) FILTER (WHERE a.acknowledged_at IS NOT NULL) AS acked,
       ROUND(100.0 * COUNT(DISTINCT a.user_id) FILTER (WHERE a.acknowledged_at IS NOT NULL)
             / NULLIF(COUNT(DISTINCT s.user_id), 0), 1) AS pct
FROM per
JOIN workforce_shifts s ON s.period_id = per.id
LEFT JOIN workforce_period_acknowledgements a
  ON a.period_id = per.id AND a.user_id = s.user_id
GROUP BY per.id, per.name
HAVING COUNT(DISTINCT s.user_id) > 0;

Decide. This is almost always a product/people problem, not infra. Page the CSM and verify with the planner that notifications fired (check notification_events for the period publish event). Only escalate to engineering if notifications are missing entirely — see publish notifications missing.

Common failure modes​

Failed publish​

A planner clicks Publish and the call returns 500 or hangs.

Most common cause. A shift in the period violates the labour compliance check (spec 23C) — the publish transaction rolls back. The UI should show the violation list; if it shows a generic error instead, check the API log:

docker logs production-api-1 --since 5m 2>&1 | grep -E "publish|compliance" | tail -50

Recover. No DB cleanup required — the transaction rolled back. The planner edits the offending shift(s) and republishes.

If the period is partially published (some assignments saved but state still draft), inspect:

SELECT id, name, state, published_at FROM workforce_schedule_periods WHERE id = '<period-id>';

If state = 'published' but published_at is NULL, that's a corrupted state — never observed in production but recoverable by:

UPDATE workforce_schedule_periods SET state = 'draft' WHERE id = '<period-id>' AND published_at IS NULL;

Swap submission fails​

Agent gets 422 on swap creation. Was historically caused by missing employee_id ↔ user_id mapping (see PR #100, the email-bridge audit).

Diagnose.

-- Confirm the agent exists in both tables and the email-bridge resolves
SELECT u.id AS user_id, u.email, e.id AS employee_id, e.work_email
FROM users u
LEFT JOIN employees e
  ON e.tenant_id = u.tenant_id AND e.work_email = u.email
WHERE u.id = '<reported-user-id>';

If employee_id is NULL → the user has no matching employee row. They were created via SSO without an HR record. Onboarding owns this fix.

Publish notifications missing​

Planner publishes; agents report no notification.

Diagnose.

SELECT * FROM notification_events
WHERE event_type = 'workforce.period.published'
  AND created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC LIMIT 20;

-- Then check delivery
SELECT * FROM notification_deliveries
WHERE event_id = '<event-id>'
ORDER BY created_at;

If no event row → publish handler didn't enqueue. Check API log around publish timestamp for errors. If event exists but no deliveries → BullMQ worker is down or queue is wedged:

docker logs production-worker-1 --since 30m 2>&1 | grep -iE "error|fatal" | tail -50

Wallboard shows stale data​

The wallboard (/wallboard) is WebSocket-driven from chat-gateway. If staffing tiles freeze:

docker logs production-chat-gateway-1 --since 5m 2>&1 | tail -100
docker exec production-redis-1 redis-cli LLEN wallboard:broadcast:queue

Restart chat-gateway if the worker is alive but the gateway is stuck:

docker compose -p production restart chat-gateway

Browser clients auto-reconnect within 5s.

Keyset pagination slow​

If a list endpoint (/swaps, /exceptions, /shifts) takes >1s on a large tenant, the composite index from migration 139 (PR #103) may be missing. Verify:

\d+ workforce_swap_requests
-- Look for: workforce_swap_requests_tenant_created_id_idx

If absent, re-apply migration 139 or check why it skipped during deploy. Missing indexes silently degrade to seq-scans.

Manual maintenance procedures​

Re-run the optimization reaper​

See scheduling.optimization_run.stuck above.

Force-archive an abandoned draft period​

UPDATE workforce_schedule_periods
SET state = 'archived', archived_at = NOW(), archived_reason = 'ops: abandoned draft'
WHERE id = '<period-id>' AND state = 'draft';

Audit row is not auto-emitted for direct DB updates — note the action in the incident log.

Cancel a pending swap on behalf of a user​

UPDATE workforce_swap_requests
SET state = 'cancelled', cancelled_at = NOW(), cancelled_reason = 'ops: <reason>'
WHERE id = '<swap-id>' AND state IN ('draft', 'submitted', 'accepted');

Notify both the requester and the target agent.

Replay a stuck BullMQ job​

docker exec -it production-redis-1 redis-cli
> KEYS bull:*:failed
> LRANGE bull:workforce:failed 0 10

Use the BullMQ web UI (if installed) or:

docker exec production-worker-1 node -e "
  const { Queue } = require('bullmq');
  const q = new Queue('workforce', { connection: { host: 'redis', port: 6379 } });
  q.getFailed(0, 10).then(jobs => jobs.forEach(j => j.retry())).then(() => process.exit(0));
"

Escalation​

SymptomFirst responderEscalate to
All scheduling 5xxOn-call SREBackend team lead after 15min
Single-tenant data corruptionOn-call SREBackend + CSM together
Reaper itself failingOn-call SREBackend team lead immediately
Compliance-engine bug (false positive blocks publish)Backend team leadSpec owner (WFM team)
Wallboard downOn-call SRERealtime team lead
Self-schedule cache empty / agents see no available slotsOn-call SREBackend team lead — see §Self-scheduling below

Self-scheduling (spec 23K)​

The self-scheduling subsystem adds three worker handlers and 19 new API endpoints. SREs investigating issues here:

selfschedule.availability_recompute.empty_cache​

Trigger. Agents report /my-schedule/self-schedule shows "No pickable intervals" for any LOB despite scheduled coverage existing.

Diagnose.

-- Are there any cache rows at all?
SELECT COUNT(*), MIN(computed_at), MAX(computed_at)
FROM self_schedule_available_blocks;

-- Are any LOBs participating? (LOCKED tier doesn't recompute)
SELECT client_lob_id, guardrail_tier
FROM self_schedule_lob_configs
WHERE guardrail_tier IN ('OPEN', 'RESTRICTED');
# Worker logs for the recompute job (every 30 min)
docker logs production-worker-1 --since 1h 2>&1 | grep self-schedule-availability-recompute

Recover. Trigger manually inside the worker container:

docker exec -it production-worker-1 node -e "
  require('./dist/handlers/self-schedule-availability-recompute')
    .recomputeAvailableBlocks(require('./dist/db').db, console)
    .then(r => console.log(JSON.stringify(r))).catch(console.error);
"

If lobsScanned = 0: tenant has no participating LOBs (planner needs to set tier in /settings/self-schedule). If upsertedCount = 0 but LOBs are participating: no shifts overlap the next 14 days for those LOBs (planner needs to publish a period covering future dates).

selfschedule.sla_reaper.backlog​

Trigger. PENDING_APPROVAL self-schedule picks accumulating past their LOB's planner_review_sla_hours.

Diagnose.

SELECT t.slug, COUNT(*) AS backlog,
       MIN(NOW() - pk.submitted_at) AS oldest
FROM self_schedule_picks pk
JOIN tenants t ON t.id = pk.tenant_id
WHERE pk.state = 'PENDING_APPROVAL'
GROUP BY t.slug
ORDER BY backlog DESC;

If oldest is past SLA, reaper should have caught it. Check it's running:

docker logs production-worker-1 --since 30m 2>&1 | grep self-schedule-sla-reaper

The reaper auto-rejects with AUTO_REJECTED_SLA; planner inattention beyond that is a CSM escalation, not engineering.

selfschedule.trade_expiry_reaper.stuck_open​

OPEN trades past their expires_at should auto-transition to EXPIRED within 5 minutes. If they don't:

SELECT id, expires_at, NOW() - expires_at AS overdue
FROM self_schedule_trades
WHERE state = 'OPEN' AND expires_at < NOW() - INTERVAL '10 minutes'
LIMIT 10;

Same recovery pattern as the other reapers — check worker logs, restart if stuck.

Common self-schedule failure modes​

Pick fails with SELF_SCHEDULE_COMPLIANCE_BLOCK 409. The agent's proposed shift violates a labour rule (overtime, min rest, weekly cap). Check the rule code in error.details.rule_code. Not an operations issue — the agent is being protected from scheduling themselves into a violation. Surface to planner if they want a case-by-case override (planner approve will RE-RUN the same compliance check at approval time).

Pick fails with SELF_SCHEDULE_PATTERN_MISALIGNED 400. Since migration 417 (spec 23 §5.4D) a pattern row is a shift rule set: a start window and a length range. The proposed shift is not one any rule set available to the LOB can produce: its local start is outside every window or off its increment, its wall-clock length is outside every range or off its increment, or its local start date is on a weekday or outside the effective dates no rule set offers. "Available" is the LOB's default bag when it has one, otherwise every active rule set that applies to the LOB. A LOB whose default bag produces nothing refuses every pick. Check:

-- Does the LOB have a default bag?
SELECT d.bag_id, b.name FROM workforce_lob_shift_bag_defaults d
  JOIN workforce_shift_bags b ON b.id = d.bag_id
 WHERE d.tenant_id = '<tenant>' AND d.client_lob_id = '<lob>';

-- The active rule sets that apply to the LOB (the fallback when it has no default bag).
SELECT p.name, p.days_of_week_mask, p.start_time, p.start_time_latest, p.start_increment_minutes,
       p.length_min_minutes, p.length_max_minutes, p.length_increment_minutes, p.effective_from, p.effective_to
  FROM workforce_shift_patterns p
  LEFT JOIN client_account_lobs l ON l.id = '<lob>' AND l.tenant_id = p.tenant_id
 WHERE p.tenant_id = '<tenant>' AND p.active = true
   AND (p.client_lob_id = '<lob>'
        OR (p.client_lob_id IS NULL AND (p.client_account_id IS NULL OR p.client_account_id = l.client_account_id)));

Before deploying migration 417. The migration refuses to convert a pattern it cannot express as a rule set without changing what generation produces, and raises with the counts. Run this read-only check first; every count must be 0:

SELECT
  (SELECT count(*) FROM workforce_shift_patterns p JOIN client_account_lobs l ON l.id = p.client_lob_id
    WHERE l.tenant_id <> p.tenant_id) AS lob_of_another_tenant,
  (SELECT count(*) FROM workforce_shift_patterns
    WHERE EXTRACT(SECOND FROM start_time) <> 0 OR EXTRACT(SECOND FROM end_time) <> 0) AS seconds_in_a_time,
  (SELECT count(*) FROM workforce_shift_patterns WHERE jsonb_typeof(break_config_json) <> 'array') AS breaks_not_an_array;

Then look at the break entries by eye (SELECT id, name, break_config_json FROM workforce_shift_patterns WHERE break_config_json <> '[]'::jsonb;): each entry must be an object whose offset_minutes and duration_minutes are whole numbers, with offset plus duration at most 1440. Each entry with minutes becomes an unpaid BREAK activity rule marked converted_from_break_config, and one workforce.shift_pattern.breaks_converted audit row per pattern records the original JSON.

The old pattern route accepted breaks that do not fit their own pattern. They convert as they are and generation reads them as it did; the rule set editor does not hold a converted rule to its fit checks until someone edits it. To know how many there are (informational, nothing to fix before deploying):

WITH breaks AS (
  SELECT p.id, p.name, p.start_time, p.end_time, p.break_config_json,
         COALESCE(NULLIF(((EXTRACT(EPOCH FROM (p.end_time - p.start_time)) / 60)::int + 1440) % 1440, 0), 1440) AS length,
         COALESCE(b.item->>'offset_minutes', '0')::int AS offset_minutes,
         (b.item->>'duration_minutes')::int AS duration_minutes, b.ord
    FROM workforce_shift_patterns p
    CROSS JOIN LATERAL jsonb_array_elements(
      CASE WHEN jsonb_typeof(p.break_config_json) = 'array' THEN p.break_config_json ELSE '[]'::jsonb END) WITH ORDINALITY AS b(item, ord)
   WHERE jsonb_typeof(b.item) = 'object'
     AND COALESCE(b.item->>'duration_minutes', '') ~ '^[0-9]{1,4}$'
     AND COALESCE(b.item->>'offset_minutes', '0') ~ '^[0-9]{1,4}$'
     AND (b.item->>'duration_minutes')::int > 0
)
SELECT DISTINCT a.id, a.name, a.start_time, a.end_time, a.break_config_json
  FROM breaks a
 WHERE a.offset_minutes + a.duration_minutes > a.length
    OR EXISTS (SELECT 1 FROM breaks o
                WHERE o.id = a.id AND o.ord <> a.ord
                  AND o.offset_minutes < a.offset_minutes + a.duration_minutes
                  AND a.offset_minutes < o.offset_minutes + o.duration_minutes);

The first condition finds a break that ends after its pattern does; the second, two breaks of one pattern that overlap.

Pick fails with SELF_SCHEDULE_AMBIGUOUS_PERIOD 409. Multiple published periods overlap the target_date for that LOB. Either archive the stale period or add a schedule_period_id to the pick submission (UI doesn't expose this today — operations rare).

Multi-site, business continuity, and shift patterns (specs 23E / 23H / 23I)​

Three operator surfaces shipped on 2026-05-03 (PRs #151, #152, #153) that filled in UI for previously-API-only modules:

  • /workforce/shift-patterns: shift pattern library (since migration 417 a redirect to Schedule > Shift rule sets)
  • /workforce/incidents — incident command center
  • /workforce/borrow-requests — cross-site borrow with dual-manager approval

Backend was live well before this; the operational signals below are what you need to know now that operators have a real UI.

Shift rule sets (formerly /workforce/shift-patterns): common questions​

Since migration 417 (spec 23 §5.4D) a shift pattern is a shift rule set, edited on Schedule > Shift rule sets (tab id patterns). /workforce/shift-patterns only redirects there, and the old pattern writes answer 410 SHIFT_PATTERNS_API_RETIRED.

Q: An operator says "the optimizer rejected my schedule because no shift patterns matched." The run is refused with "No active shift rule sets found" (the optimizer's own error reads the same) only when the tenant has no active rule set at all. For a LOB, generation and self-schedule use the LOB's default bag's active rule sets that apply to it, or, with no default bag, every active rule set that applies to it. A LOB with a default bag whose rule sets produce nothing gets no shifts and refuses every self-schedule pick. The two SQL checks under SELF_SCHEDULE_PATTERN_MISALIGNED above show which rule sets a LOB uses.

Q: A planner copied a bag to another client and that client's schedules changed. Copying a bag copies the source client's rule sets into the target client for the whole client (no LOB), inactive. Nothing changes for the target until a planner reactivates a copy; from then on every LOB of the target client with no default bag uses it. Set a default bag on each LOB, or deactivate the copies it should not use. The audit event workforce.shift_bag.copy on the new bag lists every source and copy rule set id, and workforce.shift_pattern.reactivate shows who turned a copy on.

Q: A LOB's self-schedule refuses every pick after a default bag was set. No active rule set in that bag applies to the LOB (a bag of the client can hold only a sibling LOB's rule sets). The default screen marks such a bag and the save warns (DEFAULT_BAG_GIVES_LOB_NO_RULE_SETS); clear or change the default.

Q: A planner gets 403 SHIFT_BAG_USED_OUTSIDE_SCOPE or RULE_SET_USED_OUTSIDE_SCOPE. The bag, or a bag holding the rule set, is for all clients and is the default of a LOB of a client the planner does not manage. Someone who manages every client makes the change.

Q: Overnight shifts. A rule set's start window may not cross midnight, but a shift may: a 22:00 start with an 8 hour length ends at 06:00 the next day, read as a wall clock. A night whose starts fall on both sides of 00:00 is two rule sets.

Q: Bitmask convention. Mon=bit0 (=1), Tue=bit1 (=2), ..., Sun=bit6 (=64). Mon-Fri = 1+2+4+8+16 = 31. The 23K pick-alignment SQL at self-schedule.ts:706 uses the same convention; do NOT use JS getDay() (Sun=0, Mon=1) or PG EXTRACT(DOW) (Sun=0, Sat=6) without translating.

/workforce/incidents — common questions​

Q: How are affected shifts identified? On POST /incidents, the declare handler (routes/workforce/incidents.ts:79) joins workforce_shift_assignments → users → employees → employee_site_assignments filtered by is_primary = true AND site_id = <incident.site_id> for the look-ahead window (default 24h, configurable per incident). One incident_shift_mappings row is inserted per affected shift, all with action = 'UNCHANGED'. Reallocation actions are deferred to the planner workflow on /schedule.

Q: The "1 affected shifts" / "1 affected shift" plural quirk. Fixed in PR #152 — the API was returning COUNT(*) as a bigint, the JS pg driver delivered it as a string, and the UI's count === 1 ternary always rendered plural. SQL now casts to ::int. If you see plural for count=1 in production, you're on a stale image.

Q: How does Cancel differ from Resolve? Cancel = "this declaration was a mistake or no longer applies." Resolve = "the incident is genuinely over." Both terminate the lifecycle; Cancel does NOT prompt for a resolution note. Audit events differ accordingly.

/workforce/borrow-requests — common questions​

Q: A planner says "I approved the borrow but the request is still pending." The dual-manager gate (spec 23H §6.5) requires BOTH source-side and destination-side approval. The list view shows status PENDING until both have decided.

-- See decision state for a request
SELECT id, status, source_manager_decision, destination_manager_decision,
       agent_count, approved_agent_count
FROM cross_site_borrow_requests
WHERE id = '<req-id>';

Q: Status transitions.

  • Source REJECTS → status REJECTED, destination side never decides
  • Source APPROVES with approved_count == agent_count → destination ACCEPT → APPROVED
  • Source APPROVES with approved_count < agent_count → destination ACCEPT → PARTIALLY_APPROVED
  • Source APPROVES, destination REJECTS → REJECTED
  • Anyone CANCELs while PENDING → CANCELLED

Q: Why same-site borrows are blocked. Server enforces requesting_site_id != source_site_id (route line 57). The UI also gates the Submit button when both selectors point at the same site. Adding a borrow against your own site doesn't make operational sense and would loop the dual-manager check.

Q: Escalation timer. escalation_at is set to created_at + 24h on creation. Today the field exists for future use (intended to drive "borrow request unanswered for 24h" alerts) but no worker consumes it yet — flagged for a follow-up.

Change log​

  • 2026-05-03 v1.2 — added ## Multi-site, business continuity, and shift patterns (specs 23E / 23H / 23I) section covering the three operator surfaces shipped in PRs #151 / #152 / #153.
  • 2026-05-02 v1.1 — added ## Self-scheduling (spec 23K) section covering the three new worker handlers and common self-schedule failure modes.
  • 2026-05-02 v1.0 — initial runbook, derived from the hardening pass (PRs #100–#109).
Last updated on