Pulse Surveys Operator Runbook

This runbook is paired with the PipelineStatusPanel on /settings/pulse (HR-admin view) and the GET /api/v1/pulse/operational-status endpoint behind it.

Spec reference: 27B Pulse Surveys §1–§9 (see specs/27B_Pulse_Surveys_Spec.md in the repo). Production-readiness loop is complete: B1 + H1 + H3 + M1 + M2 + M3 + M4 + L1 + L2 all shipped. H2 (backfill script) is N/A for pulse — see "Why pulse has no backfill script" below.

The pulse pipeline is a fan of three workers, mostly independent. Failures don't cascade.

Quick reference — service map

Component	Where	Owns
`pulse-survey-scheduler`	worker · `handlers/pulse-survey-scheduler.ts`	Opens new survey periods per LOB on cadence, fans out token-auth invites to active employees (anti-joining the opt-out table).
`pulse-survey-scoring`	worker · `handlers/pulse-survey-scoring.ts`	Closes any OPEN period whose `period_end` has passed, computes eNPS, applies min-group-size suppression, UPSERTs LOB + TENANT snapshots.
`pulse-survey-alerts`	worker · `handlers/pulse-survey-alerts.ts`	Evaluates score_drop / floor_consecutive / participation_low against per-LOB config, fans out aggregated notifications to hr_admin + LOB managers.
Settings UI	web · `/settings/pulse` (HR-admin)	Renders the panel + cadence/threshold form.
Results dashboard	web · `/pulse` (HR + LOB managers)	Period table + scope cards + verbatim keywords.
Operational endpoint	api · `routes/pulse/operational-status.ts`	Reads `worker_cron_completions` and pulse tables for the panel.

Cron-chain cadences

Queue	Cron	Cadence	Fresh band	Stale band
`pulse-survey-scheduler`	`0 6 * * *`	Daily 06:00 UTC	≤26h	≤48h
`pulse-survey-scoring`	`0 * * * *`	Hourly	≤1.5h	≤3h
`pulse-survey-alerts`	`0 /4 * *`	Every 4h	≤5h	≤8h

The scoring worker has the tightest bands because it's the highest-cadence: one missed tick already means closed periods stayed un-scored for up to 2h. Three missed ticks = the dashboard is showing stale data nobody's caught up.

The status panel

The panel renders four blocks:

Overall pill — worst of the three workers. Tells you "is anything broken" without scanning the worker table.
Coverage stats — LOBs configured, open periods, closed-in-30d, recent participation rate (averaged across the most-recent CLOSED period per LOB). The juxtaposition tells the story: 0 active configs = engine never set up; configs > 0 but participation < 40% = invites are landing but not converting.
14-day response-volume sparkline — raw pulse_survey_responses row counts per day. Earlier signal than worker freshness when invitations stop converting.
Workers table — one row per worker with its freshness pill, last-run timestamp, and a per-worker metadata hint (lob_count / closing_count / snapshot_count).

Failure mode: a worker is `stale` or `critical`

`stale` (>fresh-band, ≤stale-band)

One missed run. Tail the live logs first:

ssh serge@clear "docker compose -f /home/serge/frontline2/app/docker-compose.demo.yml \
  logs worker --since 24h | grep pulse-survey"

Common causes for each worker:

Scheduler (daily)

Worker container restarted during the 06:00 UTC window. BullMQ's repeat schedule should pick up the next day.
Cadence math edge case. Scheduler opens a new period when last_period.period_start + cadence_days <= today. If a period was created manually outside the worker, the next worker run will re-evaluate; the period UNIQUE on (tenant, lob, period_start) keeps it idempotent.

Scoring (hourly)

No CLOSED periods to score. closing_count = 0 is normal — periods only close when their period_end passes. A "stale" pill without recent period-ends is expected.
Pool exhaustion. A long-running query in another worker can saturate the pool — check pulse_survey_periods for any OPEN row with a past period_end that's still not closed.

Alerts (every 4h)

Snapshot table is empty. No CLOSED periods → no LOB snapshots → snapshot_count = 0. Wait for the next scoring tick to close a period.
Per-LOB config missing the threshold fields. The alerts handler falls back to defaults (15pt drop, -10 floor, 40% participation floor) when the config row is missing fields, so "stale" here usually means the worker didn't run at all, not that thresholds are wrong.

`critical` (>stale-band)

Two or more missed runs. Confirm the worker container is up:

ssh serge@clear "docker compose ps worker"

Then check worker_cron_completions to confirm what the API sees:

SELECT queue_name, last_completed_at, last_run_metadata
  FROM worker_cron_completions
 WHERE queue_name LIKE 'pulse-%'
 ORDER BY last_completed_at DESC;

If the worker logs show the handler completed but the marker wasn't updated, markJobCompleted write failed — check pool connectivity. The helper itself logs cron_chain: markJobCompleted failed.

`never_ran`

No row in worker_cron_completions for that queue. Either a fresh tenant install (expected for the first day) or queue registration is missing from app/services/worker/src/worker.ts.

Failure mode: participation rate is critically low (< 40%)

The participation_rate_recent stat tone goes red at < 40%. Three common causes:

Token URL bounce-back

The agent invite uses a token URL outside the authenticated (app) layout. The middleware exempts /pulse-respond/; without that exemption agents click and get bounced to /login. Verify in app/services/web/src/middleware.ts:

if (pathname.startsWith("/pulse-respond/")) return NextResponse.next();

This was the most common rollout-day failure mode (v1.3 added the exempt; M1 audit verified it's still in place).

Token expiry

Tokens expire 48h after sent_at. A long weekend can leave invites expired before the agent opens them. Check:

SELECT COUNT(*) FILTER (WHERE responded_at IS NOT NULL) AS responded,
       COUNT(*) AS total,
       MIN(sent_at) AS oldest_send
  FROM pulse_survey_sends s
  JOIN pulse_survey_periods p ON p.id = s.survey_period_id
 WHERE p.tenant_id = $1 AND p.status = 'CLOSED' AND p.period_end >= CURRENT_DATE - 30;

If responded / total < 0.40, invites are landing but not converting in the 48h window. Consider extending the cadence or re-checking that email delivery is healthy.

Opt-outs

pulse_survey_opt_outs has rows for employees who've permanently opted out. They're anti-joined out of the scheduler's send fan-out, but a sudden spike in opt-outs lowers the eligible-population denominator. Check:

SELECT COUNT(*) FROM pulse_survey_opt_outs
 WHERE tenant_id = $1 AND re_opted_in_at IS NULL;

Failure mode: HR isn't receiving alert notifications

Three checks:

Notification category routing — pulse_* and pulse.* event_types route to the performance category (added in v1.4 B1). If HR has muted that category, the alerts are still firing — they're just suppressed at the channel layer.
```
SELECT notification_category, in_app_enabled, email_enabled
  FROM notification_preferences
 WHERE user_id = $1 AND notification_category = 'performance';
```
Dedup ledger — pulse_survey_alerts_sent records every fire to prevent re-firing the same alert within the same period. A missing notification could be a dedup hit:
```
SELECT alert_type, fired_at, period_id
  FROM pulse_survey_alerts_sent
 WHERE tenant_id = $1
 ORDER BY fired_at DESC LIMIT 20;
```
Recipient resolution — the alerts worker resolves recipients via user_role_bindings per-LOB (for pulse.participation_low) or tenant-wide (for pulse.score_drop / pulse.floor_consecutive). If no active hr_admin exists on the tenant, the worker logs the gap and continues:

pulse_survey_alerts: tenant has no active hr_admin — fan-out skipped

Failure mode: pipeline anomalies fired

The status panel surfaces pipeline-level anomalies in an amber card. HR admins also receive an aggregated in-app notification (pulse.anomaly_detected, routes through the performance category) when anomalies fire — one notification per HR admin per tenant per day, deterministically deduped via BullMQ jobId so the hourly scoring worker doesn't re-fire the same finding six times a day.

Two anomaly kinds ship today (detected by the pulse-survey-scoring worker):

`participation_cliff`

A closed period's participation_rate_pct fell more than 30% below the LATERAL-picked prior closed period for the same LOB. The LATERAL skips suppressed snapshots — comparing against a suppressed value would compare to null.

Common causes:

Cadence too short. A LOB switched from monthly to weekly without HR re-communicating to the team; agents start ignoring invites.
Token URL bounce-back regression. The middleware exempt at /pulse-respond/ got moved or removed (the v1.3 rollout-day bite). Re-check app/services/web/src/middleware.ts.
Opt-out spike. A bad month — burnout, layoffs, or a controversial change drove more employees to opt out. Use the "Opt-out volume by LOB" cheat-sheet query below.

Diagnosis: Compare the responded-to-sent ratio between the two periods:

SELECT p.id, p.period_start,
       p.sent_count, p.responded_count,
       ROUND(p.responded_count::numeric / NULLIF(p.sent_count, 0) * 100, 2) AS pct
  FROM pulse_survey_periods p
 WHERE p.tenant_id = $1
   AND p.client_lob_id = $2
   AND p.status = 'CLOSED'
 ORDER BY p.period_start DESC
 LIMIT 5;

`scoring_lag`

A period whose period_end is more than 1 day behind today is still status = 'OPEN'. The hourly scoring worker should have closed it; it hasn't.

Common causes:

Scoring worker is critical. Check the per-worker pill on the panel. The scoring worker reads pulse_survey_periods looking for OPEN rows past their end-date; if the worker isn't running, this anomaly is what surfaces the problem before HR notices stale snapshots.
Period missing required data. A period with sent_count = 0 (no recipients) is technically eligible to close — the worker handles this — but check for any pre-conditions on closeAndScore that might be silently bouncing the row.
Transaction stuck. Rare. The closing path uses a per-row transaction; a stuck pooled connection holding a lock would block one period without blocking the next. Check pg_stat_activity for long-running queries.

Diagnosis: Find every period that's overdue:

SELECT p.id, p.tenant_id, p.client_lob_id, p.period_start, p.period_end,
       (CURRENT_DATE - p.period_end) AS days_overdue,
       p.sent_count, p.responded_count
  FROM pulse_survey_periods p
 WHERE p.status = 'OPEN'
   AND p.period_end < CURRENT_DATE - INTERVAL '1 day'
 ORDER BY p.period_end;

For each stuck period, re-run the scoring worker (see Recovery section). If a specific period repeatedly fails to close, log into the worker and trace closeAndScore for that row id.

Why anomalies don't abort the run

Detection runs at end-of-success in the scoring worker. The actual snapshot writes already happened. Aborting on an anomaly would deepen the data gap; the marker JSONB carries the findings forward and the panel + notification surface them.

Why pulse has no backfill script

Unlike the Performance Hub (pnpm backfill:perf-hub) and Gamification (pnpm backfill:gamification-points), pulse doesn't ship a standalone backfill script. The reasoning matters because operators sometimes expect symmetry across modules:

You can't backfill agent responses. A response is an act — an employee opened a token URL and submitted answers. If they didn't do it on day X, no script can synthesize it on day Y. The privacy contract makes this concrete: pulse_survey_responses has no employee_id, so even if we wanted to "recreate" a response we couldn't attribute it.
You can't backfill historical periods. A pulse_survey_periods row represents an invitation that went out. Inviting agents retroactively makes no sense — the email already didn't send, the 48h token window already passed.
You can't backfill snapshots beyond what scoring already computes. The scoring worker UPSERTs snapshots when a period closes; the upsert is idempotent. If a snapshot's data was wrong because of a bug, re-running the scoring worker against the still-OPEN period (or the now-CLOSED one) recomputes the same row — no "stretch the lookback" knob is meaningful because the data scope is the period itself, fixed by its boundaries.

For the realistic recovery scenarios:

A stuck period needs to be closed — see the "Recovery: re-running a single cron tick" section below.
The scoring worker had a regression and wrote wrong snapshots — fix the bug, then re-run scoring; the UPSERT corrects the rows.
A tenant just enabled pulse — accept that pulse data starts accumulating from day 1. There's no equivalent to "credit historical events" because there are no historical events for pulse.

This is not a gap. It's the shape of the data model.

Recovery: re-running a single cron tick

To trigger a one-off run (no dedicated backfill script for pulse yet — H2 of the prod-readiness loop):

ssh serge@clear "docker compose exec api node -e \"
  const { Queue } = require('bullmq');
  const q = new Queue('pulse-survey-scoring', {
    connection: { host: 'redis', port: 6379 }
  });
  q.add('manual-recovery', {}).then(() => process.exit(0));
\""

Substitute the queue name for the worker you want to re-run. All three workers are idempotent (UPSERT into snapshots, dedup ledger for alerts, UNIQUE on periods), so re-running over the same day is safe.

Diagnostic queries cheat sheet

Current period status across the tenant

SELECT cal.name AS lob, p.status, p.period_start, p.period_end,
       p.sent_count, p.responded_count, p.participation_rate_pct
  FROM pulse_survey_periods p
  JOIN client_account_lobs cal ON cal.id = p.client_lob_id
 WHERE p.tenant_id = $1
   AND p.period_end >= CURRENT_DATE - INTERVAL '30 days'
 ORDER BY p.period_start DESC;

eNPS trend per LOB over the last 6 months

SELECT s.scope_id AS lob_id, s.period_start, s.enps_score, s.suppressed
  FROM pulse_score_snapshots s
 WHERE s.tenant_id = $1 AND s.scope_type = 'LOB'
   AND s.period_start >= CURRENT_DATE - INTERVAL '6 months'
 ORDER BY s.scope_id, s.period_start;

Show the chain status all at once

SELECT queue_name, last_completed_at,
       (now() - last_completed_at) AS age,
       last_run_metadata
  FROM worker_cron_completions
 WHERE queue_name LIKE 'pulse-%'
 ORDER BY queue_name;

Opt-out volume by LOB

SELECT eph.client_lob_id, COUNT(*) AS opted_out
  FROM pulse_survey_opt_outs o
  JOIN employees e ON e.id = o.employee_id
  JOIN employee_profile_history eph
    ON eph.employee_id = e.id AND eph.effective_to IS NULL
 WHERE o.tenant_id = $1 AND o.re_opted_in_at IS NULL
 GROUP BY eph.client_lob_id;

Escalation

If a worker is critical for >72h, page the on-call. The audit trail is audit_events (search resource_type = 'pulse_*') and pulse_survey_alerts_sent — both are append-only.

Privacy contract reminder: pulse_survey_responses has no employee_id column by schema design. The only link from a response to an employee is via pulse_survey_sends.responded_at (the responded_at field), and that link is one-way (you can tell that somebody responded to a given send, but you can't trace a specific response row back to a sender). Do not add an employee_id join shortcut to debug a low-participation case — the absence is the privacy guarantee. AC-003 in spec 27B §11 is the canonical reference.

Quick reference — service map​

Cron-chain cadences​

The status panel​

Failure mode: a worker is stale or critical​

stale (>fresh-band, ≤stale-band)​

critical (>stale-band)​

never_ran​

Failure mode: participation rate is critically low (< 40%)​

Token URL bounce-back​

Token expiry​

Opt-outs​

Failure mode: HR isn't receiving alert notifications​

Failure mode: pipeline anomalies fired​

participation_cliff​

scoring_lag​

Why anomalies don't abort the run​

Why pulse has no backfill script​

Recovery: re-running a single cron tick​

Diagnostic queries cheat sheet​

Current period status across the tenant​

eNPS trend per LOB over the last 6 months​

Show the chain status all at once​

Opt-out volume by LOB​

Escalation​