Skip to main content

Crash-resume — kill Pulse mid-flight

Intermediate ⏱ 10 min durability · offsets · production

You will build: confidence. You'll kill a running Pulse the ugly way, restart it, and verify that your pipeline picks up exactly where it stopped — because per-agent stream offsets are persisted, not held in memory.

events flowing → kill -9 → restart → same offsets, no gap, no dupes

Prerequisites​

1 · Establish a flowing baseline​

pulse events publish --topic hello.in --value '{"severity":"error","msg":"before-crash-1"}'
pulse events publish --topic hello.in --value '{"severity":"error","msg":"before-crash-2"}'
pulse events tail --topic hello.triage.out

Two events through, visible on the output. Leave the tail running in its own terminal.

2 · Kill it — no graceful shutdown​

Find the Pulse process and kill it hard (kill -9), or just close the terminal running pulse server start --dev. The tail disconnects: the server is genuinely gone mid-flight.

3 · Restart​

pulse server start --dev

Same data directory, same apps: your pipeline redeploys from its persisted state — you don't re-run pulse deploy.

4 · Prove the resume​

pulse events publish --topic hello.in --value '{"severity":"error","msg":"after-crash"}'
pulse events tail --topic hello.triage.out

The new event flows through normally. Now scroll the topic from the beginning: the two before-crash events are still there — durable topics survived — and they were not reprocessed on restart: no duplicate outputs, because each agent resumed from its persisted per-(agent, topic) offset.

What just happened​

Two separate guarantees worked together:

  1. Topics are durable logs — events written before the crash are still readable after it.
  2. Consumption is checkpointed — each agent's read position is persisted, so a restart resumes exactly where it stopped: nothing skipped (no gap), nothing replayed into your sinks (no double-act).

This is the boring, load-bearing foundation under everything else in the To Production path — and it's on by default, not a mode you enable.

Troubleshooting​

After restart my app is gone

You started the server with a different data directory. Dev mode persists to its default data dir — restart with the same one you used before the kill.

I see the old events again in my tail

Reading a topic from the beginning is a reader choice (that's the durability working). The guarantee is about the agents: check the stage's output topic — each pre-crash input produced exactly one output.

Next​