Skip to main content

Sources & sinks — the connector tour

Beginner ⏱ 10 min sources · sinks · connectors

You will build: the same app fed three different ways — webhook, tailed file, polled HTTP — so you see that stages never care where events come from.

[webhook | file-tail | http-poll] → stages → sink

Prerequisites​

  • Any scaffolded app (start here if you don't have one).

1 · List what's available​

pulse new --list-sources
pulse new --list-sinks

Sources include webhook · file-tail · file-source · http-poll · jdbc-source …; sinks include webhook · slack · kafka · jdbc · mqtt · s3 · file …. The palette comes from the installed connector schemas — this list is your ground truth, always current for your version.

2 · Feed by webhook (what you already did)​

pulse events publish --topic hello.in --value @sample.json

Under the hood this POSTs to the app's ingress — any HTTP client can do the same (see Call your app from anywhere).

3 · Switch to a tailed file​

source:
  kind: file-tail
  path: /tmp/hello-events.log

Redeploy, then feed it by appending lines:

echo '{"severity":"error","msg":"disk full"}' >> /tmp/hello-events.log

4 · Switch to HTTP polling​

source:
  kind: http-poll
  url: https://api.example.com/events
  intervalSeconds: 30

Pulse polls the endpoint and turns each new record into an event. Same stages, untouched, third source in a row.

5 · Point the sink somewhere real​

sink:
  kind: webhook
  url: ${secret:DOWNSTREAM_WEBHOOK}

Set the secret first: pulse secret set DOWNSTREAM_WEBHOOK https://…. A Slack sink works the same way with a Slack webhook URL.

What just happened​

Sources and sinks are interchangeable adapters around a stable core. Your logic lives in stages against topics; the connector palette is how the outside world reaches those topics. This is why migrating an app from "reading a log file" to "receiving webhooks at scale" is a two-line diff.

Troubleshooting​

file-tail sees nothing

Check the path is absolute and the file exists before deploy. Then append with >> — the source picks up new lines; it doesn't re-read history by default.

http-poll floods me with duplicates

Give the source a way to know what's new (cursor/timestamp param on your endpoint). Add a dedup operator in a streaming stage as a belt-and-braces guard.

Next​