Skip to main content

Stream-stream joins — order meets shipment

Intermediate ⏱ 15 min streaming · join · correlation

You will build: a pipeline that joins orders with their shipments inside a 5-minute window — the primitive behind order↔payment matching, alert↔ack correlation and supply-chain gap detection.

orders + shipments → streaming:join → (matched pairs out)

Prerequisites​

1 · Scaffold an app with a streaming stage​

pulse new fulfillment --source webhook --stage streaming:match

2 · Declare the join​

stages:
  - name: match
    engine: streaming
    operators:
      - type: join
        leftTopic: fulfillment.in
        rightTopic: shipments
        on: orderId
        window: "5m"
        joinType: inner        # inner · left · right · full

The join pairs every left event with right events sharing the same on key, as long as both arrive within the window. You don't wire the second topic anywhere else — declaring the join auto-subscribes the stage to rightTopic.

3 · Deploy and feed both sides​

pulse secret set PULSE_WEBHOOK_SECRET dev-secret   # the scaffolded webhook source requires it
pulse deploy .
pulse events publish --topic fulfillment.in --value '{"orderId":"o-42","item":"drone","amount":249}'
pulse events publish --topic shipments --value '{"orderId":"o-42","carrier":"dhl","tracking":"T-981"}'

4 · Watch the pair come out​

pulse events tail --topic fulfillment.match.out

One joined event for o-42, carrying fields from both sides — the order and its shipment, correlated in flight.

5 · Make the misses the signal​

        joinType: left

With a left join, an order that never meets a shipment inside the window still emits — with the right side empty. Chain a rule-based stage on "shipment missing" and you've built shipped-but-never-scanned detection: the gaps are the alerts.

What just happened​

You correlated two independent event flows declaratively — no consumer groups, no state store code, no reconciliation batch. The window bounds memory and defines lateness; the join type decides whether matches, misses or both flow downstream.

Troubleshooting​

No joined events appear

Check the on field name exists in both payloads with identical values, and that both events were sent within the window. pulse events tail --topic shipments confirms the right side actually landed.

Deploy rejects the operator

leftTopic, rightTopic and window are required; joinType must be inner | left | right | full. The validator names the offending field.

Next​