Ch. 5: Stream live clicks

iii-stream is for real-time data transmission: pushing data to a client the moment it changes, like a live feed of clicks for a dashboard. A stream is bidirectional (subscribers can send messages back as well as receive them), but here you only need to broadcast clicks outward. You’ll move the live-broadcast concern into its own click-streamer worker so the link worker stays focused on links.

Add the workers

iii-stream is how we will send clicks to clients in Chapter 7. We’ll make a new click-streamer worker to manage the streaming, so create it the same way you created link in Chapter 1 and analytics in Chapter 4. iii-stream must start with the engine, so declare it in the Compose engine configuration:

worker-compose.yaml

engine:
  workers:
    iii-stream: {}

mkdir -p click-streamer/src

Create the worker manifest and package metadata:

click-streamer/iii.worker.yaml

name: click-streamer
scripts:
  start: pnpm start

click-streamer/package.json

{
  "name": "click-streamer",
  "version": "0.1.0",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "tsx watch src/index.ts"
  },
  "dependencies": {
    "iii-sdk": "0.21.4",
    "@iii-dev/helpers": "0.21.4",
    "tsx": "^4.22.3"
  }
}

cd click-streamer && pnpm install && cd ..

Broadcast clicks in real time

Have link announce each click with a pubsub event. Then, have click-streamer push the event onto the live feed.

Add a `link.clicked` event to the `link` worker

First, publish a link.clicked event from link::record_click:

link/src/index.ts

worker.registerFunction(
  "link::record_click",
  async (payload: { code: string; clicked_at: string }) => {
    await worker.trigger({
      function_id: "database::execute",
      payload: {
        db: DB,
        sql: "INSERT INTO clicks (code, clicked_at) VALUES (?, ?)",
        params: [payload.code, payload.clicked_at],
      },
    });
    worker.trigger({
      function_id: "publish",
      payload: { topic: "link.clicked", data: payload },
      action: TriggerAction.Void(),
    });
    return { recorded: true };
  },
);

In the new code above we didn’t use await and set the action to TriggerAction.Void(). This causes the function to return immediately before it completes. This is a simple performance enhancement with things like pubsub where we don’t need guaranteed execution.

Setup the `click-streamer` worker

Now write the click-streamer worker. It subscribes to link.clicked and broadcasts each click to a clicks stream with stream::set. A stream::set both stores the item and pushes it to every WebSocket subscribed to that stream and group. Create click-streamer/src/index.ts:

click-streamer/src/index.ts

import { registerWorker } from "iii-sdk";
import { Logger } from "@iii-dev/helpers/observability";

const worker = registerWorker(process.env.III_URL ?? "ws://localhost:49134", {
  workerName: "click-streamer",
});
const logger = new Logger();

worker.registerFunction(
  "click-streamer::broadcast",
  async (data: { code: string; clicked_at: string }) => {
    await worker.trigger({
      function_id: "stream::set",
      payload: {
        stream_name: "clicks",
        group_id: "all",
        item_id: `${data.code}-${data.clicked_at}`,
        data,
      },
    });
    return { streamed: true };
  },
);

worker.registerTrigger({
  type: "subscribe",
  function_id: "click-streamer::broadcast",
  config: { topic: "link.clicked" },
});

logger.info("click-streamer ready");

iii trigger -n linkly compose::add worker=./click-streamer

The browser you build in Chapter 7 subscribes to clicks/all and counts those broadcasts live.

See it work

With the engine running, create and follow a link a few times:

curl -s -X POST http://127.0.0.1:3111/links \
  -H 'Content-Type: application/json' -d '{"url":"https://iii.dev","code":"stream-me"}'
for n in $(seq 1 3); do curl -s -o /dev/null http://127.0.0.1:3111/s/stream-me; done

Then read the live clicks stream:

iii trigger stream::list stream_name=clicks group_id=all

Each redirect lands in the stream as the click-streamer worker broadcasts it.

Conclusion

Linkly now streams every click to subscribers in real time through a dedicated click-streamer worker. Next, in Ch. 6: Move bulk data with channels, you bulk-load links from a CSV in a single streamed upload.

​Add the workers

​Broadcast clicks in real time

​Add a link.clicked event to the link worker

​Setup the click-streamer worker

​See it work

​Conclusion

Add the workers

Broadcast clicks in real time

Add a `link.clicked` event to the `link` worker

Setup the `click-streamer` worker

See it work

Conclusion