PHASE 4Intermediate ~8 min· topic 18 of 18Level 3

Problem 4.18 — In-Memory Pub/Sub System

In one line

Topics, publishers, and subscribers in one process: fan-out delivery, per-subscriber ordering, slow-consumer isolation, and offsets for replay, a small model of Kafka and Redis Pub/Sub.

0/18 · 0%

Think of it like this

A newsletter service. Writers publish to a topic ('sports'); everyone subscribed gets a copy; a subscriber on holiday picks up where they left off when they return (offset); one slow reader doesn't delay everyone else's delivery.

Key ideas

01
Requirements: create topics; publish messages to a topic; subscribers receive messages in order per topic; multiple subscribers each get every message (fan-out); a slow subscriber must not block publishers or other subscribers; optional replay from an offset.
02
Design: Topic holds an append-only list of messages (the log) and its subscriptions; each Subscription has its own OFFSET and a worker that delivers messages in order; publishers just append and notify. This is Kafka's model in miniature (Phase 12.39).
03
Concurrency: appends are synchronised per topic (or use a lock-free structure); each subscription runs on its own thread/executor, so a slow consumer only falls behind (its offset lags) instead of blocking others. Bound memory with retention (drop messages older than N or below the minimum offset).
04
Patterns: OBSERVER (subscribers), PRODUCER-CONSUMER (Phase 5), and ITERATOR over the log from an offset.

Code & diagrams

PubSub.javajava

Each subscription tracks its own offset and delivers on its own thread.

interface Subscriber { void onMessage(String topic, long offset, String message); }

final class Topic {
  private final String name;
  private final List<String> log = new ArrayList<>();
  private final List<Subscription> subs = new CopyOnWriteArrayList<>();
  Topic(String name) { this.name = name; }

  synchronized long publish(String msg) {
    log.add(msg);
    subs.forEach(Subscription::wake);
    return log.size() - 1;
  }
  synchronized Optional<String> read(long offset) {
    return offset < log.size() ? Optional.of(log.get((int) offset)) : Optional.empty();
  }
  Subscription subscribe(Subscriber s, long fromOffset) {
    Subscription sub = new Subscription(this, s, fromOffset);
    subs.add(sub);
    return sub;
  }
  String name() { return name; }
}

final class Subscription implements Runnable {
  private final Topic topic; private final Subscriber subscriber;
  private long offset; private final Semaphore signal = new Semaphore(0);
  Subscription(Topic t, Subscriber s, long from) {
    topic = t; subscriber = s; offset = from;
    Thread.ofVirtual().start(this);                        // one worker per subscription
  }
  void wake() { signal.release(); }
  public void run() {
    while (!Thread.currentThread().isInterrupted()) {
      Optional<String> m = topic.read(offset);
      if (m.isPresent()) { subscriber.onMessage(topic.name(), offset, m.get()); offset++; }   // in-order, at-least-once
      else signal.acquireUninterruptibly();                 // wait for new messages
    }
  }
}

Explain without notes

How does this design stop one slow subscriber from delaying others?

Practice

Add consumer groups: several subscribers share the work of one subscription.

Trade-offs

↔
Keeping a log enables replay and independent consumers but costs memory/retention; fire-and-forget pub/sub (Redis) is simpler but drops messages for offline subscribers.

Run it in production

You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:

Stateful · Consumer groups & delivery semantics

Rebalancing, offsets, at-least-once, idempotency, DLQs, and consumer lag.

Stateful · RabbitMQ, SQS/SNS & choosing a broker

Queues vs logs, visibility timeouts, DLQs, and fan-out in practice.

Completion checklist

I can implement fan-out with per-subscriber offsets and isolation
I can relate it to Kafka's partitions and consumer groups

Back to phase

Command Palette