Chat System Message Lifecycle Visualization

Mental Model

Chat delivery is a state-routing problem. The core system architecture has to route messages in real-time across a horizontally scaled gateway tier, handle offline delivery transitions when recipients are disconnected, and recover gracefully when nodes fail or clients reconnect.

The core mental model divides responsibilities clearly:

• Gateways own connections because WebSockets are long-lived and stateful.
• Services own decisions because validation, persistence, and routing should scale horizontally.
• Stores own correctness because Pub/Sub and sockets are fast paths, not durable sources of truth.

For sandbox parameter tuning, you can also use the retained chat system simulator.

Components and Why They Exist

Clients and Load Balancer

Clients represent user devices that establish WebSocket sessions. The load balancer distributes upgrade requests across gateways. Clients dispatch heartbeat signals and messages, while the load balancer balances TCP connections and routes reconnecting users. In the demo, observe the client status change to online, and verify that reconnecting clients are dynamically routed to healthy gateways after failures.

WebSocket Gateways

Stateful edge nodes maintaining open socket connections. Manage TCP connections, parse incoming WebSocket frames, write outbound messages, and track heartbeat keepalives. In the demo, notice that Watch for: The selected gateway owns socket state, heartbeat status, and outbound writes for whichever client is currently connected to it.

Chat Service and Session Registry

The stateless core application tier and user routing database. The chat service validates incoming messages and writes to storage. The session registry holds key-value mappings of online users to their current gateway node. In the demo, observe the chat service querying the session registry to locate Bob’s active gateway.

Pub/Sub, Message Store, and Offline Queue

Pub/Sub moves fast online events, the message store preserves timelines, and the offline queue tracks messages for disconnected clients. Transport routing events, persist message/receipt state, and buffer backlog messages. In the demo, notice that Watch for: Pub/Sub should show routing motion for online delivery, while the message store and offline queue should carry the durable recovery path.

Push Service and Receipt Service

FCM/APNs wake disconnected clients, and receipt processing tracks delivery states. The push service wakes sleeping devices, and the receipt service registers delivery/read transitions and updates timelines. In the demo, you can see the push service wake an offline client, and watch receipt notifications propagate back to the sender.

Edge Cases and Misconceptions

• Pub/Sub Is Not Message Durability: Redis Pub/Sub is fire-and-forget. It does not buffer messages for disconnected gateways or offline users. A durable database is always required to sync offline history.
• Presence Registry Drift: If a gateway crashes abruptly, the registry might think the user is still connected to it. Systems use TTL-based lease renewal and heartbeat checks to clean up dead connections.
• Push Notification Is Only a Wakeup: APNs or FCM push notifications are best-effort delivery mechanisms. Relying on them as the primary message delivery pathway introduces ordering issues; clients should always fetch missed frames from the message store upon wakeup.
• Read Receipts vs. Delivery Receipts: A delivery receipt indicates the gateway/client received the message packets. A read receipt indicates the user viewed the conversation. These are different user transitions and require separate database and queue handlers.