Design a Chat System

1Clarifying the requirements#

Pin down what you are on the hook for before drawing a single box. Scope swings wildly depending on the answers, so treat these as load-bearing, not small talk.

Candidate: Are we doing one-on-one chat, group chat, or both? And is there a group size cap? Interviewer: Both. One-on-one and small groups, say up to 100 members.

Candidate: Good, 100 is an important cap. It means I can lean on fan-out on read and not worry about the WhatsApp-style 500k broadcast channel. Do we need delivery and read receipts, typing indicators, and online status? Interviewer: Yes to delivery and read receipts, and online status. Typing indicators are a bonus if we have time.

Candidate: Media? Images, voice notes, files? Interviewer: Assume text for the core design. Mention how you would bolt on media.

Candidate: Message history: do we keep it forever, and does it sync across a user's devices? Interviewer: Keep it indefinitely, and yes, multi-device sync matters.

Candidate: Last one. End-to-end encryption? Interviewer: Out of scope today. Flag where it would change things.

What those answers just told me:

• The 100-member cap kills write fan-out. I can store one copy per conversation and read-fan-out, no million-follower broadcast problem.
• "Keep history forever" makes this a sharded-store problem from year one, not a cache problem.
• "Multi-device sync matters" means I need a per-device cursor, and I should design sync as the primitive, not an afterthought.
• E2E out of scope, but I should name where it would reshape the model instead of pretending it bolts on.

Functional requirements

• Send and receive text messages in one-on-one and group chats (up to 100 people).
• Deliver messages in near real time to online recipients, and store them for offline ones.
• Persist full message history, retrievable with pagination.
• Delivery receipts (sent, delivered, read) and online presence.
• Multi-device: a user logged in on phone and laptop sees the same thread.

Non-functional requirements

• Low latency: end to end under 500 ms for online users on a decent network.
• High availability for the send and receive path. A message I hit send on should not vanish.
• Durability: once the server acks a message, we never lose it.
• Scale to tens of millions of daily users.
• Ordered within a conversation. Messages in one thread show up in a consistent sequence.

2Back-of-the-envelope estimation#

Get the scale in your head, because it decides the architecture. One number drives the whole transport choice, and I will flag it when I reach it.

Users and traffic.

• 20 million daily active users.
• 40 messages sent per user per day, so 20M x 40 = 800 million messages/day.
• 800M over 86,400 seconds is about 9,300 writes/sec average.
• Evenings are spiky, so plan for 3x: roughly 28,000 writes/sec at peak.

Fan-out.

• A one-on-one message hits 1 recipient; a group averages maybe 8.
• Call the effective fan-out factor 3.
• Delivery events run around 28,000 x 3 = 84,000/sec at peak.

Connections (the one that drives the design).

• If 10% of DAU are online at once, that is 2 million concurrent WebSocket connections.
• One box holds maybe 50k to 100k sockets comfortably.
• So I need on the order of 20 to 40 connection servers, rounded up for headroom.

Storage.

• Average message plus metadata (ids, timestamps, status) is about 300 bytes.
• 800M/day x 300 bytes = 240 GB/day, about 7.2 TB/month, roughly 87 TB/year.
• Kept indefinitely, so this is a sharded store from day one. Text is cheap, but it compounds.

Bandwidth.

• 84k delivery events/sec x 300 bytes is about 25 MB/sec of payload leaving the delivery layer.
• Trivial on the wire. The hard part is the connection count and the write rate, not raw bytes.

Takeaway.

• This is a connection-heavy, write-heavy, storage-accumulating system.
• The transport choice and the datastore choice carry the design. Bytes per second are a rounding error.

3Core APIs#

Chat is mostly a persistent socket, but I still want clean HTTP for the lifecycle (creating conversations, loading history on cold start) and a tight message schema over the socket for the hot path.

POST /v1/conversations
Body: { "type": "group", "member_ids": ["u_12", "u_88"], "title": "Team" }
Returns: { "conversation_id": "c_501" }

GET /v1/conversations/c_501/messages?before=m_9000&limit=50
Returns: paginated messages, newest-first, cursor-based

POST /v1/conversations/c_501/messages
Body: { "client_msg_id": "uuid-abc", "text": "hey" }
Returns: { "message_id": "m_9001", "seq": 4412, "ts": 1730600000 }

Two things about that HTTP surface:

• The POST .../messages endpoint is the fallback for when a socket is not available.
• The primary send path is the socket. HTTP is the cold-start and backup path, not the hot one.

Over the WebSocket, both directions speak small JSON frames:

{ "op": "SEND", "conversation_id": "c_501", "client_msg_id": "uuid-abc", "text": "on my way" }
{ "op": "MSG",  "conversation_id": "c_501", "message_id": "m_9001", "seq": 4412, "from": "u_12", "text": "on my way", "ts": 1730600000 }
{ "op": "ACK",  "client_msg_id": "uuid-abc", "message_id": "m_9001", "seq": 4412 }
{ "op": "RECEIPT", "message_id": "m_9001", "state": "read", "by": "u_88" }

That client_msg_id is load-bearing, and I say why out loud because it sets up a later deep dive.

• The client generates a UUID before sending, and the server dedupes on it.
• If the socket drops and the client retries, we do not get a duplicate message.
• This is how you get at-least-once delivery over the wire but exactly-once storage.
• Any chat design without it is quietly broken the first time a network flaps.

4High-level design#

The spine, drawn fast:

• Clients hold a long-lived WebSocket to a connection server.
• A sent message gets persisted, then pushed to whichever connection servers hold the recipients' sockets.
• The catch: User A is on connection server 1, User B is on server 7, and server 1 has never heard of B.
• So we need a way to route a message from box A to box B. That routing layer is the heart of it.

The pieces:

• Load balancer. Terminates the WebSocket upgrade and spreads connections. Sticky-per-connection is fine (a socket lives on one box anyway), but it does not need message-level stickiness because routing happens inside.
• Connection servers. Stateful on purpose. Each owns a set of live sockets and knows which user_id sits on which socket. With 2 million sockets over 20 to 40 boxes, each tracks 50k to 100k user-to-socket mappings in its own memory. The only stateful tier in the hot path.
• Message service. Stateless. Validates, assigns a sequence number, writes to the store, publishes a delivery event. Holds no connection state, so I scale and restart it freely.
• Message store. The durable record of every message. Sharded by conversation.
• Presence store (Redis). Who is online, and critically which connection server holds each online user. This is the routing table that answers "where is B?"
• Message queue / pub-sub. Carries delivery events from the message service to the right connection servers. Redis pub-sub or Kafka depending on the durability I want.
• Push service. For offline recipients, hands off to APNs or FCM for a banner notification.

The core flow, end to end:

The single most important line in this design: ack on store write, not on recipient delivery.

• Durability is the server's promise, and it happens first.
• Delivery is best-effort-then-stored. If B is offline, the message is already safe and B pulls it on next connect.
• Ack only on recipient delivery and a sender sits forever on "sending" because B's phone is in a tunnel.
• That is both a worse experience and a worse guarantee. Store first, always.

5Data model#

Use two stores on purpose, each picked for its access pattern. Message history and conversation metadata are different beasts, so I do not force them into one database.

• Messages go in a wide-column store (Cassandra or similar). The access pattern is "give me the last N messages in this conversation," the volume is enormous, and it is append-heavy.
• Metadata (users, membership) goes relational, where I want joins and constraints and the row count is small by comparison.

messages (wide-column, partition key = conversation_id, clustering key = seq desc)

• Partition by conversation_id so loading a thread is a single-partition range scan, the hot path for every chat open.

conversations (relational)

conversation_members (relational)

That last_read_seq per member is doing a lot of work.

• Unread count is last_message_seq - last_read_seq, a subtraction, not a scan.
• A read receipt is just bumping that number and broadcasting it.
• No per-message-per-recipient receipts table, which would blow up the write rate at 84k delivery events/sec.

Presence keeps nothing durable.

• In Redis: presence:{user_id} -> {server_id, last_seen} with a short TTL the connection server refreshes.
• TTL lapses, the user is treated as offline. Cheap, self-healing, no cleanup job.
• It doubles as the routing table the deep dive leans on.

6Deep dives#

6.1 Routing a message to the right connection server#

This is the signature hard part, and the part candidates fumble. User A is on Conn Server 1, User B is on Conn Server 7, and the two boxes have never heard of each other.

How the routing works:

• When B connects, their connection server writes presence:u_88 -> {server_id: ws7} to Redis.
• When the message service needs to deliver to B, it looks up B's server and publishes to a channel ws7 is subscribed to.
• ws7 receives the event and writes the frame to B's socket.
• Redis does double duty: it answers "is B online and where," and the pub-sub channel is the pipe to exactly that box.

Two ways to wire the pub-sub, and it is a real trade-off worth saying out loud:

I go with per-server channels.

• The presence lookup is one Redis GET, well under a millisecond.
• It keeps delivery traffic proportional to the message rate instead of multiplying by server count.

6.2 Offline delivery and sync#

Treat offline as the normal case, not the exception. With only 10% of DAU online at once, most recipients at any moment are not holding a socket.

When there is no live socket:

• The message service finds no presence entry (or the push fails) and does nothing special for the socket.
• The message is already in the store.
• It calls the push service for a lightweight APNs/FCM banner so the phone surfaces it.

The sync model is a per-conversation cursor. Each client remembers the highest seq it has seen per conversation. On reconnect it sends a handshake:

{ "op": "SYNC", "cursors": { "c_501": 4410, "c_777": 120 } }

Why this is clean:

• The server returns everything after each cursor.
• Because seq is monotonic per conversation and messages are clustered by seq descending, it is a clean range scan: "all messages in c_501 where seq > 4410."
• No scan over messages the client already has.

Multi-device falls out for free.

• A laptop offline for a day sends its stale cursors and gets caught up exactly like a phone.
• No special multi-device path, just per-device cursors against the same ordered log.
• Both devices read the same partition, which is the single source of truth.

This is why I spent a sequence number per conversation instead of leaning on timestamps.

• Clocks drift and collide; two messages a millisecond apart can land out of order or with identical timestamps.
• A per-conversation counter gives a total order inside the thread and a dead-simple sync primitive at once.
• I generate it at the message service via an atomic increment on conversations.last_message_seq, or a per-shard sequencer if that row gets hot (more on that contention below).

6.3 Group fan-out#

One-on-one is a fan-out of 1. Groups change the shape: a 100-person group where someone posts is 1 write and up to 99 deliveries.

I pick read fan-out.

• A 100-member group post becomes 1 storage write plus up to 99 cheap pub-sub publishes, not 99 storage writes.
• Offline members get nothing extra stored; they range-scan the same partition on sync.

The thing to watch, raised before the interviewer does: a hot conversation.

• A busy 100-person group hammers one partition, since everything for a conversation lives on one shard by design.
• Fix if it becomes real: bucket the partition by time, so the key becomes conversation_id + day and one thread's load spreads across partitions over time.
• Cost: a slightly more complex read that may touch two buckets near a day boundary.
• I reach for this only once a specific thread proves hot, not preemptively, because it complicates every read.

6.4 Receipts and presence at scale#

Receipts and presence look trivial and are secretly a write-amplification trap, so they earn a deep dive.

Read receipts, done naively, explode.

• A read receipt in a 100-person group is 99 writes and 99 pushes every time one person opens the chat.
• Across a busy group, receipts dwarf actual messages.
• Fix: last_read_seq on conversation_members. Reading a thread is a single bump of one row; the broadcast is one small RECEIPT frame fanned out like a message.
• For one-on-one that is one write and one push. For groups I aggregate: periodically broadcast the set of members whose last_read_seq crossed a message, so the rate is bounded by the interval, not by reader count.

Presence is the sneakier one.

• Naive design broadcasts online/offline to all of a user's contacts on every state change.
• A user with 500 contacts flipping online fires 500 events, and flapping mobile networks make that nearly constant.
• Damp it two ways. First, debounce state changes, so a phone bouncing off a cell tower three times in ten seconds produces at most one change.
• Second, only push presence to people actively viewing a thread with that user, not every contact everywhere. Nobody cares that someone went online in a chat they are not looking at.
• Offline detection rides the Redis TTL, not an explicit disconnect, which also handles the ugly case where a socket dies with no clean close and no FIN arrives.

7Bottlenecks and trade-offs#

Connection servers are stateful, and that is the main operational cost.

• Two million live sockets means a box dying sends 50k to 100k clients reconnecting at once.
• Defend with fast client reconnect using exponential backoff with jitter, so orphaned clients do not thunder onto one surviving box in lockstep.
• Keep connection servers thin: they hold sockets and route frames, nothing heavy.
• The stateless message service absorbs the logic, so I scale and restart it without dropping a socket.

Presence is the sneaky scaling problem.

• The fix is in 6.4: debounce state changes, scope pushes to people viewing a thread with that user, detect offline on the TTL.
• I flag it again here because it is the piece most likely to quietly take down Redis under load. Naming it as a known hazard is worth points.

Ordering versus latency.

• The per-conversation sequence number means the message service touches one counter on every write.
• For a hot thread that counter is a contention point: every send serializes through one atomic increment.
• Escape hatch: a per-shard sequencer, or snowflake-style ids with a sortable timestamp plus a tiebreaker, trading a dense counter for a sortable-but-gappy one.
• For chat, gaps are fine as long as order holds, because the client only needs "newer than my cursor," not a dense count.

Exactly-once is a lie we manage, not achieve.

• The wire is at-least-once: clients retry, networks duplicate.
• Lean on client_msg_id dedupe at the store, so a second write with the same id is a no-op that returns the original seq.
• Lean on the monotonic seq so clients detect and drop a duplicate they already rendered.
• End to end it behaves like exactly-once from the user's eyes, which is what matters.

Datastore choice.

• A wide-column store for messages gives up rich queries and joins on message data.
• I cannot easily ask "all messages containing the word refund across all conversations" from the primary store.
• If search matters, stream messages into a search index like Elasticsearch off the write path.
• Keep the primary store tuned for the one query that is 99% of traffic: the conversation range scan. Excellent at the dominant pattern beats mediocre at everything.

Media, encryption, the stuff I deferred.

• Media does not go through the message store. The client uploads to object storage, gets a URL, and the message carries the URL, so a 10 MB voice note never touches the 300-byte-average path.
• End-to-end encryption moves sequence numbering and dedupe client-side and turns the server into a dumb relay that cannot read payloads, which breaks server-side search and read receipts.
• Worth flagging that E2E is not a bolt-on; it reshapes the model.

Treat the socket as a fast path over a durable log, never as the source of truth.

8Summary#

The decisions I would stand behind:

• WebSockets on stateful connection servers for the real-time path, with a stateless message service doing the work. Forced by the 2 million concurrent connections number.
• A presence table in Redis that doubles as the routing table, mapping each online user to the connection server holding their socket, so delivery events go to exactly the right box instead of all forty.
• Durability first: ack the sender on store write, not on recipient delivery. The socket push is an optimization on a durable, ordered log, so a dropped push is a non-event.
• Per-conversation monotonic sequence numbers for ordering and for a cursor-based sync that handles offline and multi-device identically.
• Read fan-out for groups, one stored copy per conversation, with time-bucketing as the escape valve for hot threads.
• client_msg_id dedupe to turn at-least-once wire delivery into exactly-once storage.
• Out of scope by choice: media bytes, E2E encryption, typing indicators beyond a sketch. Naming them keeps me honest about what the time covered.

For how specific companies push on this one, see our related guides on real-time systems and messaging infrastructure interviews.