Design a Video Conferencing System

1Clarifying the requirements#

I always spend the first five minutes here. With a media problem, a wrong assumption early wrecks the whole design, because a 5-person call and a 10,000-viewer webinar are not the same system in a different hat. Different topologies, different latency budgets, different cost models.

Candidate: Are we optimizing for small calls or huge webinars? Those are different systems.

Interviewer: Let's support both. Normal meetings up to maybe 50 active participants, and a broadcast mode for up to 10,000 viewers.

Candidate: Do we need recording and screen share?

Interviewer: Yes, both. Recording to cloud storage, and screen share as a second video track.

Candidate: What about the floor on latency? Interactive video usually wants under 200ms mouth to ear, because past that people start talking over each other.

Interviewer: Target 150ms for interactive participants. Webinar-only viewers can tolerate 2 to 5 seconds.

Candidate: Last one. Do we build the media transport ourselves or assume WebRTC on the client?

Interviewer: Assume WebRTC clients. You own the server side.

That last answer is the big one. Here is why it reshapes the whole problem:

• WebRTC hands us codecs (VP8, VP9, H.264, Opus for audio), on-the-wire encryption (DTLS-SRTP), NAT traversal (ICE with STUN and TURN), and congestion control, all for free on the client.
• Build media transport yourself and this becomes a six-month project just to get two laptops talking.
• With WebRTC assumed, our job narrows to two backend planes: the signaling plane that helps peers negotiate, and the media plane that moves and routes packets.
• I keep coming back to that split, because it is the spine of everything.

Functional requirements

• Create and join a meeting by room ID, with auth.
• Real-time audio and video between participants.
• Screen sharing as an additional media track.
• Server-side recording to durable storage.
• Webinar mode: a few presenters, thousands of passive viewers.
• Mute, camera off, leave, and a participant roster.

Non-functional requirements

• Interactive latency under 150ms mouth to ear at p50.
• Survive a participant's bad network (packet loss up to 10%) without freezing the whole call. This is the signature hard part and gets its own deep dive.
• 99.95% availability for the signaling service.
• Scale to millions of concurrent participants globally.
• Encrypted media on the wire. The server relays encrypted SRTP packets, and for recording we decrypt in a controlled, trusted path.

2Back-of-the-envelope estimation#

Anchor on numbers, because the media math drives the topology choice. I am working toward one number that decides everything, and I will flag it when I reach it.

Users.

• 20 million daily active users.
• At peak, 2 million are in a call at the same time.
• Average call size, 5 people.

Concurrent calls.

• 2,000,000 participants / 5 per call = 400,000 simultaneous meetings.

Signaling QPS.

• Signaling is low volume: join, leave, mute, ICE candidate exchange. Call it 10 messages per participant per minute.
• 2,000,000 * 10 / 60 = ~333,000 signaling messages/sec.
• That is nothing for a WebSocket fleet. One modern node holds tens of thousands of open sockets, so a few hundred nodes cover it with headroom.
• Signaling is easy. Media is the beast, and here is why.

Bandwidth per stream.

• A 720p video stream at reasonable quality is about 1.5 Mbps.
• Audio (Opus) is ~50 Kbps.
• Round a full video plus audio stream to ~1.6 Mbps.

The topology decision falls out of this. In a 5-person call where everyone sees everyone, each person receives 4 incoming streams. The only real question: who does the copying, the clients or a server?

So we pick an SFU, and I would say so immediately rather than dance around mesh. Now the server bandwidth, the number I promised.

• Each participant sends 1.6 Mbps in. The server forwards it to (N-1) others.
• For 400,000 meetings of 5: ingress is 2,000,000 * 1.6 Mbps = 3.2 Tbps.
• Egress is 2,000,000 * 4 * 1.6 = 12.8 Tbps.

Storage for recording.

• A recorded meeting at 1.5 Mbps is about 675 MB/hour of video.
• Say 5% of 400,000 concurrent meetings record, average 40 minutes: 20,000 meetings * 0.45 GB = 9 TB per peak-hour.
• Over a year that is petabytes, so recordings go to object storage (S3 or GCS) with lifecycle tiering to cold storage, never a database.

3Core APIs#

Signaling is a mix of plain HTTP for setup and a persistent WebSocket for the real-time exchange. The media never touches these; it flows over SRTP on UDP straight to the media server. I keep that separation explicit so nobody is tempted to shove video frames through the control plane.

POST /v1/meetings
Authorization: Bearer <jwt>
{
  "title": "Weekly sync",
  "mode": "interactive",
  "max_participants": 50,
  "record": true
}

201 Created
{
  "meeting_id": "mtg_8KzQ",
  "join_token": "ey...",
  "region": "us-east"
}

POST /v1/meetings/mtg_8KzQ/join
Authorization: Bearer <jwt>

200 OK
{
  "participant_id": "p_3fa",
  "ws_url": "wss://sig-us-east.example.com/mtg_8KzQ",
  "media_server": "sfu-71.us-east.example.com",
  "ice_servers": ["stun:stun.example.com:3478", "turn:turn.example.com:3478"]
}

The join response is doing real work:

• It hands back the signaling WebSocket URL.
• It names the specific SFU the allocator picked for this room.
• It lists the ICE servers the client needs for NAT traversal.
• The SFU is chosen at join time, not later. That is deliberate: a room pins to an SFU, and I want the client talking to the right box from the first packet.

Over the WebSocket, messages are small JSON frames. This is the SDP and ICE dance, the negotiation that sets up each peer connection:

{ "type": "offer", "from": "p_3fa", "sdp": "v=0..." }
{ "type": "answer", "from": "sfu", "sdp": "v=0..." }
{ "type": "ice-candidate", "from": "p_3fa", "candidate": "candidate:..." }
{ "type": "participant-joined", "participant_id": "p_9bc", "name": "Mia" }
{ "type": "track-published", "participant_id": "p_9bc", "track": "screen" }
{ "type": "mute", "participant_id": "p_3fa", "kind": "audio", "muted": true }

A detail worth stating out loud:

• In an SFU model the client negotiates with the SFU, not with every other peer. So offer goes client to SFU, answer comes back from the SFU.
• Events like participant-joined and track-published are broadcast so each client knows to subscribe to the new track.
• That is why the SFU can be the one point that sees and routes all media.

4High-level design#

The spine splits cleanly into control (who is in the room, auth, state), signaling (the WebSocket that negotiates connections), and media (the SFU fleet that moves packets).

Walking the diagram.

• A client hits the global load balancer, which routes by geography so a Tokyo user lands on Tokyo infrastructure.
• The Meeting API handles create and join, writes to the Meetings DB, and asks the SFU Allocator which media server has capacity in the right region.
• The client opens a Signaling WebSocket, and all signaling for a room is coordinated through Redis pub/sub so any signaling node can serve any participant.
• Media goes straight from client to the assigned SFU over SRTP on UDP.
• If a client is behind a symmetric NAT that blocks direct UDP, it falls back to a TURN relay that forwards the media to the SFU.
• The Recorder is a special headless participant that subscribes to all tracks and writes to object storage.

Why Redis pub/sub for signaling state.

• A room's participants might land on different signaling nodes, because the balancer picks a node per connection.
• When p_3fa publishes a new track, every other participant needs to know, and some are on other nodes.
• Redis gives us a per-room channel (room:{id}:events), so a message published by one node reaches whichever node holds each participant's socket.
• Without it, two people in the same meeting on different signaling nodes would never see each other's events.

5Data model#

Use the store that fits the access pattern. Meeting metadata and live participant state churn at completely different rates, so I do not force them into one database.

Meeting metadata is relational and lives in Postgres: low volume, needs consistency, and I want joins for the roster and recording history.

Participants track who is in the room over the lifetime of the meeting:

Recordings point at the durable artifact, not the bytes:

Live state lives in Redis, not Postgres, and that choice is load-bearing.

• It is just keys: room:{id}:participants is a set of participant IDs, and room:{id}:p:{pid} is a hash holding that participant's current tracks and SFU assignment.
• Each key carries a 30-second TTL refreshed by a heartbeat over the WebSocket.
• If a laptop sleeps or the socket dies, the key expires in 30 seconds and the roster corrects itself with no cleanup job.
• That self-healing property is exactly why high-churn live state does not belong in a relational store.

6Deep dives#

6.1 Simulcast and the bandwidth problem#

Remember that 12.8 Tbps egress number. The trick that makes SFUs affordable is simulcast.

How it works.

• Instead of sending one video stream, each client encodes the same camera into three resolutions at once: say 180p, 360p, and 720p, each at a different bitrate.
• The client uploads all three. The uplink can handle it because the lower layers are small, maybe 2 Mbps total for the set.
• Now the SFU chooses, per receiver, which layer to forward.

Why it cuts the bill so hard.

• In a 20-person grid where only the active speaker is large, the SFU sends 720p for the speaker and 180p thumbnails for everyone else.
• Most streams at any moment are tiny thumbnails, not full resolution, so egress drops by a large factor.
• If a receiver's downlink degrades, the SFU drops them to a lower layer without telling the sender to re-encode anything.
• The whole game: the sender encodes once, the server selects per receiver from RTCP feedback, nobody renegotiates, and the decision is a cheap forwarding choice, not an expensive transcode.

Simulcast versus SVC, the trade-off to name.

• SVC (scalable video coding) puts layered dependencies in one stream; the SFU just drops the top layers.
• SVC is more upload-efficient (one layered stream instead of three separate encodes) and switches more smoothly, because layers are dependent rather than independent key streams.
• The catch: hardware encoder support is spottier and it is harder to reason about when debugging.
• I ship simulcast first because every browser supports it, and keep SVC as an optimization for clients that advertise it (modern VP9 and AV1 stacks).
• Saying that trade-off out loud is the difference between sounding like you read a blog and sounding like you have shipped this.

6.2 Webinar fan-out: SFU cascading#

A 10,000-viewer webinar breaks a single SFU.

• One SFU box handles maybe a few thousand outbound streams before its NIC and CPU saturate. 10,000 is well past that.
• The answer is cascading: arrange SFUs in a tree.
• The presenter publishes to an origin SFU. The origin forwards to a layer of edge SFUs, each in a different region, and each edge fans out to the viewers near it.
• One upload from the presenter becomes a tree instead of 10,000 direct forwards hammering one box. Each edge handles only its local slice, say a couple thousand viewers.

This also fixes latency for global viewers.

• An Asian viewer connects to an Asian edge, not an origin in Virginia, so no trans-Pacific round trip on every packet.
• The cost is one extra hop of latency per cascade level, fine for webinar viewers who tolerate 2 to 5 seconds.
• The origin-to-edge link is a single server-to-server stream per layer, cheap compared to fanning out to individuals.

For the truly passive tail, go further.

• Transcode the stream to HLS or LL-HLS and serve it through a CDN.
• That trades 2 to 5 seconds of latency for near-infinite scale and pennies per viewer, because the CDN already solved global fan-out and you pay no SFU egress at all.
• A good design uses WebRTC for the interactive core (presenters and anyone who might speak) and HLS for the broadcast tail, and switches a viewer up to WebRTC only if they raise their hand to speak.
• That hybrid is how real products serve a town hall to 50,000 people without melting.

6.3 Handling a participant's bad network#

This is the signature hard part, so I want to be precise. The real-time problem: one person's bad WiFi should not freeze the room.

The mechanism.

• WebRTC congestion control (the client stack plus the SFU) estimates available bandwidth from RTCP feedback and packet arrival timing, using transport-wide congestion control.

• When loss climbs past a threshold, three things happen in order.

• Forward error correction and retransmits. For audio, send redundant packets (RED, redundant audio data) so a lost packet reconstructs without a retransmit round trip. Audio matters more than video: a frozen face is tolerable, choppy audio is not, because people stop understanding each other the moment audio breaks.

• Layer downgrade. The SFU drops that one receiver to a lower simulcast layer (720p to 360p to 180p), cutting required bandwidth by a big step with no re-encode.
• Video pause, audio survives. If it gets truly bad, say sustained 10% loss, the SFU stops forwarding video to that receiver entirely and keeps audio flowing. The person stays in the conversation, and their video resumes when the network recovers.

The principle to land hard.

• All three adaptations happen on the receive path, per receiver, inside the SFU, without ever touching the sender.
• The sender keeps publishing its three clean layers to a healthy SFU link regardless of what any receiver suffers.
• This isolation is why SFUs beat mesh for reliability, not just bandwidth.
• In mesh, a receiver with bad network sends loss reports back to the sender, and the sender might lower its encode for everyone, so one bad laptop drags down the whole call.
• In an SFU, one bad network is contained to one leg of the tree and nobody else notices.

7Bottlenecks and trade-offs#

The SFU is stateful, and that is the pain.

• Unlike a stateless web server, you cannot round-robin participants across SFUs. Everyone in a room should sit on the same SFU (or a cascade of them) so forwarding stays local.
• That makes the allocator a real service: it tracks per-SFU CPU, bandwidth, and participant count, and pins a room to an SFU at creation time.
• When an SFU dies, every call on it drops at once. Clients must detect the media stall (no RTP for a second or two) and re-signal to a new SFU, rebuilding peer connections from scratch.
• Recovery in a couple of seconds is the realistic target, not zero. I would be honest about that rather than pretend live media fails over instantly.

Media servers are CPU and NIC bound, not memory bound.

• Forwarding SRTP is packet shuffling plus occasional re-keying and RTCP processing, so you provision for network throughput and context switches, not gigabytes of RAM.
• Cap a single SFU at a conservative stream count and scale horizontally, because a saturated SFU degrades everyone on it at the same moment.
• Better to run boxes at 60% and add more than to pack them to 95% and watch a whole region of calls stutter together.

TURN is expensive, so minimize it.

• Relayed media doubles your server bandwidth cost (packet comes in to TURN, goes back out to the SFU) and adds a hop of latency.
• Maybe 15 to 20% of connections need TURN because of strict symmetric NATs and corporate firewalls that block UDP.
• You cannot avoid it entirely, so keep TURN servers regional and let ICE prefer a direct or STUN-assisted path, falling back to TURN only when nothing else connects.

Recording decryption is a security surface.

• SRTP means the server relays encrypted media by default, which is good.
• To record, the recorder must decrypt the media to mux it into a playable file. So the recorder is a trusted participant that holds keys, runs in a locked-down path, and re-encrypts the file at rest.
• If a customer demands true end-to-end encryption where keys never touch the server, you lose server-side recording and you lose the SFU's ability to inspect frames for layer selection, because the SFU can only forward opaque encrypted packets.
• That is why E2EE meetings are a separate, more limited mode (no cloud recording, reduced layer flexibility). I state that trade honestly rather than hand-wave perfect encryption and full features at once.

Global routing versus room locality.

• We want each user on a nearby SFU for latency, but everyone in one room on one SFU for simplicity. For a call spanning three continents, those two goals fight directly.
• The resolution is cascading even for small calls when participants are geographically spread: a regional SFU per cluster, meshed server to server over the backbone. More complexity, better latency.
• I would start with single-SFU-per-room and add cross-region cascading only when the data shows intercontinental calls suffering, because premature cascading is a lot of moving parts for a problem most calls do not have.

Signaling is the easy part, so do not over-engineer it.

• At 333K messages/sec, a modest WebSocket fleet with Redis pub/sub handles it with room to spare.
• The temptation is to pour design effort here because WebSockets and Redis feel familiar and safe. Resist it.
• Spend your interview minutes on the media plane, because that is where the 12.8 Tbps lives and that is what the interviewer is actually testing.

This problem is won in the numbers, and the number is 12.8 Tbps of egress.

8Summary#

The decisions that matter:

• SFU, not mesh. The bandwidth math kills peer-to-peer past 4 people (6.4 Mbps upload each). The SFU absorbs the fan-out so clients upload once.
• Three planes: control (Meeting API plus Postgres), signaling (WebSocket plus Redis pub/sub), and media (SFU fleet). Keep them separate because they scale and fail differently.
• Simulcast lets the server pick a resolution per receiver, which makes the 12.8 Tbps egress bill survivable and bad networks tolerable, with no re-encode.
• Cascading SFUs turn one upload into a tree for webinars and globally spread calls, and the passive tail drops to CDN-delivered HLS for near-infinite cheap scale.
• Receiver-side adaptation inside the SFU (FEC, then layer downgrade, then video pause) isolates one person's bad network from the room and prioritizes audio over video.
• State is the hard constraint. Rooms pin to SFUs, SFU death means a couple of seconds of client reconnection, and the allocator is a first-class service.

For related real-time and media problems, see our guides on designing a live streaming platform, a chat system, and a push notification service.