Open almost any video-infrastructure doc and you’ll hit three letters with no explanation: SFU. So — what is an SFU, and why did an entire industry quietly agree to build on one? An SFU is the routing layer that sits in the middle of a modern video call, deciding which streams go where. It isn’t glamorous, but it’s the single choice that determines whether your call scales past a handful of people, how much it costs to run, and how fast it feels. Here’s what it does — and why mesh and MCU lost.
The problem every video call has to solve
Every real-time call is really a distribution problem. Each participant produces media — their camera and mic — and needs to receive everyone else’s. With two people that’s trivial. With ten, each person’s outgoing stream has to reach nine others, and each person has to receive nine incoming streams. The question that decides everything is: who does the work of copying and routing all that media? There are only three answers, and the one you pick sets the ceiling on how many people fit in the room, how good their devices have to be, and how much latency everyone eats.
Start with the simplest answer, and watch it fall apart.
Three ways to route video: mesh, MCU, and SFU
The first option is mesh: every participant sends their stream directly to every other participant, peer to peer, with no server in the media path. Elegant for two people. But each person now uploads their video once per other participant — five people means each client encodes and uploads four copies of itself. Uplink is the scarcest resource on most connections, so mesh collapses somewhere around four or five participants.
The second is the MCU — a Multipoint Control Unit. Every participant sends one stream to a central server; the server decodes them all, mixes them into a single composited video, re-encodes it, and sends one stream back to each person. The client’s job is trivial — send one, receive one — but the server is doing enormous work. Decoding and re-encoding every stream in real time is CPU-expensive, and the mixing step adds latency. MCUs scale in client simplicity, not in server cost.
The third is the SFU — and it’s the compromise that won.
| Architecture | How it routes media | Client cost | Server cost | Latency | Practical ceiling |
|---|---|---|---|---|---|
| Mesh (P2P) | Everyone sends to everyone directly | High (N‑1 uploads) | None | Lowest | ~4‑5 people |
| MCU | Server mixes all streams into one | Lowest (1 up / 1 down) | Very high (decode + re-encode) | Higher (mixing) | Large, but costly |
| SFU | Server forwards streams selectively | Moderate (1 up / N down) | Moderate (no re-encode) | Low | Large and affordable |
To see why the SFU is the compromise everyone settled on, look at what it actually does with a stream.
What an SFU actually does
SFU stands for Selective Forwarding Unit, and the whole design is in the middle word. Each participant sends their media to the SFU once. The SFU does not decode it, mix it, or re-encode it — it simply forwards each incoming stream to the participants who should receive it, selectively. A router for media, not a mixer.
That “no re-encoding” is the entire trick. Because the SFU never decodes and re-encodes video, it avoids the MCU’s crushing CPU cost and its added latency — the frames pass through nearly untouched. And because each client only uploads once, to the SFU, instead of once per peer, it avoids mesh’s uplink problem. The client uploads one stream and downloads several; the server forwards without transcoding. Everyone’s cost stays reasonable.
That single decision — forward, don’t mix — is why the SFU displaced both alternatives.
Why the SFU won for real-time
Line these up and the SFU wins on the axes that matter for live video. Latency: forwarding is faster than mixing, so the SFU stays close to peer-to-peer even with a server in the path. Cost: no transcoding means one server handles far more participants than an MCU — though a single SFU still has a ceiling. Quality and flexibility: because streams aren’t flattened into one mixed picture, each receiver keeps every remote stream as a separate, full-quality track — so the client can lay them out however it likes, show the speaker large and others small, or drop streams it doesn’t need. A mixed MCU output can’t be un-mixed. Scale follows from the same economics: forwarding is cheap enough that large calls stay affordable, which is why real-time platforms standardized on it.
But the basic idea — forward every stream to everyone — would still drown a client in a big call. What makes a production SFU good is what it does on top.
The clever parts: simulcast, bandwidth estimation, active speaker
A naive SFU forwards every stream to everyone at full quality, and a participant on a weak connection would be buried. Real SFUs are smarter in three ways.
Simulcast is the big one. Instead of sending a single stream, each participant's client encodes their video at several quality layers at once — say high, medium, and low — and sends all of them to the SFU, which is cheap for the server and not free for the sender's uplink. The SFU then picks, per receiver, which layer to forward based on that receiver’s screen size and connection. Someone shown in a tiny tile gets the low layer; someone pinned full-screen gets the high one. The senders publish options; the SFU chooses.
Bandwidth estimation is the second. The SFU continuously watches each connection for signs of congestion and adapts what it forwards — stepping down to a lower simulcast layer before packets start dropping, rather than after.
Active speaker detection is the third. The SFU can tell who’s currently talking and prioritize accordingly — forwarding the speaker’s stream at higher quality, or driving the “who’s on screen” logic the client renders.
All of which makes the SFU sound like the answer to everything. It isn’t — and being honest about that is the point.
When mesh or MCU still make sense
The SFU won the general case, not every case. For a one-to-one call, mesh is often the right choice: two peers, no server in the media path, the lowest possible latency and cost, and better privacy because the media never touches a middlebox. Adding an SFU there buys you nothing.
The MCU still earns its place whenever you genuinely need a single composited output. Recording a call as one flat video file, broadcasting to a “dumb” endpoint that can only accept one stream, or bridging into telephony and SIP systems that expect a single mixed feed — those are mixing problems, and mixing is exactly what an MCU does. Some platforms even run an SFU for the live call and an MCU-style component only for the recording. The architectures aren’t rivals so much as tools for different shapes of problem.
Which brings the whole picture together.
The Bottom Line
An SFU — a Selective Forwarding Unit — is the server that routes real-time video by forwarding streams selectively instead of mixing or re-encoding them. It beat mesh, which can’t get past a handful of participants, and it beat the MCU, whose constant transcoding is slow and expensive. By forwarding rather than mixing, it keeps latency low, cost manageable, and every remote stream independent — and with simulcast and bandwidth estimation on top, it adapts to each participant’s device and connection. That’s why, whatever platform you’re evaluating, you’ll find an SFU at the center of it. Open-source implementations like mediasoup, Janus, and LiveKit made the pattern ubiquitous; commercial platforms build on the same idea. When someone says a video product is “SFU-based,” this is what they mean.
What’s Next?
Now that you know what carries the media, see what controls the session: What Is a Video Room API — and How Is It Different from a Video SDK? explains the server-side layer that creates and governs the rooms an SFU serves. If you’re mapping the whole stack for a real deployment, Enterprise Video Conferencing Architecture walks the layers end to end. And Video Call vs Conferencing vs Chat SDK helps you pick the right client library to sit on top.
Frequently Asked Questions
What is an SFU?
An SFU (Selective Forwarding Unit) is a server that routes real-time video by forwarding each participant’s stream selectively to the others, without decoding, mixing, or re-encoding it. It sits in the middle of a video call and decides which streams reach whom — which is what lets modern calls scale beyond a few people.
What’s the difference between an SFU and an MCU?
An MCU mixes all participants’ streams into a single composited video and re-encodes it; an SFU forwards streams individually without re-encoding. The MCU makes the client’s job trivial but costs far more server CPU and adds latency. The SFU keeps latency low and each remote stream independent, which is why it’s the default for live calls.
Why do video platforms use SFUs instead of peer-to-peer?
Because mesh (peer-to-peer) doesn’t scale. In a mesh call, every participant uploads a copy of their video to every other participant, and upload bandwidth runs out around four or five people. An SFU means each participant uploads once, to the server, which then forwards — removing the uplink bottleneck while keeping latency low.
Does an SFU decode or re-encode video?
No. That’s the defining feature of an SFU: it forwards streams without decoding or re-encoding them. Avoiding transcoding is what keeps its CPU cost and latency low compared with an MCU, which decodes and re-encodes every stream in order to mix them.
Is WebRTC an SFU?
No — they’re different layers. WebRTC is the browser technology that lets clients capture and transport real-time media; an SFU is a server role that routes that media between participants. A WebRTC call with more than a couple of people almost always routes through an SFU, but WebRTC itself isn’t one.
Does Samvyo use an SFU architecture?
Yes. Samvyo is based on an SFU architecture, which is why it can carry large real-time calls at low latency without the transcoding cost of a mixed approach. It offers the same embeddable SDKs a CPaaS provider gives you, plus on-premise and managed-cloud deployment and white-label depth — and for teams that need it, the media path, TURN, and recording can stay under their own control while the platform remains resilient by design.
