Moving a Rust WebRTC SFU to thread-per-core: 70ms → 10ms P99.99 latency

We recently migrated our open-source Rust WebRTC SFU server from Tokio's default multi-threaded work-stealing model to a thread-per-core (TPC) architecture, reducing P99.99 end-to-end latency from a spiky 70ms to a stable 10ms, while increasing system capacity by 25%.

WebRTC SFU Overview

Think of a WebRTC Selective Forwarding Unit (SFU) as a high-performance, low-latency router for real-time media. Acting like a pub/sub broker, it takes in video and data streams from a publisher and forwards them to subscribers using payload-specific transport optimizations. It doesn't do heavy server-side transcoding. It's the core technology that makes massive, Zoom-like video apps work.

Early Design

PulseBeam is a standalone WebRTC SFU server rather than a library. The first version of our WebRTC SFU was built in Rust on Tokio's multi-threaded scheduler. We used the most intuitive abstraction for the problem: each participant, room, controller, and API got its own task, and these tasks talked to each other over channels, following the CSP concurrency model. That also let us avoid the traditional mutex-based locking that has burned me before.

It worked well under low load. But once we ran a benchmark simulating a fully interactive 4-person conference on 4 CPU cores, cracks showed. WebRTC stats revealed growing, unstable inbound-rtp jitter, which dragged down the congestion control bandwidth estimate and caused stutter (more on this in the Benchmark section below).

CPU usage sat at 80% with no other heavy processes running, and the flamegraph looked unremarkable. Tokio's metrics told a different story. Task steal counts were high.

Digging into Tokio scheduler internals (https://tokio.rs/blog/2019-10-scheduler) pointed to the real culprit. We had spawned too many async tasks in the hot path.

This hurts latency because every channel send and receive acts as an async yield point where the executor can suspend the current task to run something else. Under the work stealing scheduler in Tokio, that alternative task can be a packet from a different connection that just arrived. This new packet is not more urgent, it is simply newer. Consequently, a packet already most of the way through its pipeline can be set aside for one that has not even started. Being closer to completion does not earn priority. This creates an inversion where the overall work order is determined by scheduling luck rather than how close a packet is to leaving the system.

What we generally want is:

receive a packet from a socket → process it → send it out over a socket

We want to avoid picking up new work before the current packet is fully processed, to avoid that priority inversion.

Each channel hop isn't free, either. Unlike HTTP requests, WebRTC packets are small, roughly 500 to 1,200 bytes, under typical MTU size, so per-packet processing overhead adds up fast.

Since we’re already using str0m, we can easily swap our concurrency model without rewriting the entire media stack. It’s a fantastic sans-I/O WebRTC library that doesn't force any assumptions on your threading or time management.

Removing Unnecessary Task Indirection

To flatten those tail-latency spikes, we removed the async channels entirely. Now, when a packet arrives from the socket, the SFU processes it synchronously all the way through to the egress socket.

This does reduce concurrency, since it pins a connection's whole pipeline to a single CPU core. In exchange, it removes the unpredictable scheduling overhead of context-switching at channel boundaries, giving us a predictable execution path.

WebRTC SFU workloads are stream-based and predictable compared to typical HTTP servers, which makes them a good match for a thread-per-core architecture. Statically pre-load-balancing each connection to a specific core means we eliminate cross-core contention and get higher CPU cache hit rates.

We aren't alone in this approach. High-performance streaming systems like Redpanda and Iggy, and databases like ScyllaDB, rely on this same shard-per-core model to keep latency predictable.

Other SFUs make different architectural trade-offs based on their own goals:

• LiveKit keeps its pipeline highly synchronous to minimize overhead. It relies on Go's work-stealing scheduler to maximize CPU utilization out of the box, but distributing work across a shared thread pool means managing shared state via locks, which can introduce scheduling jitter under load.
• mediasoup uses a process-per-core model in C++. It achieves excellent core isolation and cache locality by eliminating shared state between workers entirely, but managing separate OS processes and IPC adds operational and memory overhead.

None of these approaches is perfect, including ours. We chose thread-per-core because it best fits our goal of minimizing scheduling jitter. It gives us a lean, lock-free profile without multi-process plumbing. The trade-off is the classic downsides of thread-per-core, more complex cross-core communication, harder load balancing, and the risk of one hot connection saturating an entire core if it isn't carefully managed.

Moving to Multi-Core

We ended up settling on a Tokio LocalRuntime per thread. This isn't the ideal fit for a thread-per-core architecture. Libraries like compio, glommio, and monoio are built on io_uring and purpose-designed for exactly this model, but Tokio is mature and has a large ecosystem, and we wanted to avoid spending time chasing bugs at the async runtime level.

The control thread handles SDP negotiation, bookkeeping of node state, and controls traffic rate and assignment. It will delay or reject new traffic if the data threads are at maximum capacity.

We reintroduced async channels to communicate between shards in a mesh, but in a much more controlled way. There is at most one channel hop, and zero in the best case.

Benchmark

The client is written in Rust. Each room has 4 fully interactive participants, each sending the others a 360p@30 video stream at roughly 400Kbps. Every participant publishes from the same pre-recorded H264 video, so no encoding or decoding is involved. Simulcast is disabled to reduce variability. A new room spawns every second, and participant joins are spread uniformly across a 15-second window to mimic realistic human arrival patterns. Once the system reaches 120 rooms, new rooms stop spawning. At this peak, it sustains 480 concurrent participants and drives ~80% CPU usage under the work-stealing scheduler. The client captures two key metrics, ICE round-trip time and end-to-end latency.

The server ran on 4 isolated i9-12900HK P-cores, underclocked to 1.8GHz with hyper-threading and turbo boost disabled, to approximate the kind of lower-tier hardware a cost-conscious deployment would actually run on. CPU isolation was configured with the following kernel arguments: isolcpus=1-4 nohz_full=1-4 rcu_nocbs=1-4

The benchmarks were executed using the following versions:

• Work-stealing: Commit 70018b0
• Thread-per-core: Commit 83aa316
• Client: Commit 83aa316

ICE Round Trip Time

Each participant implements full ICE. The server periodically receives a STUN binding request and responds with a STUN binding response. There's no fan-out involved. Once the sender receives the response, it derives the round-trip time (RTT).

End-to-End Latency

For every frame, the publisher injects the current monotonic clock into RTP packets via the abs-capture-time header extension. Since the subscriber lives on the same machine as the publisher, we can derive end-to-end latency accurately as receive time − frame's abs-capture-time.

Bonus: Thread-per-Core at 80% CPU

The thread-per-core P99.99 stays much closer to P50 than the work stealing version, and it also has more headroom. Pushing the setup to 150 rooms brings CPU utilization to 80%, which is about 25% more traffic than the runs above, while keeping latency just as stable. At this point the server becomes NIC limited rather than CPU limited.

Same metrics at 80% CPU:

Summary

Async tasks are great for I/O-bound workloads, and work-stealing is a solid way to balance them. For an SFU, thread-per-core fits the workload better, and the payoff shows up directly in call quality. Stable tail latency keeps jitter low, which keeps the congestion controller's bandwidth estimate stable instead of oscillating. In practice that means fewer stutters, fewer quality drops, and a more consistent stream for every participant, not just on average but at the tail, where bad calls actually happen.

There's plenty more to explore in a thread-per-core architecture:

• io_uring to avoid syscalls and memory copies
• eBPF RSS steering to avoid cross-core packets
• OOP vs ECS for better CPU cache behavior

Those are topics for another time.