What is WebRTC? How real-time voice and video work, and when to buy it instead of building it

A customer clicks "talk to us" on your website and speaks to an agent, human or AI, without installing anything. Behind that button is often WebRTC. It is easy to treat it as a feature to switch on, and just as easy to underestimate how many separate problems it solves: finding the other party across firewalls, encrypting the audio, and keeping a conversation intelligible on a bad Wi-Fi connection.

What is WebRTC?

WebRTC (Web Real-Time Communication) is a set of open standards that lets applications capture audio and video and exchange them, plus arbitrary data, directly between browsers and apps in real time. MDN describes it as enabling teleconferencing peer-to-peer, without users installing plug-ins, and often without an intermediary server. The IETF's overview document states the design intent: the media path goes directly between the browsers.

For a business, the practical meaning is a call that starts in a web page, a mobile app or a server process, with a delay short enough for natural conversation. That is why voice AI assistants, video support, remote inspections, telehealth and in-browser softphones are built on it.

The standard has a third part, data channels, which carry arbitrary data between peers over the same connection. RFC 8825 requires every WebRTC endpoint to implement the specifications for them. A voice agent can use one to send transcripts or commands while the audio flows separately.

Peer-to-peer, SFU or MCU: which architecture fits?

Choose peer-to-peer for one-to-one calls, a selective forwarding unit (SFU) for group calls and AI agents, and a multipoint control unit (MCU) only when endpoints cannot handle several streams.

The IETF's RTP topologies document supports the distinctions. A media-mixing unit must decode the incoming streams and produce a mix, which consumes large amounts of processing, degrades quality by decoding and re-encoding, and adds latency. A selective forwarding middlebox, by contrast, has extremely low delay, does no signal processing, and can scale to a large number of concurrent users.

How does a WebRTC call get set up?

Setting up a call takes three steps: the two sides agree on what to send, find a network path, and then start sending. The standard does not do the first step for you, which surprises most teams.

Signaling and SDP offer and answer

Signaling is the exchange of the information both sides need before media can flow. WebRTC leaves it to the application. RFC 8825 puts client-server signaling outside the scope of the protocol suite, and the JSEP specification says the application must call the right APIs at the right times and carry the descriptions over whatever signaling channel it chooses. Many applications use a WebSocket, which WebRTC for the Curious says is common but not required. We compare the options in WebSockets vs SSE vs WebRTC.

What travels is a session description in the SDP format, and it works as an offer and an answer. The caller runs createOffer, applies it locally with setLocalDescription, and sends it. The other side applies it with setRemoteDescription, runs createAnswer, and sends the reply. The description lists codecs, the candidate network addresses where the party can be reached, and a fingerprint, which is a hash of the certificate the peer will use for encryption. The answerer can reject codecs it does not support.

ICE, STUN and TURN

Most devices sit behind a network address translator (NAT), a router feature that hides private addresses, which makes it hard for two of them to find each other. RFC 8445 describes ICE, the process for finding a path: each side exchanges a multiplicity of IP addresses and ports, and then tests them with peer-to-peer connectivity checks. The candidates come in three kinds:

• Host: an address on the device's own network interface.
• Server-reflexive: the public address a NAT assigned, discovered by asking a STUN server. STUN is a simple protocol where a server outside the NAT tells a device which address it appears to have.
• Relayed: an address allocated on a TURN server.

TURN is the last resort. RFC 8656 explains that if a host is behind a NAT, direct communication can be impossible, and it needs an intermediate node that acts as a relay. Once the checks find a working pair, one side nominates it, and media starts flowing. Candidates can also be sent as they are found, a technique called trickle ICE, so the call starts sooner.

The business consequence is that TURN relays carry real audio and video. A call that cannot go direct sends its media through your relay, so its bandwidth bill follows call minutes, and its capacity needs planning.

Why does WebRTC use UDP, and how is it secured?

WebRTC prefers UDP because real-time media loses value when it arrives late, and UDP does not hold back new packets to retransmit old ones. UDP is a transport protocol that sends packets without guaranteeing delivery or order. RFC 8836 states the requirement: interactive media needs delivery within a very limited window, with no more delay than hundreds of milliseconds end to end, and data delivered late is commonly useless. The same document notes that such applications may tolerate some packet loss.

RFC 8835 treats UDP as the baseline and also requires fallbacks: TURN over TCP and over TLS for firewalls that block all UDP, and ICE-TCP candidates. Those fallbacks work, but TCP carries a cost: head of line blocking, where one lost packet stalls everything behind it, which the RFC cites as a reason to avoid TCP between a TURN server and the peer. When a corporate network forces a call onto TCP, expect worse audio.

Encryption is not optional. RFC 8834 prohibits plain RTP and requires the secure RTP profile to protect all RTP and RTCP packets, with keys negotiated through DTLS. DTLS is the TLS handshake adapted for UDP, and SRTP is the encrypted version of the real-time media protocol. The certificate fingerprint exchanged during signaling is what ties the encrypted channel to the party you intended to reach. A malicious signaling server can still redirect a call, so the integrity of signaling and call authorization remain your responsibility.

How does WebRTC keep audio and video usable on a poor network?

It combines efficient codecs, a small playout buffer, several ways to recover from loss, and a sending rate that adapts to the path.

Codecs

The IETF requires every WebRTC endpoint to implement the Opus audio codec and G.711. RFC 6716 says Opus scales from 6 kbit/s narrowband speech to 510 kbit/s stereo music, supports frame sizes from 2.5 to 60 ms, and includes in-band forward error correction as a defense against packet loss. For video, RFC 7742 requires WebRTC browsers to implement VP8 and H.264 Constrained Baseline. The RFC adds that the working group will watch the licensing of both codecs, to promote non-royalty-bearing options.

Jitter buffers and concealment

Packets from the same call arrive at uneven intervals, a variation called jitter. A jitter buffer holds incoming audio or video for a short, adjustable time so that playback is smooth. The W3C statistics specification exposes how long samples or frames spend in it, along with a count of concealed samples, which are samples replaced by locally synthesized ones before being played. A longer buffer smooths more jitter and adds delay, so implementations balance the two.

Loss recovery

When packets go missing, the receiver can ask for a resend with a negative acknowledgment (NACK), or the sender can include redundancy. RFC 8834 describes both: NACK-based retransmission, and forward error correction, which it says gives effective protection against some packet loss at the cost of steady bandwidth overhead and increased play-out delay. WebRTC for the Curious adds that NACKs are much more bandwidth efficient than resending a whole video frame, and that FEC has lower latency than NACK when loss is steady.

Congestion control

The sender continually estimates how much the path can carry and adjusts. RFC 8836 requires that the algorithm provide as-low-as-possible-delay transit, be fair to other flows, and not starve competing TCP traffic. One widely used algorithm, Google Congestion Control, is described in WebRTC for the Curious as reducing its estimate when loss is above 10 percent, raising it when loss is below 2 percent, and watching delay growth to spot congestion before packets drop.

Why is it efficient?

Peer-to-peer media avoids a server in the middle, Opus can carry speech at as little as 6 kbit/s, and the stack trades some quality for lower delay. WebRTC for the Curious puts the trade-off plainly: the more latency you tolerate, the higher quality you can expect. For a conversation, the right setting is the lowest latency that still sounds clear.

How do voice AI agents and customer service video use WebRTC?

A browser or phone app sends microphone audio to an AI voice agent and plays the agent's speech back, over one WebRTC connection. OpenAI's Realtime API guide, as of October 2026, recommends WebRTC rather than WebSockets when a browser or mobile client connects, for more consistent performance. Server-side voice agents can still use WebSockets. In its design, the application server holds the API key and hands the browser a short-lived ephemeral key, or relays the browser's session description, so the secret never reaches the client. Audio flows on media tracks, and a data channel carries transcript updates and session commands.

That pattern matters for customer service. Audio goes straight to the model provider on a path built for low delay, while your server keeps control of identity, permissions and the call record. Human handoff can add a person to the same call room.

Illustrative example: Consider a regional insurer that wants a voice assistant on its claims web page to collect the first notice of loss and then transfer to an adjuster. The page asks the insurer's server for a short-lived key, connects by WebRTC, and the assistant gathers policy number, date and description while the transcript streams into the claims system. Policy changes and payouts are not performed by the assistant. When the caller asks for a person, the server adds an adjuster to the room, so the caller does not repeat the story. The design question that decides success is not the audio transport but what the assistant may do alone and how the transcript is stored.

Should you run TURN and SFU servers yourself or use a managed provider?

Use a managed provider unless voice and video are your product or a rule prevents it. Running the media layer is a distinct operations job.

An example of the operating surface comes from LiveKit's self-hosting documentation. It lists a UDP port range that must be exposed to advertise WebRTC host candidates, a TCP port used when clients cannot connect through UDP such as behind VPNs or corporate firewalls, a TURN over TLS port, and an embedded TURN server on another port. It adds that, in cloud environments, the ports have to be opened in the firewall. That is before capacity planning, certificate renewal, regional placement and quality monitoring.

Weigh these factors:

1. Volume. Self-hosting pays only at high, steady minutes, and most mid-sized companies are nowhere near it.
2. Regions. Media servers should be near callers. One region adds delay for everyone else.
3. Compliance. Call recordings and transcripts may be regulated data. Choose where they are stored and who processes them.
4. People. Someone needs to own call quality at 2 a.m. If no one will, buy.
5. Lock-in. Keep your signaling, authentication and call records on your side, so changing providers means changing the media layer only.

How Kastling approaches this

Kastling would start with the conversation, not the transport: who is talking to whom, what a good call looks like, what the AI may do on its own, and who reviews the outcomes. It would then connect the voice or video layer to the systems the business already runs, such as the CRM, claims or ticketing system, with a named person approving consequential actions. For most companies of this size the plan would favor a managed real-time provider and keep call authorization, recordings and transcripts in the company's own AWS or Azure environment. Testing would include poor networks and blocked UDP, not only a clean office connection. This work sits within AI & Cloud Infrastructure, and the related question of keeping a model's output grounded is covered in RAG vs fine-tuning.