LiveKit ESP32 SDK: ESP32-S31 Port
Ported LiveKit’s realtime audio/video SDK to the new ESP32-S31 chip, with board support for two Espressif dev boards, echo cancellation, and a host-side test harness.
REFERENCE HARDWARE — Espressif ESP32-S31-Korvo-1, one of the two boards supported by the port. Bench photos of both boards to follow.
Overview
LiveKit publishes an ESP32 SDK that lets a microcontroller join a LiveKit room over WebRTC: bidirectional Opus audio, H.264 video publishing, data channels, RPC, and a voice-AI agent example. Upstream supports the ESP32-S3 and ESP32-P4.
This project extended the SDK to Espressif’s ESP32-S31, a newer dual-core RISC-V chip that is only available as a preview target in ESP-IDF 6.1. The work went from a bare-chip build to two fully supported development boards with full-duplex audio, acoustic echo cancellation, camera video publishing, and a validated conversational voice-agent loop.
The work is being contributed upstream to LiveKit as three pull requests: dependency and ESP-IDF 6 compatibility, generic transport fixes, and ESP32-S31 board support. Base version: LiveKit ESP32 SDK v0.3.10 (upstream, June 2026).
Supported boards
A reusable ESP-IDF component, s31_board, carries a profile for each board and handles I2C, I2S, codec bring-up, amplifier control, camera registration, and RGB LED control.
| BOARD | AUDIO CODEC | MIC | SPEAKER | LED | CAMERA |
|---|---|---|---|---|---|
| ESP32-S31-Korvo-1 | ES8389, full duplex stereo | Onboard | Left/right connectors | WS2812 on GPIO 37 | Onboard OV3660 |
| ESP32-S31-Function-CoreBoard-1 | ES8311, mono | Onboard electret | J9 via NS4150B amp | WS2812 on GPIO 60 | None onboard |
The Function-CoreBoard profile gives the ES8311 driver separate ADC and DAC interfaces so microphone clocks stay running when playback closes. Both paths run the hardware bus at 48 kHz stereo with APLL clocking and let the media pipeline convert to and from Opus formats.
What was delivered
Buffered audio renderer
A dedicated playback path with a 64 KiB ring buffer, a writer task, 200 ms PCM prefill, re-priming on empty queues, and clean end-of-stream handling. This fixed choppy and truncated playback on the S31 media stack.
Standalone acoustic echo cancellation
An AEC capture source built on Espressif’s esp-sr library (VOIP_HIGH_PERF mode with aggressive nonlinear processing). It captures the microphone and the codec’s digital DAC reference in separate I2S slots at 48 kHz, filters each to 16 kHz, and publishes mono echo-cancelled audio. Adaptive filter state lives in PSRAM to keep internal RAM free for DMA and PCM, and DMA capacity was raised to 120 ms per direction to ride out capture-processing stalls. AEC is on by default in the examples that play audio while capturing and opt-in elsewhere.
Four working examples on S31
- minimal — bidirectional Opus audio on either board.
- voice_agent — conversational AI agent with CPU temperature reporting and board-aware RGB LED control. The Python agent queries the firmware for its board identity before greeting so it only offers peripherals that exist. Added a LiveKit Inference backend (Deepgram Nova-3 STT, GPT-4.1 mini, Cartesia Sonic-3 TTS) alongside the existing OpenAI Realtime backend.
- minimal_video — audio plus 240 × 240 software H.264 video from the Korvo-1’s onboard camera, with the encoder pinned to core 1 so it runs alongside AEC capture on core 0.
- custom_hardware_s31 — a porting template that initializes codec, I2S, and amplifier by hand for both boards, with optional boot-time speaker sweep and microphone level diagnostics.
Core SDK fixes (all chips, not only S31)
- Release the single audio subscription when its track or participant disappears, so replacement audio can be selected.
- Create subscriber SCTP data channels only when subscriber-primary negotiation requires them.
- Fetch Sandbox token-server credentials over verified HTTPS and fail the build if the certificate bundle is missing.
- Upgrade to the released Espressif media stack (esp_capture 1.0.2, av_render, esp_peer 1.5.x) and keep ESP-IDF 5.4, 5.5, and 6.x all building, including Windows response-file support and IDF 6 cJSON selection.
ESP-IDF workaround
ESP-IDF 6.1.0 has a dual-core S31 cache fault (MCAUSE 0x19) during flash operations, tracked upstream as espressif/esp-idf issue 18948 ↗. The repo ships a patch that disables branch prediction on the parked core, mirroring Espressif’s proposed fix, plus instructions for applying it safely.
Host-side interoperability harness
A Node.js tool (about 600 lines) that joins the room from a PC and exercises the device end to end: RPC, reliable and lossy data packets, text and byte streams, microphone capture and decode, speaker tone playback with configurable frequency, level, and channel, full-duplex capture-while-rendering, track-lifecycle stress, sustained audio soak, Wi-Fi outage recovery, LED cycling, and before/after heap and PSRAM accounting.
Host unit tests
Two CMake test suites that compile the audio renderer and AEC lifecycle code on the host against stubbed FreeRTOS and codec headers, so the state machines can be checked without hardware.
Documentation
A setup and status guide covering requirements, the IDF patch, per-board build commands, configuration, validation scope, and known limitations. README updates for the top-level repo, the board component, each example, the patch, and the test tools.
Validation and limits
- Both boards: Wi-Fi, HTTPS token acquisition, room join, leave and rejoin, RPC, data packets, data streams, and bidirectional Opus audio pass.
- Both boards: voice-agent conversations longer than ten minutes with AEC on, no unsolicited self-replies and no watchdog resets.
- Both boards: overlapping-speech listening tests pass with AEC on.
- Korvo-1: continuous audio plus video run of about 630 seconds with no watchdog failures.
- Korvo-1 video with AEC: about 6 to 7 fps measured at an external subscriber. The configured 10 fps target was not reached; further tuning is deferred.
- Twelve publish/unpublish track-lifecycle iterations verified against esp_peer 1.5.4.
- Video subscription and display rendering are outside the port.
- One remote audio track is subscribed at a time; no mixing.
- Function-CoreBoard audio is mono and video needs an external camera and adapter.
- Independent stereo echo cancellation is not implemented.
- Occasional playback-onset pops; pop suppression not implemented.
- Requires ESP-IDF 6.1 with preview targets and the branch-predictor patch.
Timeline
By the numbers
What’s next
The port is becoming a product: a standalone LiveKit meeting endpoint running entirely on the ESP32-S31-Korvo-1. Power it on, join a room over Wi-Fi, speak and appear to the other participants, hear the room through the onboard speakers, and see the current speaker on the board’s 4.3-inch 800 × 480 touch LCD. No PC or phone once it is provisioned.
The display follows the active speaker by default; touching a participant pins their video, and AUTO returns to following. One remote video track is decoded at a time so audio continuity always wins. Touch controls cover microphone, camera, volume, participants, and join, leave, and retry.
- Touch UI on the official Korvo BSP: room stage, roster, pin and AUTO, mic, camera, volume, and room controls.
- Bounded LiveKit Cloud sessions with real join, leave, and retry, plus camera and echo-cancelled microphone publication with remote mute indicators.
- Speaker playback with volume control through the buffered renderer while camera preview runs.
- H.264 decode to the LCD proven with a sample player and a live camera loopback at 10 fps.
- An untethered two-room listening test.
- Remote video reception and display, with safe source switching.
- Real participant roster and active-speaker follow with manual pinning.
- Mixing a small number of remote audio tracks locally.
- Quantitative echo-cancellation and sustained-load acceptance.
- An alone-in-the-room disconnect prompt.
Also ahead: an NVIDIA XR AI ↗ client. XR AI connects devices to GPU-hosted, tool-using agents over a LiveKit media hub, so the aim is to make this ESP32-S31 client an XR AI-supported client as far as the platform allows.
Credits
This is a fork of LiveKit’s open-source client-sdk-esp32 ↗ (Apache 2.0). LiveKit built the base SDK; Espressif provides the media components and the board hardware. The S31 port, board component, AEC integration, renderer, test tooling, and documentation are my work, developed with AI-assisted pair programming using Claude Code and ChatGPT Codex.