Media and voice integration

A logical voice channel is not an audio stream. A vision observation is not a camera frame. Keep those distinctions visible in your application: RMC coordinates authority and durable facts while the native provider transports the live media.

The architecture walkthrough places this media path beside durable delivery and work execution. Channels explains how native media bindings differ from observation topics. For provider coupling, deployment differences, and replacement options, read LiveKit dependencies.

⤢ Expand

Name the two video channels correctly

Provision camera and screen-share channels with an explicit binding:

await owner.createMediaVideoChannel("camera", {
  adapter_id: "livekit", source: "camera",
});
await owner.createMediaVideoChannel("screen", {
  adapter_id: "livekit", source: "screen_share",
});
await owner.createVideoChannel("vision");

createVideoChannel creates a video observation topic. createMediaVideoChannel creates the bound native-media resource. These are separate helpers despite the similar names. The current native video binding requires camera or screen-share sources; broader exported media-source constants do not imply that this endpoint supports microphone channels.

Request the right kind of media token

The browser asks RMC for a short-lived provider credential, then passes its URL and access token to the LiveKit client. LiveKit API keys remain on the server.

const credential = await room.createMediaSession("livekit", {
  publishChannelIds: ["camera"],
});
// Your existing LiveKit Room instance owns the native connection.
await livekitRoom.connect(credential.url, credential.access_token);
await livekitRoom.localParticipant.setCameraEnabled(true);

livekitRoom is your native LiveKit SDK instance, not an RMC object. This scoped request requires media:join on livekit and media:publish on camera. RMC verifies that each requested channel is open, input-capable, bound to the requested adapter, and of type media.video.

For an authorized receive-only processor, omit the options and issue a participant with media:join and media:subscribe on livekit, without an adapter-level publish grant. Do not substitute an empty publish list: that creates a token with no subscription authority.

These distinctions are enforced by createMediaSession and covered in media tests.

Start voice from a user action

// Run from your Start voice button. audioElement is your <audio> element.
const voice = await room.connectVoice({
  channelId: "voice",
  audioElement,
  onStateChange: state => showVoiceState(state),
});

// Run from your Stop voice action or session teardown.
await voice.close();

The SDK captures microphone audio, negotiates a WebRTC peer, and uses the provider data channel. The server requires voice:start on *; voice-session ownership governs later operations unless the caller has the relevant session grant. Configure the provider in the server entry point before starting voice.

onProviderEvent exposes live provider events, which are not automatically durable room facts. Subscribe to the room for normalized conversation events. The current server also persists normalized transcript deltas and provisional turn updates before final turn commitment; event granularity explains that boundary. Provider playback state, microphone meters, and typing UI remain live state.

The SDK supports interruption, playback reporting, explicit reconnect, and an AbortSignal covering negotiation and subsequent lifetime. Its close path stops captured tracks and tears down peer/audio resources before awaiting remote session deletion. Inspect voice.ts and voice lifecycle tests.

Trace an OpenAI Realtime turn

The current voice integration has two provider connections with different responsibilities:

Starting a turn follows this sequence:

1. The SDK captures the microphone, creates a WebRTC peer and data channel, and sends its SDP offer to RMC's /voice/sessions endpoint.
2. RMC authenticates voice:start, reserves a durable voice-session record, filters the room snapshot for that participant, and selects only the tools the participant may use.
3. The Realtime adapter sends the SDP plus model configuration to OpenAI. Its instructions include the authorized context projection: committed turns, visible current state and artifacts, and visible open delegations. RMC returns OpenAI's SDP answer to the browser. RMC never receives the resulting audio stream.
4. The adapter attaches a sideband WebSocket to the provider call. OpenAI events arriving there are normalized into provider-neutral VoiceEvent values. Audio-delta events and unknown provider payloads are deliberately ignored by the durable path.
5. The server callback publishes each selected voice.* event through the room runtime. When it contains a turn update, the callback also updates the conversation projection, which appends a separate conversation.turn.updated or conversation.turn.committed event.
6. Room subscriptions deliver those committed events independently of the provider data channel. Use onProviderEvent for immediate provider UI and a room subscription for replayable application state.

The principal mappings are:

One provider occurrence can therefore create two room events: a normalized voice fact and a versioned conversation-turn fact. In the current implementation, actor_id on these voice facts is the participant that owns the voice session. The turn's speaker distinguishes user and assistant content; voice-session lifecycle payloads record the configured model.

The room also feeds selected facts back into the active model. A subscription starting at the session's context cursor forwards authorized committed text, visual observations, speech requests, and delegation updates over sideband. Text or completed delegation input can request a new model response. A Realtime function call enters RMC's normal tool authorization and optional-confirmation path; the adapter returns the tool output to the provider. The special rmc.delegate tool starts background work and returns a handle immediately, while later delegation events can be forwarded into the live interaction.

Read the browser negotiation, voice-session bridge, provider adapter, and provider-neutral voice contract. The events page explains where this path joins ordinary publication.

Persist observations, not raw media

The existing vision processor consumes authorized LiveKit tracks, samples frames, runs local Apple Vision, and publishes bounded video.observation.created events to vision. Camera/screen observations carry provenance identifying the sample, participant, media channel, track, session epoch, policy revision, processor, and model. Do not fabricate these fields in a browser to impersonate a processor.

Raw frames do not belong in durable event payloads, state, or artifacts. The demo has a separate authenticated in-memory sampled-frame ingress that can pass images to authorized active OpenAI Realtime sessions. That is a purpose-built path with validation, bounds, and lifecycle checks; it is not a general ephemeral pub/sub API. RMC does not persist those image bytes. Provider-side handling follows the deployment's OpenAI data controls.

Read the vision processor guide, frame ingress, and video event validation. Use existing basic presence diagnostics through mediaPresence() and refreshMediaPresence("livekit"); do not introduce typing or meter events into the durable log to imitate a presence service.

Close the native resources you own

When leaving a view, close its subscriptions. When leaving the media session, close voice and disconnect LiveKit, including owned local tracks. When the product ends the durable session, use the authorized room-close operation. Calling room.dispose() alone only releases the SDK stream.

Revocation and room closure commit local authority changes and provider cleanup intent before attempting provider removal. Cleanup failures are durably retried. LiveKit token revocation differs between Cloud and self-hosted deployments, and connected clients may receive refreshed credentials. Initial token TTL is not the complete access-ending guarantee. Read setup and revocation and verify reconnect denial in the deployed configuration.

The LiveKit adapter, provider cleanup runtime, and optional integration test harness guide describe the current integration boundary.