# Orbit WebRTC client contract — Mac and Apple TV

Public base URL: https://orbit.vainsoft.io
LAN base URL: http://192.168.1.100:6086
GET /api/webrtc/contract returns the current machine-readable contract.
GET /api/webrtc/status reports active peers, selected codecs, actual encoder backend, encode time and received input count.

## Signaling

1. GET /api/bootstrap and retain token and session.game_pid.
2. POST /api/webrtc/ice with X-Orbit-Token and {}. Create the RTCPeerConnection using the returned iceServers and iceTransportPolicy; retain expires_at.
3. Add a recvonly video transceiver and, for games, a recvonly audio transceiver. Do not open a microphone or camera.
4. Create a reliable, ordered RTCDataChannel named orbit-input.
5. Create/set the local offer. Wait for ICE gathering to complete; send the entire SDP with candidates.
6. POST /api/webrtc/offer with X-Orbit-Token: TOKEN and Content-Type: application/json:

```json
{"type":"offer","sdp":"COMPLETE_OFFER_SDP","target":"game","codec":"h264","bitrate":6000000}
```

7. Apply the returned type and sdp as the remote answer. Save id for cleanup.
8. Attach the remote video/audio tracks to the native renderer and audio output.

The answer also contains selected_codec, requested_codec, fallback_reason, target, video and audio.
Read selected_codec; the requested codec is not a guarantee. video.level_id reports the encoded HEVC level,
video.bitrate reports the effective bitrate, and bitrate_note explains any cap.
POST /api/webrtc/peers/ID/close with the token and {} when leaving, sleeping or changing sessions.
Closing a peer releases held controls and does not stop the game. Fetch a fresh token after an Orbit restart.
The server allows four concurrent peers; capture is shared and encoding is per receiver.

Use the selected base URL for every API call. With HTTPS, use wss:// for legacy
viewer/audio/controller WebSockets. Keep credentials for any Apache authentication
on that same origin. GET /api/proxy/guide describes the deployment.

Public signaling uses HTTPS port 443 through Apache; direct LAN signaling uses TCP port 6086. WebRTC media/data use DTLS-SRTP/SCTP over ICE-selected UDP ports.
Direct LAN connections are preferred when reachable. Remote clients can use turn.vainsoft.io:
UDP or TCP on 3478, or TLS over TCP on 5349. Apache carries signaling; TURN carries
relayed media and input. The TCP/TLS connections are to the TURN host, not the Apache host.

## ICE configuration for Mac, Apple TV and browsers

Before creating each peer connection:

    POST /api/webrtc/ice
    X-Orbit-Token: TOKEN
    Content-Type: application/json

    {}

The response is:

    {
      "iceServers": [
        {"urls": ["stun:turn.vainsoft.io:3478"]},
        {
          "urls": [
            "turn:turn.vainsoft.io:3478?transport=udp",
            "turn:turn.vainsoft.io:3478?transport=tcp",
            "turns:turn.vainsoft.io:5349?transport=tcp"
          ],
          "username": "EXPIRY:UNIQUE_SESSION",
          "credential": "TEMPORARY_PASSWORD",
          "credentialType": "password"
        }
      ],
      "iceTransportPolicy": "all",
      "expires_at": 1793620000,
      "ttl_seconds": 43200
    }

expires_at is an absolute Unix timestamp in seconds; the example is illustrative.
Map each entry to the SDK's RTCIceServer using its URLs, username and credential.
Use the SDK's equivalent of ICE transport policy all. Do not use the legacy static
ice_servers array in the contract to configure a connection: ice_config identifies
this credential endpoint. Fetch fresh credentials for every connection/reconnect,
and reconnect before expires_at for sessions lasting more than 12 hours.
Allow up to 15 seconds for ICE gathering and send the complete gathered SDP.

For a relay-only diagnostic, set the client's ICE transport policy to relay.
Production should use all so a fast direct path can win. Inspect the selected
candidate pair in client getStats: a relay candidate proves TURN was selected;
merely receiving a successful offer answer does not. Test UDP and TCP/TLS separately
by restricting the test client's ICE URLs. This is a connection/bandwidth test,
not a game FPS benchmark.

The endpoint returns expiring HMAC credentials; the TURN shared key stays server-side.
Do not persist temporary credentials in game profiles, analytics or logs. A 503 means
credential issuance failed: retry rather than silently using an empty ICE-server list.
The endpoint requires Orbit's mutation token and returns Cache-Control: no-store.
Apache authentication, when configured, must protect it along with the other API paths.
The server also gathers TURN candidates. aiortc uses the first supported TURN URL
(UDP in this deployment); browsers/native clients receive the full fallback URL list.

## Video and codecs

Current stream profile: 1280×720, up to 30 FPS, 8-bit SDR. The game's internal FPS is independent.

Every game now has a 1080p60 performance goal in `games[].performance_target`.
Read `games[].backends[backend].performance` for measured progress. This goal is
not a transport capability or proof of 60 FPS gameplay; see /api/launch/guide.

There is currently no stream-profile selector in /api/webrtc/offer. The eight planned resolution/FPS modes
(720p, 1080p, 1440p and 2160p, each at 30/60 FPS) are not deployed. Clients should read video.width,
video.height and video.fps from /api/webrtc/contract and the offer response, and show only those supported
capabilities. Do not send a profile field expecting it to change capture or encoding.
Game benchmark recommendations belong to the game/settings/build; client bandwidth and decoder capability
are separate limits. A 4K-capable client does not establish that Astroneer can run at 4K60.
bitrate is an integer from 1,000,000 to 12,000,000 bits/s; default 6,000,000.

- h264: default, NVIDIA h264_nvenc, baseline, packetization-mode=1. Falls back to software x264 only if NVENC initialization fails.
- h265: NVIDIA hevc_nvenc, Main profile, level 3.1 or higher (profile-id=1; tier-flag=0; level-id>=93). Client must advertise compatible video/H265 in its SDP. RFC 7798 single-NAL/FU payloads, no DONL, single stream.
- av1: software libsvtav1 on two threads, profile 0, low-delay. Experimental; this A4000 has no AV1 hardware encoder. Initial bitrate stays fixed.
- auto: prefer H.265 when the client advertises it, otherwise H.264. It never automatically selects software AV1.

HEVC level 3.1 clients receive a level 3.1 stream, capped at 10 Mbit/s; the default 6 Mbit/s is unchanged.
For a requested bitrate above 10 Mbit/s, clients supporting level 4 or higher receive level 4 at up to 12 Mbit/s.
The SDP answer and encoded bitstream use the same level. NVENC peak bitrate and buffer size are explicit
so its defaults cannot silently exceed the negotiated level.

For a requested unsupported codec, include H.264 receive capability so the server can fall back cleanly.
A missing common codec is a request error. H.265/AV1 hardware decoding on a device does not by itself mean
its chosen WebRTC SDK exposes that codec: inspect the SDK's supported receive codecs and actual SDP.
Do not rewrite the SDP to advertise a decoder that the app does not have.
Mac/tvOS builds and actual Apple hardware require client-side validation; the Linux server cannot validate those binaries.

H.264/H.265 accept receiver bitrate feedback, capped at the requested bitrate. AV1 currently keeps its initial bitrate.
The server uses low-delay encoding with no B-frame reordering and a one-second keyframe interval.
RTP retransmission/PLI feedback are supported by the transport; avoid adding an extra multi-frame native display queue.
For Apple rendering use the SDK's native VideoToolbox/Metal path where available. Render the newest decoded
frame promptly, avoid UIImage/base64/JPEG conversions, and keep audio on the SDK's native Opus playback path.
Do not also open the legacy PCM audio socket while WebRTC audio is playing.

## Input channel

Send JSON strings on orbit-input. Keys use the same X11 keysyms as the existing RFB client:

```json
{"type":"key","keysym":106,"down":true}
{"type":"key","keysym":106,"down":false}
{"type":"pointer","x":0.5,"y":0.5,"buttons":0}
{"type":"wheel","delta_y":-1}
{"type":"release"}
{"type":"ping","id":123}
```

J/106 = Cross, L/108 = Circle, K/107 = Square, I/105 = Triangle.
WASD = left stick; TFGH = right stick; Q/E = L1/R1; Z/C = L2/R2.
Enter/0xff0d = Options; arrows 0xff51–0xff54; Shift/0xffe1 = L3; Control/0xffe3 = R3.
Map physical gamepad buttons on the client to these events. Stick support is digital, not native analog axes.
Space → J → Cross is a valid client mapping. Send both press and release.

Pointer x/y are normalized 0–1 across the virtual display. RFB button-mask bits:
1=left, 2=middle, 4=right. Wheel delta_y is -5 through +5, negative means up.
Mouse motion remains absolute pointer input; it is not automatically an analog camera mapping.

Send a ping every second. The reply is {"type":"pong","id":123,"server_time_ms":...}.
Calculate RTT on the client using its own monotonic clock; this is transport RTT, not input-to-photon latency.
After three seconds without input/pings, held controls are released. Send release on focus loss,
controller disconnect and app backgrounding. Stop sending before closing the peer.
Input messages are bounded to 2 KB and 300/s. Coalesce pointer motion and send gamepad state changes.

## Controller lab

POST /api/controller/start first, then create a WebRTC offer with target="controller".
Receive video and send orbit-input events as above; no game audio is attached.
GET /api/controller?since=SEQUENCE reads actual Linux SDL receipt, including key press/release and mouse movement.
The lab is a separate display and does not inject input into the running game.
The existing RFB controller lab remains available; see /api/controller/guide.

## Measurements and limits

Use receiver getStats for codec, framesDecoded, packetsLost, jitter, jitterBufferDelay and decoded size.
Subtract successive jitterBufferDelay / jitterBufferEmittedCount samples to estimate current playout buffering.
Do not label RTT or buffer time as total controller-to-screen latency.
Use /api/webrtc/status to distinguish encoding cost from game FPS and network loss.
The stream cannot remove slow frames generated by the emulator.

Implementation uses pinned aiortc 1.15.0 and PyAV 17.1.0. H.264/H.265/AV1 are send-only codecs here.
The independent H.265 test receiver uses a 512-packet reassembly buffer so large keyframes
are not truncated by aiortc's small default. This is packet reassembly capacity, not an extra
multi-frame presentation queue. Ensure the client SDK can reassemble complete keyframes.

Additional codec packetization follows:
- https://www.rfc-editor.org/rfc/rfc7798
- https://aomediacodec.github.io/av1-rtp-spec/