# Orbit controller lab: Mac client handoff

Public server: https://orbit.vainsoft.io
LAN server: http://192.168.1.100:6086
Browser screen: https://orbit.vainsoft.io/#controller
Use wss:// with the HTTPS base, or ws:// with the direct LAN HTTP base.
Machine-readable contract: GET /api/controller/contract

The lab uses a separate 1280×720 X11 display and a native Linux SDL receiver.
It uses the same x11vnc → RFB-over-WebSocket input/video path as the game viewer.
Input received by Linux updates both the streamed controller card and the JSON trace.
No running game is required; the lab never routes test input to Astroneer.

## Connect

1. GET /api/bootstrap and read the top-level token.
2. POST /api/controller/start with header X-Orbit-Token: TOKEN.
   JSON body {} is sufficient. This is idempotent while the receiver is running.
3. Connect your existing RFB/noVNC client to:
   wss://orbit.vainsoft.io/api/controller/stream?token=TOKEN&client=Mac-client
4. Complete the normal RFB handshake. Send ordinary RFB key/pointer messages,
   as for /api/stream. This is a binary RFB WebSocket, NOT a JSON input socket.
5. Watch the video, open the browser lab alongside the client, or poll
   GET /api/controller?since=0&limit=80. Poll at roughly 100–200 ms, passing the
   previous response's seq as the next since value.

Native URLSession clients can omit Origin. A browser client should load from the
Orbit server origin. When an Origin header is present it must match the server
scheme and host. The token changes when Orbit restarts: bootstrap again.

The lab listener underneath the bridge is loopback-only. The Mac client needs
only port 6086, with no SSH or separate VNC port.

## First mapping test: Space → Cross

KytyPS5 defaults map J to Cross (the south face button / Xbox A). Translate Space
on the CLIENT, then send the J event. A literal Space is intentionally shown as
unmapped in the lab.

With a connected noVNC RFB instance:

```js
// On the physical Space key going down:
rfb.sendKey(0x006a, "KeyJ", true);
// On the physical Space key being released:
rfb.sendKey(0x006a, "KeyJ", false);
```

For a native RFB implementation, the standard KeyEvent bytes after handshake are:

- J down: 04 01 00 00 00 00 00 6a
- J up:   04 00 00 00 00 00 00 6a

The raw keysym is lowercase j, not the letter A. Send both transitions, suppress
the original Space event, avoid duplicating auto-repeat key-downs, and release
held controls when the app loses focus. The browser lab has an optional
"Example client mapping: Space → J → Cross" switch demonstrating this.

Expected telemetry while held:

```json
{
  "state": {
    "keys": ["j"],
    "pad": {
      "buttons": ["Cross"],
      "axes": {"left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0},
      "triggers": {"L2": 0, "R2": 0}
    }
  }
}
```

Expected trace: a linux-sdl-receiver key_down event with key "j", then key_up
with key "j". After release, keys and pad.buttons should be empty.
A short tap may occur between polls; inspect events rather than only the current state.

## Default mapping reference

| Send host key | Expected pad control |
| --- | --- |
| J / L / K / I | Cross / Circle / Square / Triangle |
| Q / E | L1 / R1 |
| Z / C | L2 / R2, also digital trigger 1 while held |
| Shift / Ctrl | L3 / R3 |
| Enter | Options |
| Arrow keys | D-pad |
| Tab / Backspace | TouchPad |
| A / D | Left-stick X: -1 / +1 |
| W / S | Left-stick Y: -1 / +1 |
| F / H | Right-stick X: -1 / +1 |
| T / G | Right-stick Y: -1 / +1 |

Opposite directions cancel. Both axes can be active simultaneously.
axis_bytes exposes Kyty's corresponding digital values: 0, 128 (center), 255.
F2/F6/F7/F8 and 1/2/3 are listed as emulator shortcuts; this isolated receiver
does not execute them.

Reference: the pinned KytyPS5 893b035 hostInput.cpp defaults. Custom emulator
input maps are not loaded by the lab.

## Mouse and controller limits

RFB pointer messages use absolute framebuffer coordinates, 0–1279 by 0–719.
The lab records position, observed movement deltas, buttons and wheel events.
Buttons 1/2/3 are left/middle/right in the Linux receiver. Wheel activity is
reported separately; horizontal and vertical totals are provided.

Raw mouse clicks have no default PS5-button mapping. For a mouse-button-to-R2
binding, translate the click to C down/up in the client; expect R2 and trigger 1.
Raw mouse motion does not light up a stick: smooth analog mouse-look needs its
own integration. F7 mouse-to-stick emulation is not simulated by this receiver.

Basic RFB does not carry native gamepad axes. A Mac controller can test bindings
by translating its buttons/digital stick directions into the host keys above.
Analog gamepad reports, rumble, gyro and guest-game acknowledgement are outside
this receiver's scope. The display shows an EXPECTED controller state derived
from actual Linux key receipts, not a readback of KytyPS5 scePadRead.

## Telemetry and capture

GET /api/controller returns:

- schema_version: currently 1.
- session_id: changes on each new lab session.
- running, error, display, resolution.
- seq: latest server sequence. Save it as your next since cursor.
- oldest_seq and truncated: allow detection of an expired cursor.
- last_receipt_at_ms: most recent receiver message, including heartbeats.
- clients: connected RFB clients with label, peer address and ID.
- state: current raw keys, mouse state, inferred pad state and host event_count.
- events: received transitions newer than since, bounded by limit (1–200).

Event fields include seq, session_id, kind, source, host_seq, host_time_ms,
server_time_ms, and event-specific fields (key, keycode, scancode, x/y, etc.).
source "linux-sdl-receiver" means the native host window received the event.
source "orbit" denotes lifecycle events. Times are host-side timestamps, not
network latency measurements. Never infer Mac-to-Linux latency without clock sync.

GET /api/controller/capture downloads the latest 1,000 events, current state and
mapping contract. This is the same evidence available to the browser and Codex.
The trace is held in memory and is cleared when a new test session starts.

POST /api/controller/release releases held X11 keys/buttons on the test display
and adds an explicit reset event. Use the same X-Orbit-Token header.
POST /api/controller/stop stops only the isolated lab.
Disconnecting the final lab viewer also releases held controls. When multiple
clients are connected they share a display: raw input cannot be attributed to
one client after X11 has merged it. Test one sender at a time when attribution matters.

## Suggested checks

1. Literal Space: raw "space", unmapped; no Cross.
2. Remapped Space: raw "j", Cross held; clean release.
3. W+D chord: left_x +1, left_y -1. Release both: centered.
4. A+D: left_x 0. Release D while A is held: left_x -1.
5. C hold: R2 and right trigger 1; release returns both to neutral.
6. Quick J tap: both transitions appear in the trace even if the current-state poll misses the hold.
7. Mouse movement/click/wheel: coordinates and raw events arrive.
8. Blur/disconnect: no stuck controls. Use release if developing an incomplete client.

Browser input during these checks affects the lab only. Existing game PID,
saves, profile settings and library files remain untouched.


## Mac client latency and rendering guidance

These changes reduce transport/client delay; they cannot shorten a slow guest frame.
The server captures on a 10 ms polling interval with a 5 ms update-defer setting.
Adaptive long screen-polling backoff is disabled. These are configured waits,
not measured end-to-end latency guarantees.

- Use the LAN address directly on port 6086. Prefer Ethernet at both ends.
- Send key/button down and up immediately, independently of rendering or telemetry
  polling. Coalesce pointer-motion updates to roughly 60 Hz; preserve button edges.
  Never delay a press until the next JSON poll or video frame.
- For noVNC start with qualityLevel=6 and compressionLevel=2 (Orbit's settings).
  Native RFB clients can request Tight/JPEG with equivalent moderate quality and
  low zlib effort. Lower JPEG quality if bandwidth is saturated. Higher compression
  can cost host CPU; measure rather than assuming it improves latency.
- Keep one persistent RFB connection. Decode network updates in order off the UI
  thread. RFB rectangles and compression streams depend on previous updates: do not
  drop arbitrary received updates or bytes. Apply every update to the backing
  framebuffer, but present only the newest completed framebuffer at display refresh.
  Avoid queuing a history of rendered frames behind the current one.
- Request incremental framebuffer updates promptly. Avoid repeated full-screen
  requests and avoid a second frame-rate limiter below the received update rate.
- Keep networking/input responsive while decoding or uploading textures. Reuse
  image/texture buffers and update changed regions when your renderer supports it.
- Measure network round-trip, RFB update/decode time, and time waiting to present
  separately. The Linux lab receipt timestamp and the Mac clock are different clocks;
  subtracting them without clock synchronization does not measure one-way latency.
  The lab's 100–200 ms JSON polling is for diagnostics, not gameplay input.

The current transport is RFB, not H.264/WebRTC. Hardware video encoding with NVENC
would require a server streaming implementation and a matching Mac decoder; selecting
a Mac-side video codec alone does not enable it.

## Game audio

Game audio is now a separate PCM WebSocket on the same port. Read [the audio handoff](/api/audio/guide) and [format contract](/api/audio/contract). The RFB connection continues to carry video and input.


## WebRTC transport

The same input mapping is now available over the orbit-input WebRTC data channel.
See /api/webrtc/guide for H.264/H.265/AV1 video, Opus audio, and target=controller
for testing actual SDL receipt in this lab. /api/controller still exposes the receipt trace.