Direct tank drive
LR
- Left stick Y → left tread
- Right stick Y → right tread
- A or both shoulders → continuous dead-man
- Left/right index triggers → matching grippers
03 visionOS · spatial command · live H.264
VisionProController
R.O.B.'s spatial
control deck.
Operate R.O.B. from a native Apple Vision Pro cockpit with independent tread and gripper control, head-relative neck and torso steering, authenticated live video, voice input, measured Amber telemetry, and a safety contract that expires stale intent in 250 milliseconds.
250 msInput lease
10 HzSafety evaluation
25 HzHead orientation
2 × 7Measured Amber joints
2 QUICIndependent control + video
01 Spatially composed cockpit
Controls wrap around the camera. Safety stays in front.
The 1,760×920 SwiftUI dashboard arranges three purpose-built panels around the forward video surface. The control and telemetry wings rotate eight degrees inward and sit 42 points forward in depth, creating a readable cockpit instead of a flat settings page.
The center prioritizes the camera. The left wing owns motion and stop state. The right wing owns connection, pairing, voice, telemetry, and the safety legend. The window explicitly claims focus for GameController events and stops/disconnects when it disappears.
Precise by design. This is a spatially styled native window—not an immersive room-scale scene. The current product contains no ImmersiveSpace, volumetric robot, or RealityKit arm visualization.
LEFT WINGRobot Controltreads · grippers · speed · arm · stop
FORWARDROB Wide Cameracodec · dimensions · FPS · frames · bytes
RIGHT WINGSession + telemetrypairing · voice · battery · arms · safety
02 Physical + spatial input
Two controllers. Two treads. Two grippers.
ROB's tank drive remains truly independent across every input path. The control domain may convert tread values to linear/angular form internally, but the Cerebro adapter reconstructs each side independently and clamps each to ±0.5 without cross-normalizing one tread against the other.
Tracked pair
LR
- One-handed controller assigned by chirality to each tread
- Both grip buttons must remain held for dead-man
- Each trigger controls its matching gripper
- ARKit publishes position + normalized quaternion; lost tracking clears pose
Press and hold
↑←→↓
- Forward, left, right, and reverse remain available without a physical controller
- Virtual motion cancels as soon as the active press ends
- Accessibility labels announce tread demand and the hold-to-command requirement
VISION PRO
−60°+60°NECK PANBeyond ±60° → bounded rotating torso03 Head-relative embodiment
Look where R.O.B. should look.
While the dead-man is held, the current Vision Pro head pose becomes a fresh neutral baseline. Device-anchor orientation is sampled every 40 ms. Relative yaw maps across a ±60° camera-neck range; relative pitch maps across ±35°. Yaw beyond the neck limit becomes a bounded request for the rotating torso, reaching full normalized travel at 180°.
- Releasing the dead-man clears camera and torso demand
- Lost device-anchor tracking clears demand instead of replaying the last pose
- Leaving the active scene or disconnecting stops publication
- Only fresh, armed controller snapshots carry neck and torso intent
04 Authenticated live video
A camera stream that can never block a stop.
VisionProController authenticates control first, preserves Cerebro's exact live session UUID, then establishes a second domain-separated video connection. A failed or congested camera path can remove video, but it cannot delay motion, dead-man, or stop traffic.
CONTROL PLANE_robctl._udp / robctl/2motion · stop · authority · voice · arm protocol
EXACT LIVESESSION UUID
VIDEO PLANE_robvideo._udp / robvideo/1capabilities · subscription · feedback · H.264 media
Two proofs, one robot
Both services use TLS 1.3 and the exact pinned Cerebro leaf, but video has its own fresh reciprocal HMAC transcript so a control proof cannot be replayed against media. The subscription must carry the control connection's current UUID.
Strict binary receiver
32-byte RVID frames wrap fixed 92-byte RBVD media headers, raw SPS/PPS, and complete AVCC access units. Annex-B, invalid dimensions, stream/session mismatch, length conflict, and malformed current-stream media fail closed.
Bounded recovery
Configuration is capped at 64 KiB and one access unit at 2 MiB. A sequence gap suppresses predictive frames and rate-limits a keyframe request. Exactly one configuration and one matching decodable keyframe may wait for UI activation.
Frame-rate UI without frame-rate state
AVSampleBufferVideoRenderer drives a display layer directly. Only throttled statistics reach SwiftUI every 500 ms, preventing per-frame network state from invalidating the entire interface.
05 Voice command + puppet speech
Let Cerebro interpret it—or let R.O.B. say it exactly.
The Vision Pro microphone and Apple Speech framework handle explicit authorization, current-locale recognition, live partial transcripts, and automatic final submission. On-device recognition is required whenever the selected recognizer supports it. Manual text entry always remains available.
Command mode sends bounded text into Cerebro's existing intelligence/command boundary. Puppet Speech asks R.O.B. to speak the phrase verbatim without passing it through the AI or motion parser. Neither mode can arm motion or bypass the stop contract.
1–1,024 characters · trimmed before transport
06 Amber arm observability
Fourteen measured joints. A preview path that refuses to pretend.
The production observation path receives left and right Amber arm state over rob-arm-control/1. Every sample contains seven positions, seven velocities, seven currents, and seven statuses. Sequence, time, age, shape, finite values, and defensive numerical bounds are validated before the newest sample enters the session snapshot.
J1+18.4°
J2−7.2°
J3+24.1°
7 position · 7 velocity · 7 current · 7 statusJ1−19.0°
J2+4.6°
J3−27.8°
7 position · 7 velocity · 7 current · 7 statusTarget-intent preview, not live teleoperation. The protocol validates exactly seven targets, B1 joint limits, 0.65–10 second duration, authenticated sender, live session, sequence, issue time, and a 50–1,000 ms lease. It has dispositions for stale telemetry, wrong mode, step/speed limit, busy arm, expired or invalid input—but v1 always forces
execution_eligible=false. No current Vision UI submits targets or invokes Amber hardware.07 Safety by freshness
Connected is not armed. Armed is not fresh. Fresh is not forever.
Motion starts inhibited. The app must be connected, explicitly armed, active, receiving current input, and continuously satisfying the dead-man. RobotSession reevaluates the decision every 100 ms; input expires after 250 ms.
CONNECTED+ARMED+ACTIVE SCENE+FRESH INPUT+DEAD-MAN HELD=MOTION
dead-man released250 ms lease expiredscene inactivecontroller disconnectedtransport failedoperator disarmedsoftware stop latchedrobot watchdog
The red control is a latched software stop. It sends a braked/stopped snapshot and releases authority; resetting it leaves motion disarmed. It supplements—and never claims to trigger, reset, or replace—the physical robot's independently wired emergency stop.
08 Full offline simulator
The camera path is synthetic. The codec path is real.
The built-in simulator creates a fresh session, integrates differential motion, drains an 84% starting battery, publishes telemetry every 200 ms, and enforces its own 350 ms receiver watchdog. Its animated BGRA grid and robot marker go through VideoToolbox H.264 compression, the bounded newest-frame channel, SPS/PPS generation, the same receiver validator, sample-buffer reconstruction, and the same AVFoundation display surface as production video.
09 Swift 6 architecture
UI on the main actor. State machines in actors. Media in its own lane.
ROBControlCore is a pure-Foundation, transport-independent domain package. ROBCerebroTransport contains Network, Security, CryptoKit, Bonjour, Keychain, TLS, HMAC, and the narrow compatibility adapter. ROBVideoPipeline owns Core Video, VideoToolbox, Core Media, and AVFoundation. The package has no third-party dependencies.
SwiftUI cockpitRobotViewModel · GameController · head · speech
→RobotSession actordead-man · leases · snapshots · domain commands
→TransportSimulator or Cerebro control + video clients
→Video pipelinevalidator · H.264 receiver · sample-buffer renderer
Implementation status
Spatial control, without inflated claims.
Native three-panel cockpit; Simulator and Cerebro endpoints; Keychain pairing; pinned QUIC/TLS control; independent treads and grippers; gamepad, PSVR Sense buttons/axes, screen controls, and head-relative neck/torso demand; voice command and puppet speech; demand-driven H.264; simulator; measured Amber telemetry; safety latch and 250 ms lease.
PSVR Sense six-DoF poses require visionOS 26. Spatial controller poses are transported, and seven-joint target intents are fully validated, but there is no current IK, arm-target UI, RealityKit arm, or physical Amber execution. Real Cerebro base battery/pose mapping is not yet populated.
No local Gemini, MLX, Core ML, object recognition, autonomy, immersive space, depth/stereo/spatial video, recording, or replay. RPLidar framing exists at a low transport layer, but it is intentionally ignored by the adapter and has no decoder, model, map, obstacle overlay, or UI.
Orbitus Robotics · built on the physical platform
Software that knows it has a body.
R.O.B.'s applications are designed around real cameras, real networks, real actuators, and the discipline required to put intelligent behavior near a moving machine.