Cerebro is R.O.B.'s always-on Mac brain and engineering console: synchronized RGB-D and panoramic vision, private Swift MLX intelligence, optional Gemini Live conversation, authenticated Lidar and controller links, deterministic autonomy, supervised arm motion, show control, and deep hardware diagnostics.
CAMERA / RGB-DSTREAMINGCONTROL / ROBCTL2ARMED BY OPERATOR
30 FPSSynchronized OAK RGB-D
3 modelsLocal LLM · VLM · embeddings
0.2 secDeterministic autonomy tick
5 HzController freshness heartbeat
TLS 1.3Pinned controller and video links
01 Perception fabric
See in color, depth, stereo, and 360°.
Cerebro treats perception as a supervised, backpressure-aware service—not a camera window. Each generation of synchronized frames can feed Apple Vision, local MLX, Gemini sampling, calibration, remote video, show context, SceneKit diagnostics, and hologram export without allowing slow consumers to pile up behind live capture.
OAK / DEPTHAI 3.8RGB + depth + stereo640×400 · 30 FPS · 20 ms synchronization
CameraFrameSet
APPLE VISIONSWIFT MLX VLMGEMINI LIVEH.264 VIDEOHOLOGRAMSCENE SNAPSHOT
01.1
A crash-isolated RGB-D camera service
A supervised DepthAI helper owns the Luxonis device outside Cerebro's process and returns a single timestamp-aligned frame set over a user-only Unix socket. RGB arrives as a native sample buffer; depth arrives aligned to RGB as little-endian millimeters; rectified left and right mono images and calibrated intrinsics travel in the same bounded packet.
DepthAI robotics preset, LR check, subpixel depth, undistorted RGB
Nonblocking queue depth of one and newest-frame admission
Disconnect, malformed packet, SDK failure, or helper exit becomes recoverable camera state
AVFoundation remains available as an ordinary-webcam RGB fallback
01.2
Native Apple Vision overlays
The live camera interface runs active 2D human-body and hand-pose requests, configurable 3D human pose, and QR/barcode calibration on gated cadences. A native overlay draws skeletons, hands, projected 3D pose, sword geometry, custom detections, depth color, confidence, and status without blocking capture.
QR + RGB-D reverse-pose calibration estimates camera-to-robot transform and RMS confidence
01.3
A second, panoramic perception lane
The Insta360 service maintains an OSC session and heartbeat, requests a 1920×960 stitched H.264 panorama, optionally applies gyro stabilization, and starts its local frame decoder only when diagnostics or analysis demand it. Whole-panorama analysis or six local sectors can be sampled independently.
Live preview, bytes, frames, FPS, session and gyro diagnostics
Independent MLX, body-pose, object/scene, and custom-model toggles
Newest-frame-only advisory perception, never the motor loop
01.4
Detectors you can add at runtime
The detector registry gives every source its own enable state and rate. Built-in classification and saliency can be combined with hot-imported Core ML models whose recognized regions and centers become bounded scene observations. Main-camera body pose stays on its dedicated low-latency path.
Per-source processing rates from 0–30 FPS
Hot-added .mlmodel and .mlmodelc packages
Source-generation checks discard results from replaced camera sessions
Scene snapshots label sensor JSON as untrusted observation, never instruction
02 Hybrid intelligence
Private on the Mac. Live when the moment needs more.
Operators can choose a real in-process Swift MLX stack, an isolated loopback llama.cpp server, or Gemini Live. All three sit outside deterministic motor control. Their output must cross strict schemas, timeouts, authorization, and human-visible state before it can influence the physical machine.
LOCAL / PRIVATESwift MLXMetal
Three local models. One guarded runtime.
ROBMLXEngine is an actor-isolated native Swift inference runtime with pinned packages, first-use Hugging Face downloads, offline reuse, cancellation, and complete diagnostics.
Dialogue LLM
Llama 3.2 1B Instruct · 4-bit
Camera VLM
Qwen2-VL 2B Instruct · 4-bit
Embeddings
TaylorAI gte-tiny
Schema-validated local stage plans; attempted model tool calls throw
Per-source sampled VLM frames with only one inference in flight
Fresh, high-confidence stage observations only; explicitly uncertain and non-executable
Process-local semantic memory capped at 200 entries with cosine retrieval
Live load/generation latency, tokens/sec, frame count, download, and Metal memory telemetry
OPTIONAL / MULTIMODALGemini LiveRealtime
Camera, microphone, dialogue, and supervised tools.
The default Robotics streaming model receives independently enabled audio and sampled video, supports response text or audio, maintains resumable WebSocket sessions, and can optionally use server-side Google Search.
Microphone
Mono PCM16 · 16 kHz
Camera
JPEG · ≤1 FPS · ≤768 px
Deadlines
15 sec start · 120 sec turn
Raw mic, sampled camera, connection, Search, and response mode are separate switches
Idempotent tool ledger and fresh-controller approval gate
Diagnostics retain counters and redacted state—not media, transcripts, credentials, tool arguments, or raw responses
LOCAL ALTERNATIVE
Loopback llama.cpp
A second local provider can run out of process for crash isolation. Cerebro accepts literal-loopback endpoints only, rejects redirects and non-loopback final URLs, bounds health and response bodies, supports cancellation, and applies the same five-key dialogue schema and fallbacks as MLX.
APPLE-NATIVE REASONING
Foundation Models
On macOS 26 where Apple Intelligence is available, a structured scene interpreter can answer, clarify, describe, suggest navigation, inspect, stop, or choose no action. It has no motor or servo tool; navigation and inspection remain suggestions requiring human confirmation.
VOICE PIPELINE
Listen locally. Speak naturally.
One audio-engine tap serves on-device Apple Speech and optional Gemini raw audio. Local recognition requires on-device processing, supports partial results and English/Japanese output, and pauses while R.O.B. speaks to avoid feedback. Conversation history stays local, capped at 300 messages and seven days.
03 Lidar + autonomy
Cloud intelligence talks. Local code owns the treads.
A separately paired RPLidar publisher can send authenticated scans into Cerebro. The robot derives four directional free-space sectors and exposes only fresh, bounded geometry to the scene model and autonomy coordinator. A stale scan does not become optimistic clearance—it stops movement and waits.
AI supplies conversation and social timing. The deterministic planner is the only component that produces left and right tread values.
Pair the role. Pin the robot. Reject the downgrade.
Cerebro's modern control plane is a reliable ordered QUIC stream over UDP with TLS 1.3, exact leaf-certificate pinning, a fresh reciprocal HMAC proof, and per-device secrets stored in Keychain. Bonjour advertises routing hints, never identity. Controller and Lidar publishers receive different server-owned roles and cannot promote themselves by editing a pairing payload.
Vision camera clientsession-bound videoH.264 · feedback · recovery IDR
Independent identities
Every device receives its own controller UUID, 256-bit secret, role, and revocation record. Revoking one tombstones it and disconnects its live session without disturbing the others.
Fresh authority
Frames carry protocol version, live UUID, message UUID, monotonic sequence, and bounded length. Cloned credentials, replayed sequences, stale sessions, oversized payloads, and role mismatches fail closed.
No automatic downgrade
The historical plaintext transport is an explicit legacy adapter, disabled by default. A failure in modern discovery or authentication never silently falls back to it.
Video cannot delay control
The H.264 sender uses its own service, connection, queue, framing, proof domain, and bounded newest-frame state. Media congestion cannot queue a stop behind video frames.
05 Embodied control
Every actuator has a gate.
Cerebro bridges high-level operator intent to the physical base, servos, rotating torso, and dual seven-joint Amber arms. Freshness, acknowledged modes, bounded values, measured telemetry, and explicit authority live between a button—or a model—and motion.
BASE
Passive Arduino discovery
USB callout devices are opened at 250,000 baud, reset, and observed for the exact Base startup line. Cerebro sends no probe bytes to unidentified boards. At 5 Hz it forwards independent treads, brakes, flipper, and linear-actuator state; three missed snapshots produce one neutral/braked frame, then USB writes stop so the Arduino watchdog can de-energize the base.
NECK + GRIPPERS
Maestro servo control
VID/product discovery reconnects the Pololu Maestro. Normalized head demands become bounded, slew-limited pan, tilt, and upper-tilt commands. Arm and gripper channels retain calibration controls; remote gripper changes are edge-triggered rather than resent forever.
ROTATING TORSO
Tic safe-start ownership
Vision head yaw can request a normalized half-turn around a stored baseline; the local UI has a full-turn range. Position conversion and slew limits remain on the robot Mac. Releasing control enters safe start and de-energizes; a missing ticcmd disables only that feature.
AMBER B1
Measured arm gateway
An SSH-tunneled, token-authenticated gateway exposes seven-joint position, velocity, current, and status telemetry; verified mode transitions; and trajectories bounded by B1 limits and duration. Named gestures are immutable approved snapshots and settle against multiple measured position/velocity samples before completion.
2 × 7Measured arm joints
Short-lived Gemini or controller authority can be granted for named gestures, but it still requires a healthy tunnel, fresh telemetry, position mode, bounded steps and speeds, acknowledged commands, and a calibrated executor. Revocation, timeout, disconnect, or cancellation holds the current position. Vision Pro target intents already have a complete authentication and safety-validation protocol, but protocol v1 deliberately keeps physical execution disabled.
06 Performance engine
Rehearsed reliability. Room for improvisation.
The stage runner validates an authored show before it speaks or moves. Offline, local-model, and adaptive modes share one cue engine, one cancellation path, mandatory fallbacks, and hard content boundaries that keep show files from smuggling in raw joints, hosts, ports, shell commands, or model-authored motion.
Swift MLX or llama.cpp chooses only an allow-listed beat and delivery, plus a short offline line.
03
Adapt live
Cerebro writes the trusted Gemini brief and can add current camera/audio awareness.
04
Fall back
Gemini line → validated local line → mandatory authored line, all within the cue deadline.
05
Stop cleanly
Cancellation, timeout, spoken stop, and priority model stop close the active lane without waiting behind ordinary work.
Maker FaireComedy ActsJunior Maker ComedyGalactic Saber BattleProgressive Saber Training
07 RGB-D hologram capture
Turn a synchronized moment into a portable 3D artifact.
Development Mode can capture a depth-textured still or up to fifteen seconds of animated RGB-D with optional audio, then export a self-contained web viewer and USDZ package. The same capture can orbit in a browser, open in Apple AR Quick Look on iPhone, iPad, or Vision Pro, and be shared through a temporary ten-minute local link.
Advanced capability. Hologram export is implemented and intentionally exposed through Cerebro's Development menu rather than the default show-operator workflow.
08 Robot-grade operations
An always-on app has to fail differently.
Cerebro isolates optional dependencies and repairs the smallest failed subsystem instead of taking robot control down with it.
Single owner
An atomic per-user process lock prevents an installed copy and an Xcode copy from initializing the same hardware or listeners.
Supervised restart
A LaunchAgent restarts a crash in two seconds, opens its circuit after ten consecutive failures, and resets the count after five healthy minutes.
Wake recovery
macOS wake immediately triggers camera, dependency, Lidar, and Insta360 health passes.
Managed Python
Choose Python 3.9+, autodetect, select venv/Conda, or create an app-owned environment; package installation happens only after operator action.
Dependency preflight
Every compiled subprocess path must be absolute, present, executable, and not a directory immediately before launch.
Least privilege
Package-manager workflows never ingest an administrator password; Keychain secrets never appear in diagnostics or process arguments.
Implementation status
Built, bounded, and explicit about the frontier.
IMPLEMENTED
OAK RGB-D + stereo; Insta360 panorama; Apple Vision overlays; native Swift MLX LLM/VLM/embeddings; Gemini Live; Foundation Models scene interpreter; RPLidar scans; bounded Social Roam; authenticated controller network; base/servo/Tic I/O; measured Amber gateway; stage runner; diagnostics; Development-mode holograms.
PREVIEW / FOUNDATION
Cerebro's secure Vision Pro H.264 sender and media plane are implemented; the receiving adapter is a separate product boundary. Vision Pro arm intents receive full validation and measured feedback but cannot execute hardware in protocol v1.
LEGACY / LAB
Kinect/NiTE/PCL, RealSense, ReSpeaker task controllers, Leap, and historical RTSP sources remain in the codebase but are not presented as supported production sensors. Lidar maps are authenticated and bounded but the main controller currently ignores them.
Orbitus Robotics · built on the physical platform
Software that knows it has a body.
R.O.B.'s applications are designed around real cameras, real networks, real actuators, and the discipline required to put intelligent behavior near a moving machine.
