> ## Documentation Index > Fetch the complete documentation index at: https://hud-f5fd7c15-lukass-phys-experimental.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Robots > How HUD models physical robots: capability contracts, bridges that connect to sim or hardware, and the agent harness that drives VLA policies. The `robot` capability is in **beta**. The wire protocol is versioned `openpi/0`; the contract schema is v0. Expect additive changes while the design settles. HUD runs robot environments the same way it runs everything else - an environment declares tasks and capabilities, an agent drives a live `Run`, but a 50 Hz policy can't stream actions over tool calls. So the `robot` capability is instead a continuous **observation/action loop over WebSocket**: the environment streams observations (camera frames, robot state) and the agent streams back actions, as fast as the policy can run. The wire format is **openpi**-inspired (msgpack with numpy serialization), so existing openpi policy servers only need a thin adapter. Everything below ships behind the `robot` extra (pulls in numpy + openpi-client): ```bash uv theme={"dark"} uv add 'hud[robot]' ``` ```bash pip theme={"dark"} pip install 'hud[robot]' ``` ## Overview Like with other HUD workflows there's the environment side and the agent side. For robotics the **environment side** translates incoming actions into changes in the digital or physical environment and serves observations. The **agent side** owns the policy: it reads those observations, runs inference, and sends actions back. Both sides need building, and this is where robotics differs from the rest of HUD. For LLM agents you can lean on a standard inference provider and a stock harness, so often the environment is the only thing you write. For robot policies there is no equivalent - no hosted inference provider, no standard harness. HUD ships tooling for **both** sides: a handful of small, named abstractions you implement, with the framework owning everything in between (the serve loop, the wire protocol, telemetry to platform). ```mermaid theme={"dark"} flowchart LR subgraph ENVS["environment side"] subgraph EP["RobotEndpoint"] BR["RobotBridge"] end end subgraph AGS["agent side"] subgraph HA["RobotAgent"] direction LR AD["Adapter"] <--> MO["Model"] end end EP <-->|talks to| HA classDef node fill:#efece8,stroke:#2b2722,stroke-width:1px,color:#2b2722; class BR,AD,MO node; style EP fill:transparent,stroke:#8a8580,stroke-width:1px; style HA fill:transparent,stroke:#8a8580,stroke-width:1px; style ENVS fill:transparent,stroke:#2b2722,stroke-width:1.5px; style AGS fill:transparent,stroke:#2b2722,stroke-width:1.5px; ``` **Environment side** - owns the simulator and serves frames: * **`RobotBridge`** - the one class you implement around your sim: `reset` / `step` / `get_observation`. The framework owns the WebSocket serve loop and the single-agent connection. * **`RobotEndpoint`** - wraps the bridge - the environment server's handle for the sim (even if the sim is running in another process) **Agent side** - runs the policy and streams actions: * **`RobotAgent`** - the harness: connects to the env and bridge, owns adapter and model, drives model until env terminates. * **`Model`** - the actual stateless checkpoint of the model (includes pre-/post-processing) * **`Adapter`** - translates the env's observation space to the model's, and the model's action space to the env's [**The contract**](#contract) (of the environment) - the one artifact both sides share: a self-describing JSON schema of the environment including the embodiment's control rate, observation and action spaces, carried in the capability's manifest params. The agent wires observations to policy inputs purely from the manifest; there is no shared config. In addition, the contract serves to define useful metadata that you can easily toggle to improve trace visualization on the platform. ## Environment side HUD has two ways to build the environment side: [`env.gym`](#gym-style-envs) for anything gym-shaped, which derives the contract, capability, and serving for you, or a [custom `RobotBridge`](#custom-bridges) for a sim that isn't. ## Gym-style envs Calling `env.gym(make_env)` derives the [contract](#contract) from a sample observation, serves the env over the `robot` WebSocket, and returns a [`RobotEndpoint`](#custom-bridges) - the handle templates drive episodes through: ```python env.py theme={"dark"} from hud import Environment env = Environment(name="my-sim") sim = env.gym(make_env) # make_env: any callable returning a gym-style env @env.template() async def pick_and_place(task: str = "default", seed: int = 0): ep = await sim.reset(task=task, seed=seed) # {prompt, token} yield {"prompt": ep["prompt"]} yield await sim.result() ``` `env.gym` takes one argument, in any of three forms: * **A factory callable**, like `make_env` above - a module-level function that returns a gym-style env. Its signature splits task args into build args (a change rebuilds the env) and episodic args, which flow to `env.reset(seed=..., options=...)`. * **A gymnasium registry id**, a string such as `"CartPole-v1"` - built with `gym.make` (or `gym.make_vec` when `num_envs` is passed). * **An already-constructed env** made through the gymnasium registry - `env.gym` reduces it to its spec and rebuilds it in the sim process. `env.gym` serves either a standard, single-instance env or a vectorized one - one gym env that internally holds `num_envs` copies of the environment, stepped together in the same process for GPU efficiency. Each copy is a **slot**, and a vectorized sim hands back a slot token on `reset` so the rollout that claimed it can step and grade that copy alone, leaving the rest untouched. A single-env sim has only one slot, so the template may omit the token, as above. A vectorized sim (`num_envs > 1`) must thread it through so each session grades its own slot - yield `{"prompt": ep["prompt"], "robot": {"token": ep["token"]}}` and call `sim.result(token=ep["token"])`; see [vectorized envs and evals](#vectorized-envs-and-evals) for the full pattern. You can find gym-shaped environments in various places - [LeRobot EnvHub](https://huggingface.co/docs/lerobot/en/envhub), including Isaac Lab Arena environments, or environments built into LeRobot directly. Vectorized (`num_envs=4`), with cameras enabled for RTX frames. `make_env` downloads the [EnvHub](https://huggingface.co/docs/lerobot/en/envhub_isaaclab_arena) repo's module and calls its `make_env` the same way LeRobot's loader does. ```python env.py theme={"dark"} from __future__ import annotations import os from hud import Environment HUB = "nvidia/isaaclab-arena-envs" def make_env(num_envs: int = 4): from lerobot.envs.configs import IsaaclabArenaEnv from lerobot.envs.utils import ( _call_make_env, _download_hub_file, _import_hub_module, _normalize_hub_result, ) cfg = IsaaclabArenaEnv( hub_path=HUB, environment="gr1_microwave", embodiment="gr1_pink", object="mustard_bottle", headless=True, enable_cameras=True, # required with headless for RTX frames camera_keys="robot_pov_cam_rgb", num_envs=num_envs, episode_length=50, state_keys="robot_joint_pos", state_dim=54, action_dim=36, # GR1 task="Reach out to the microwave and open it.", ) repo_id, _, local_file, _ = _download_hub_file(cfg.hub_path, True, None) module = _import_hub_module(local_file, repo_id) suites = _normalize_hub_result(_call_make_env(module, num_envs, False, cfg)) return next(iter(next(iter(suites.values())).values())) env = Environment(name="arena") sim = env.gym(make_env, num_envs=4, fps=30) @env.template() async def episode(seed: int = 0): ep = await sim.reset(seed=seed) # Vectorized: token pins this rollout's slot. yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}} yield await sim.result(token=ep["token"]) ``` Meta-World ships inside LeRobot itself rather than as an EnvHub repo, so `make_env` calls LeRobot's own factory directly. GymBridge keeps the env's `pixels` and `agent_pos` keys as-is. ```python metaworld_sim.py theme={"dark"} from __future__ import annotations import os os.environ.setdefault("MUJOCO_GL", "egl") def make_env(task: str = "assembly-v3"): from lerobot.envs.configs import MetaworldEnv from lerobot.envs.factory import make_env as lerobot_make_env # Unwrap vec-of-one; pixels_agent_pos = camera + 4-D eef/gripper state. suites = lerobot_make_env(MetaworldEnv(task=task, obs_type="pixels_agent_pos"), n_envs=1) suite = next(iter(suites)) task_id = next(iter(suites[suite])) return suites[suite][task_id].envs[0] ``` ```python env.py theme={"dark"} from metaworld_sim import make_env from hud import Environment env = Environment(name="metaworld") # LeRobot hardcodes render_fps=80; Meta-World control is slower - pin playback. sim = env.gym(make_env, fps=20) @env.template() async def episode(task: str = "assembly-v3", seed: int = 0): """One Meta-World episode; task name is a build arg (e.g. assembly-v3, dial-turn-v3).""" ep = await sim.reset(task=task, seed=seed) yield {"prompt": ep["prompt"]} yield await sim.result() ``` Needs `metaworld` / `lerobot[metaworld]` installe. ## Custom bridges For a sim that isn't gym-shaped, subclass `RobotBridge` instead: ```python theme={"dark"} from hud.environment.robot import RobotBridge class MySimBridge(RobotBridge): def reset(self, task_id: str, seed: int = 0) -> str: ... # build the episode (runs on the sim thread) return self.task_description # becomes the task prompt def step(self, action) -> None: ... # advance one tick; set success / terminated def get_observation(self): # Always [N, ...] arrays + [N] terminated (N=1 is fine). return {"agentview_image": frames, "state": vecs}, done_mask ``` Those three methods are all you write. Under the hood the framework takes care of communication with the agent and starting/stopping as well as stepping of the simulator at the *control rate*. All three hooks run on the sim thread (main under `serve_bridge`), so thread-affine sims such as Isaac are safe without extra hopping in your subclass. * **`reset`** starts a fresh episode for a task and returns its prompt (the text the agent is given). * **`step`** applies one action and advances the sim a tick, setting `success` / `terminated` as the episode plays out. * **`get_observation`** returns a structured dict of the current observation plus whether the episode is done. The `get_observation` function has a strict output convention, see below to follow it. **The `data` dict is the strict part.** It is what the agent indexes by name and feeds straight to the policy, so a few things have to be exactly right: * **Values are numpy arrays** - nothing else survives the trip into the adapter and the trace viewer. * **Each key is an observation feature's name, verbatim from the contract.** The agent does `data[name]` directly off the contract * **Images are `HWC` arrays** (`[H, W, 3]`, `uint8` RGB). * **State is a single 1-D array**, passed to the policy as `float32`; everything rank-1 is treated as state. * **`terminated` is a sibling, not part of `data`** - return it as the second item of your `(data, terminated)` tuple and the framework attaches it to the frame. ```python theme={"dark"} def get_observation(self): data = { "observation/image": rgb, # [256, 256, 3] uint8, RGB, HWC "observation/wrist_image": wrist_rgb, # [256, 256, 3] uint8, RGB, HWC "observation/state": np.concatenate([ # [8] float32, in contract order eef_pos, # xyz (3,) eef_axis_angle, # orientation (3,) gripper_qpos, # gripper (2,) ]).astype(np.float32), } return data, self.terminated # terminated is a sibling key the framework adds ``` Actions come back the same way: the agent sends them under openpi's `actions` key, and your `step(action)` receives an already-decoded numpy array - you never touch the codec. `RobotEndpoint` is the env's control handle on the bridge - the one surface it drives an episode through. Pass your bridge class (or instance); `start` spawns the sim process and serves it, `capability` publishes the `robot` binding (contract comes from the bridge), `reset` / `result` run the episode. Control-plane only - the agent's observe/act loop tunnels straight to the bridge's WebSocket. ```python theme={"dark"} from hud import Environment from hud.environment.robot import RobotEndpoint env = Environment(name="my-sim") endpoint = RobotEndpoint(MySimBridge()) # start() spawns + serves the bridge process @env.initialize async def _up(): await endpoint.start() env.add_capability(await endpoint.capability()) @env.shutdown async def _down(): await endpoint.stop() @env.template() async def pick_and_place(task_id: str, seed: int = 0): ep = await endpoint.reset(task_id=task_id, seed=seed) # {prompt, token} yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}} yield await endpoint.result(token=ep["token"]) # this slot's {score, success, total_reward} ``` ## Agent side The harness lives in `hud.agents.robot`. We provide a base class called `RobotAgent`. It connects to the `robot` binding, reads the contract, then runs the rollout loop including model inference until the environment terminates. You supply two objects. * **`Model`** - something with an `infer()` function that returns action chunks (pre-/post-processing included) * **`Adapter`** - translates env ↔ model spaces. Run it with the normal engine - `Taskset(...).run(agent, runtime=...)` - against any substrate serving an env with the robot capability and an adaptable embodiment. ## LeRobot integration HUD integrates with [LeRobot](https://github.com/huggingface/lerobot) natively, so a stock checkpoint is a complete agent in a few lines. The two bundled seams *are* the LeRobot convention: * **`LeRobotModel(policy, preprocess, postprocess)`** runs the policy through its own LeRobot pre/post-processors, so the checkpoint behaves exactly as it does upstream. Pass an `Ensembler` to reduce overlapping action chunks to one action per step. * **`LeRobotAdapter(model_image_keys=...)`** maps the env's cameras and state onto the policy's inputs from the [contract](#contract) - HWC `uint8` → CHW float, state and prompt passed through. ```python theme={"dark"} import torch from lerobot.policies.factory import make_pre_post_processors from lerobot.policies.pi05.modeling_pi05 import PI05Policy from hud.agents.robot import RobotAgent, LeRobotModel, LeRobotAdapter class PI05Agent(RobotAgent): def __init__(self): device = "cuda" if torch.cuda.is_available() else "cpu" policy = PI05Policy.from_pretrained("lerobot/pi05_libero_finetuned").to(device).eval() pre, post = make_pre_post_processors(policy.config, "lerobot/pi05_libero_finetuned", preprocessor_overrides={"device_processor": {"device": device}}) self.model = LeRobotModel(policy, pre, post) self.adapter = LeRobotAdapter(model_image_keys=list(policy.config.image_features)) ``` Anything past the stock image/state convention is just a subclass of `Model` or `Adapter`; the LeRobot classes are the batteries-included default. See the [robot benchmark cookbook](/v6/cookbooks/robot-benchmark) for a full LIBERO + pi0.5 run. ## The Model `Model` owns *how to run* a policy. To wrap a non-LeRobot checkpoint, subclass it and implement one method - `infer`; the episode loop, threading, and the wire are handled for you. ```python theme={"dark"} import numpy as np from hud.agents.robot import Model class MyModel(Model): def __init__(self, policy): self.policy = policy def reset(self) -> None: ... # clear per-episode state (optional) def infer(self, batch) -> np.ndarray: chunk = self.policy(batch) # run your policy return np.asarray(chunk, np.float32) # [T, A] chunk, in the env's action space ``` * **Input** (`batch`) - the policy-ready inputs your [`Adapter`](#agent-side) produced for this step (images, a state vector, the task prompt - whatever your policy consumes). `Model` and `Adapter` are a matched pair, so the batch is exactly what your adapter emits. * **Output** - a `[T, A]` `float32` numpy array: an action chunk of `T` timesteps × `A` action dims, already in the env's action space. Single-action policies return `T = 1`. * **`reset()`** - optional; clear per-episode state (an action queue, a chunk buffer) at the start of each episode. The harness awaits `ainfer`, which runs your (blocking) `infer` in a worker thread by default - override `ainfer` only if your policy is natively async. For chunked policies, reduce each `[T, A]` chunk to one action per step with an `Ensembler`. ## Batching Running many rollouts at once, each calling `infer` once per step, wastes the GPU on single-observation forwards. `BatchedAgent` fixes that: it wraps a `RobotAgent`, gives each concurrent rollout its own episode state, and shares one batched model that coalesces their per-step `infer` calls into a single stacked forward. ```python theme={"dark"} from hud.agents.robot.batching import BatchedAgent agent = BatchedAgent(MyRobotAgent()) # sizes itself to the live concurrency job = await taskset.run(agent, runtime=DockerRuntime("hud-libero-env"), max_concurrent=8) ``` Each tick, the concurrent observations queued within a short window (`max_wait_s`, default 50 ms) are stacked into one `[N, ...]` batch, run through a single forward, and scattered back to each rollout - so the batch is as wide as whatever is in flight, and the tail of a suite never stalls. `max_concurrent` already bounds the width; pass `batch_size=` only to cap the forward below it (e.g. VRAM headroom). `BatchedAgent` needs an **in-process, stateless** model whose `infer` runs the whole `[N, ...]` batch in one forward - `LeRobotModel` qualifies. A remote OpenPI policy server is not supported (its protocol has no batched-request shape); run one agent per rollout against it instead. `BatchedAgent` also takes ownership of the agent you pass (it swaps in the batched model in place), so give it a dedicated instance rather than one you also use for unbatched runs. ## Vectorized envs and evals Grading N rollouts by booting N separate containers wastes a GPU sim's whole advantage - each container pays for its own physics from scratch, when a sim like Isaac Sim could instead step all N copies together in one [vectorized](#gym-style-envs) instance. Running an eval against a vectorized sim needs no changes to the agent or the env's contract, only to how the eval provisions rollouts: wrap the runtime in [`Shared(provider, width=N)`](/v6/reference/runtime#shared) so `group` rollouts share that one substrate instead of provisioning `N` containers. ```python theme={"dark"} from hud.eval import Shared, DockerRuntime job = await taskset.run( MyAgent(), runtime=Shared(DockerRuntime("hud-isaac-env"), width=8), group=8, max_concurrent=8, ) ``` `group` and `max_concurrent` are the ordinary [`Taskset.run` parameters](/v6/reference/tasks#running). A few things to note: * **The agent doesn't change.** Each rollout still drives one ordinary WebSocket connection and calls `reset` / `result` as usual - the environment resolves each call to that rollout's own [slot](#gym-style-envs). * **Slots fill in order, then stop.** The first `reset` claims slot 0; once every slot is taken, a further `reset` errors instead of queuing, so set `max_concurrent` to `width` so every slot fills. * **EnvHub sims work unchanged.** A [LeRobot EnvHub](https://huggingface.co/docs/lerobot/en/envhub) package's `make_env` often already returns a vectorized sim, so `env.gym(make_env, num_envs=N)` serves it directly. ## Contract Embodiments and policies disagree on cameras, state layout, action semantics, and control rate, so pairing a model with an env always needs a wiring step. The **contract** makes it explicit: a JSON document in the capability manifest that the agent reads back with `RobotClient.spaces()`, which splits `features` into an observation and an action space by each feature's `role` - so a policy wires itself with no shared config. Here's a proper minimal contract - one camera, a state vector, and an action: ```json theme={"dark"} { "control_rate": 10, "features": { "observation/image": { "role": "observation", "type": "rgb", "names": ["height", "width", "channel"] }, "observation/state": { "role": "observation", "names": ["eef_x", "eef_y", "eef_z", "axis_x", "axis_y", "axis_z", "grip_l", "grip_r"] }, "action": { "role": "action", "names": ["dx", "dy", "dz", "drx", "dry", "drz", "gripper"] } } } ``` `role` and `type` wire the spaces; `control_rate` and `names` are what make the run legible on the platform, so a real contract should always set them: * **`role`** (`observation` / `action`) - `spaces()` splits the contract by it and the `Adapter` wires against that split. Required on every feature. * **`type`** on image observations - `rgb`/`bgr`/`gray`/`depth` is how the bundled adapter spots a camera; the first observation *without* an image type becomes the state. Omit it and your image is mistaken for the state. (On the state and action, `type` is descriptive.) * **`control_rate`** (top level, Hz) - the agent reads it as the playback fps for the recorded episode, so the trace viewer plays back at real speed and any saved dataset has the right frame rate. Set it to your real control rate. * **`names`** (per feature) - per-element labels the trace viewer shows on each state/action slice; without them the dimensions display unlabeled. Feature keys are openpi flat slash-paths and must match *verbatim* the keys your bridge returns from `get_observation` (`action` is the single action feature). Everything else - `robot_type`, `dtype`, `shape`, `stats` - is descriptive and never enforced. Full list in the reference below. | Field | Where | Meaning | | ----------------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `robot_type` | top level | Embodiment id, shown in the trace viewer. Descriptive. | | `control_rate` | top level | Control-loop frequency in Hz; the agent reads it as the playback/record fps. Recommended. | | `features` | top level | Map of feature name → feature spec (rows below). | | `role` | feature | `observation` or `action` - **the only field that splits the spaces**. Load-bearing. | | `type` | feature | Representation tag. Observations: `rgb`/`bgr`/`gray`/`depth` mark an image (load-bearing for the bundled adapter); others (`ee_abs`, `ee_del`, `joint_pos`, …) are descriptive control/state modes. | | `dtype` | feature | `image` for frames, else a numpy dtype (`float32`). Descriptive - not checked against your arrays. | | `shape` | feature | Declared dims (`[H, W, 3]`, `[8]`). Descriptive; every feature is rank ≥ 1 (scalars are `[1]`). | | `names` | feature | Per-element labels; what the trace viewer uses to label state/action slices. | | `stats` | feature | Per-element `mean` / `std` / `min` / `max` for a custom adapter. The stock LeRobot path uses the checkpoint's own normalization, so you can omit it. | | `state_type` / `state_representation` / `frame` | feature | Closed-symbol embodiment metadata (EEF vs joint, quaternion vs axis-angle, world vs base frame). Descriptive. | The v0 schema is deliberately narrow: **one embodiment, one observation space, one action space per contract**. The framework never validates your arrays against `shape` / `dtype`; the full authoring spec - the closed symbol sets and known traps - lives outside the SDK alongside the contract corpus. ## Recording datasets Set `agent.save = True` (wire it to a `--save` flag on your runner) to also record every `(observation, executed action)` tick into a **LeRobot v3 dataset** - the rollouts you just ran, ready to finetune a policy on. Telemetry streams either way; saving is the opt-in extra. Recording is **agent-side**: it consumes the observations and actions the agent already sees, so it runs in *your* process, never the environment container. Only your machine needs `pip install 'lerobot[dataset]'` - the sim's own dependency stack (e.g. Isaac/RoboLab) never enters the picture. All rollouts in a process record into one shared dataset: each buffers its episode and commits it whole, so concurrent runs (e.g. `BatchedAgent`) interleave safely, and the dataset finalizes at process exit. Destination and Hub push come from the environment: | Env var | Effect | | ------------ | ----------------------------------------------------------------------- | | `RECORD_DIR` | Dataset root (default `./data`, relative to where the rollout launched) | | `HF_REPO` | Also push the finalized dataset to this HF namespace (needs `HF_TOKEN`) | | `HF_PRIVATE` | Push the dataset private | The [contract](#contract) drives the schema with no extra wiring: image features become `observation.images.` (encoded to per-episode video), the lone state vector becomes `observation.state`, the action becomes `action`, and the task prompt rides along as each frame's `task`. ## API summary | Symbol | Where | Role | | ------------------------------------------------------ | --------------------------- | ------------------------------------------------------------------------------------------- | | `env.gym(make_env)` / `Gym` | `hud.environment.robot` | Declarative sim: contract, capability, and serving derived from a gym factory | | `wrap(env)` | `hud.environment.robot` | One-line trace streaming for any gym-style env you drive yourself | | `RobotBridge` / `GymBridge` | `hud.environment.robot` | Env-side serve loop (slot barrier + scalar wire); subclass `RobotBridge` for a custom sim | | `RobotEndpoint` | `hud.environment.robot` | Episode bookkeeping + results (`reset` → `{prompt, token}`, `result(token=...)`) | | `RobotEndpoint.capability(...)` | `hud.environment.robot` | Build the `openpi/0` capability after `start()` | | `Capability.robot(name, url, contract)` | `hud.capabilities` | Lower-level constructor (usually via `endpoint.capability`) | | `RobotClient` | `hud.capabilities.robot` | Agent-side wire client (`connect(..., token=)`, `spaces`, `get_observation`, `send_action`) | | `RobotAgent` | `hud.agents.robot` | The harness: one open-loop chunk queue per scalar connection | | `Model` / `LeRobotModel`, `Adapter` / `LeRobotAdapter` | `hud.agents.robot` | Policy + space-translation seams | | `BatchedAgent` / `BatchedModel` | `hud.agents.robot.batching` | Many concurrent rollouts against one shared, batched model | | `Shared(provider, width=N)` | `hud.eval` | Refcounting provider: N concurrent rollouts share one substrate | ## See also LIBERO in Docker, driven by pi0.5, end to end.