HUD Documentation - Evaluations and RL Environments.

The robot capability is in beta. The wire protocol is versioned openpi/0; the contract schema is v0. Expect additive changes while the design settles.

HUD runs robot environments the same way it runs everything else - an environment declares tasks and capabilities, an agent drives a live Run, but a 50 Hz policy can’t stream actions over tool calls. So the robot capability is instead a continuous observation/action loop over WebSocket: the environment streams observations (camera frames, robot state) and the agent streams back actions, as fast as the policy can run. The wire format is openpi-inspired (msgpack with numpy serialization), so existing openpi policy servers only need a thin adapter. Everything below ships behind the robot extra (pulls in numpy + openpi-client):

uv add 'hud[robot]'

pip install 'hud[robot]'

Overview

Like with other HUD workflows there’s the environment side and the agent side. For robotics the environment side translates incoming actions into changes in the digital or physical environment and serves observations. The agent side owns the policy: it reads those observations, runs inference, and sends actions back. Both sides need building, and this is where robotics differs from the rest of HUD. For LLM agents you can lean on a standard inference provider and a stock harness, so often the environment is the only thing you write. For robot policies there is no equivalent - no hosted inference provider, no standard harness. HUD ships tooling for both sides: a handful of small, named abstractions you implement, with the framework owning everything in between (the serve loop, the wire protocol, telemetry to platform). Environment side - owns the simulator and serves frames:

RobotBridge - the one class you implement around your sim: reset / step / get_observation. The framework owns the WebSocket serve loop and the single-agent connection.
RobotEndpoint - wraps the bridge - the environment server’s handle for the sim (even if the sim is running in another process)

Agent side - runs the policy and streams actions:

RobotAgent - the harness: connects to the env and bridge, owns adapter and model, drives model until env terminates.
Model - the actual stateless checkpoint of the model (includes pre-/post-processing)
Adapter - translates the env’s observation space to the model’s, and the model’s action space to the env’s

The contract (of the environment) - the one artifact both sides share: a self-describing JSON schema of the environment including the embodiment’s control rate, observation and action spaces, carried in the capability’s manifest params. The agent wires observations to policy inputs purely from the manifest; there is no shared config. In addition, the contract serves to define useful metadata that you can easily toggle to improve trace visualization on the platform.

Environment side

HUD has two ways to build the environment side: env.gym for anything gym-shaped, which derives the contract, capability, and serving for you, or a custom RobotBridge for a sim that isn’t.

Gym-style envs

Calling env.gym(make_env) derives the contract from a sample observation, serves the env over the robot WebSocket, and returns a RobotEndpoint - the handle templates drive episodes through:

env.py

from hud import Environment

env = Environment(name="my-sim")
sim = env.gym(make_env)   # make_env: any callable returning a gym-style env

@env.template()
async def pick_and_place(task: str = "default", seed: int = 0):
    ep = await sim.reset(task=task, seed=seed)  # {prompt, token}
    yield {"prompt": ep["prompt"]}
    yield await sim.result()

env.gym takes one argument, in any of three forms:

A factory callable, like make_env above - a module-level function that returns a gym-style env. Its signature splits task args into build args (a change rebuilds the env) and episodic args, which flow to env.reset(seed=..., options=...).
A gymnasium registry id, a string such as "CartPole-v1" - built with gym.make (or gym.make_vec when num_envs is passed).
An already-constructed env made through the gymnasium registry - env.gym reduces it to its spec and rebuilds it in the sim process.

env.gym serves either a standard, single-instance env or a vectorized one - one gym env that internally holds num_envs copies of the environment, stepped together in the same process for GPU efficiency. Each copy is a slot, and a vectorized sim hands back a slot token on reset so the rollout that claimed it can step and grade that copy alone, leaving the rest untouched. A single-env sim has only one slot, so the template may omit the token, as above. A vectorized sim (num_envs > 1) must thread it through so each session grades its own slot - yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}} and call sim.result(token=ep["token"]); see vectorized envs and evals for the full pattern. You can find gym-shaped environments in various places - LeRobot EnvHub, including Isaac Lab Arena environments, or environments built into LeRobot directly.

Example: Isaac Lab Arena (EnvHub)

Vectorized (num_envs=4), with cameras enabled for RTX frames. make_env downloads the EnvHub repo’s module and calls its make_env the same way LeRobot’s loader does.

env.py

from __future__ import annotations
import os
from hud import Environment

HUB = "nvidia/isaaclab-arena-envs"

def make_env(num_envs: int = 4):
    from lerobot.envs.configs import IsaaclabArenaEnv
    from lerobot.envs.utils import (
        _call_make_env,
        _download_hub_file,
        _import_hub_module,
        _normalize_hub_result,
    )

    cfg = IsaaclabArenaEnv(
        hub_path=HUB,
        environment="gr1_microwave",
        embodiment="gr1_pink",
        object="mustard_bottle",
        headless=True,
        enable_cameras=True,  # required with headless for RTX frames
        camera_keys="robot_pov_cam_rgb",
        num_envs=num_envs,
        episode_length=50,
        state_keys="robot_joint_pos",
        state_dim=54,
        action_dim=36,  # GR1
        task="Reach out to the microwave and open it.",
    )
    repo_id, _, local_file, _ = _download_hub_file(cfg.hub_path, True, None)
    module = _import_hub_module(local_file, repo_id)
    suites = _normalize_hub_result(_call_make_env(module, num_envs, False, cfg))
    return next(iter(next(iter(suites.values())).values()))


env = Environment(name="arena")
sim = env.gym(make_env, num_envs=4, fps=30)


@env.template()
async def episode(seed: int = 0):
    ep = await sim.reset(seed=seed)
    # Vectorized: token pins this rollout's slot.
    yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}}
    yield await sim.result(token=ep["token"])

Example: Meta-World (built into LeRobot)

Meta-World ships inside LeRobot itself rather than as an EnvHub repo, so make_env calls LeRobot’s own factory directly. GymBridge keeps the env’s pixels and agent_pos keys as-is.

metaworld_sim.py

from __future__ import annotations

import os

os.environ.setdefault("MUJOCO_GL", "egl")


def make_env(task: str = "assembly-v3"):
    from lerobot.envs.configs import MetaworldEnv
    from lerobot.envs.factory import make_env as lerobot_make_env

    # Unwrap vec-of-one; pixels_agent_pos = camera + 4-D eef/gripper state.
    suites = lerobot_make_env(MetaworldEnv(task=task, obs_type="pixels_agent_pos"), n_envs=1)
    suite = next(iter(suites))
    task_id = next(iter(suites[suite]))
    return suites[suite][task_id].envs[0]

env.py

from metaworld_sim import make_env

from hud import Environment

env = Environment(name="metaworld")
# LeRobot hardcodes render_fps=80; Meta-World control is slower - pin playback.
sim = env.gym(make_env, fps=20)


@env.template()
async def episode(task: str = "assembly-v3", seed: int = 0):
    """One Meta-World episode; task name is a build arg (e.g. assembly-v3, dial-turn-v3)."""
    ep = await sim.reset(task=task, seed=seed)
    yield {"prompt": ep["prompt"]}
    yield await sim.result()

Needs metaworld / lerobot[metaworld] installe.

Custom bridges

For a sim that isn’t gym-shaped, subclass RobotBridge instead:

from hud.environment.robot import RobotBridge

class MySimBridge(RobotBridge):
    def reset(self, task_id: str, seed: int = 0) -> str:
        ...                              # build the episode (runs on the sim thread)
        return self.task_description     # becomes the task prompt

    def step(self, action) -> None:
        ...  # advance one tick; set success / terminated

    def get_observation(self):
        # Always [N, ...] arrays + [N] terminated (N=1 is fine).
        return {"agentview_image": frames, "state": vecs}, done_mask

Those three methods are all you write. Under the hood the framework takes care of communication with the agent and starting/stopping as well as stepping of the simulator at the control rate. All three hooks run on the sim thread (main under serve_bridge), so thread-affine sims such as Isaac are safe without extra hopping in your subclass.

reset starts a fresh episode for a task and returns its prompt (the text the agent is given).
step applies one action and advances the sim a tick, setting success / terminated as the episode plays out.
get_observation returns a structured dict of the current observation plus whether the episode is done.

The get_observation function has a strict output convention, see below to follow it.

The openpi observation convention

The data dict is the strict part. It is what the agent indexes by name and feeds straight to the policy, so a few things have to be exactly right:

Values are numpy arrays - nothing else survives the trip into the adapter and the trace viewer.
Each key is an observation feature’s name, verbatim from the contract. The agent does data[name] directly off the contract
Images are HWC arrays ([H, W, 3], uint8 RGB).
State is a single 1-D array, passed to the policy as float32; everything rank-1 is treated as state.
terminated is a sibling, not part of data - return it as the second item of your (data, terminated) tuple and the framework attaches it to the frame.

def get_observation(self):
    data = {
        "observation/image":       rgb,          # [256, 256, 3] uint8, RGB, HWC
        "observation/wrist_image": wrist_rgb,    # [256, 256, 3] uint8, RGB, HWC
        "observation/state": np.concatenate([    # [8] float32, in contract order
            eef_pos,         # xyz                 (3,)
            eef_axis_angle,  # orientation         (3,)
            gripper_qpos,    # gripper             (2,)
        ]).astype(np.float32),
    }
    return data, self.terminated   # terminated is a sibling key the framework adds

Actions come back the same way: the agent sends them under openpi’s actions key, and your step(action) receives an already-decoded numpy array - you never touch the codec.

RobotEndpoint is the env’s control handle on the bridge - the one surface it drives an episode through. Pass your bridge class (or instance); start spawns the sim process and serves it, capability publishes the robot binding (contract comes from the bridge), reset / result run the episode. Control-plane only - the agent’s observe/act loop tunnels straight to the bridge’s WebSocket.

from hud import Environment
from hud.environment.robot import RobotEndpoint

env = Environment(name="my-sim")
endpoint = RobotEndpoint(MySimBridge())  # start() spawns + serves the bridge process

@env.initialize
async def _up():
    await endpoint.start()
    env.add_capability(await endpoint.capability())

@env.shutdown
async def _down():
    await endpoint.stop()

@env.template()
async def pick_and_place(task_id: str, seed: int = 0):
    ep = await endpoint.reset(task_id=task_id, seed=seed)  # {prompt, token}
    yield {"prompt": ep["prompt"], "robot": {"token": ep["token"]}}
    yield await endpoint.result(token=ep["token"])  # this slot's {score, success, total_reward}

Agent side

The harness lives in hud.agents.robot. We provide a base class called RobotAgent. It connects to the robot binding, reads the contract, then runs the rollout loop including model inference until the environment terminates. You supply two objects.

Model - something with an infer() function that returns action chunks (pre-/post-processing included)
Adapter - translates env ↔ model spaces.

Run it with the normal engine - Taskset(...).run(agent, runtime=...) - against any substrate serving an env with the robot capability and an adaptable embodiment.

LeRobot integration

HUD integrates with LeRobot natively, so a stock checkpoint is a complete agent in a few lines. The two bundled seams are the LeRobot convention:

LeRobotModel(policy, preprocess, postprocess) runs the policy through its own LeRobot pre/post-processors, so the checkpoint behaves exactly as it does upstream. Pass an Ensembler to reduce overlapping action chunks to one action per step.
LeRobotAdapter(model_image_keys=...) maps the env’s cameras and state onto the policy’s inputs from the contract - HWC uint8 → CHW float, state and prompt passed through.

import torch
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.pi05.modeling_pi05 import PI05Policy

from hud.agents.robot import RobotAgent, LeRobotModel, LeRobotAdapter

class PI05Agent(RobotAgent):
    def __init__(self):
        device = "cuda" if torch.cuda.is_available() else "cpu"
        policy = PI05Policy.from_pretrained("lerobot/pi05_libero_finetuned").to(device).eval()
        pre, post = make_pre_post_processors(policy.config, "lerobot/pi05_libero_finetuned",
                                             preprocessor_overrides={"device_processor": {"device": device}})
        self.model = LeRobotModel(policy, pre, post)
        self.adapter = LeRobotAdapter(model_image_keys=list(policy.config.image_features))

Anything past the stock image/state convention is just a subclass of Model or Adapter; the LeRobot classes are the batteries-included default. See the robot benchmark cookbook for a full LIBERO + pi0.5 run.

The Model

Model owns how to run a policy. To wrap a non-LeRobot checkpoint, subclass it and implement one method - infer; the episode loop, threading, and the wire are handled for you.

import numpy as np
from hud.agents.robot import Model

class MyModel(Model):
    def __init__(self, policy):
        self.policy = policy

    def reset(self) -> None:
        ...                                    # clear per-episode state (optional)

    def infer(self, batch) -> np.ndarray:
        chunk = self.policy(batch)             # run your policy
        return np.asarray(chunk, np.float32)   # [T, A] chunk, in the env's action space

Input (batch) - the policy-ready inputs your Adapter produced for this step (images, a state vector, the task prompt - whatever your policy consumes). Model and Adapter are a matched pair, so the batch is exactly what your adapter emits.
Output - a [T, A] float32 numpy array: an action chunk of T timesteps × A action dims, already in the env’s action space. Single-action policies return T = 1.
reset() - optional; clear per-episode state (an action queue, a chunk buffer) at the start of each episode.

The harness awaits ainfer, which runs your (blocking) infer in a worker thread by default - override ainfer only if your policy is natively async. For chunked policies, reduce each [T, A] chunk to one action per step with an Ensembler.

Batching

Running many rollouts at once, each calling infer once per step, wastes the GPU on single-observation forwards. BatchedAgent fixes that: it wraps a RobotAgent, gives each concurrent rollout its own episode state, and shares one batched model that coalesces their per-step infer calls into a single stacked forward.

from hud.agents.robot.batching import BatchedAgent

agent = BatchedAgent(MyRobotAgent())   # sizes itself to the live concurrency

job = await taskset.run(agent, runtime=DockerRuntime("hud-libero-env"), max_concurrent=8)

Each tick, the concurrent observations queued within a short window (max_wait_s, default 50 ms) are stacked into one [N, ...] batch, run through a single forward, and scattered back to each rollout - so the batch is as wide as whatever is in flight, and the tail of a suite never stalls. max_concurrent already bounds the width; pass batch_size= only to cap the forward below it (e.g. VRAM headroom). BatchedAgent needs an in-process, stateless model whose infer runs the whole [N, ...] batch in one forward - LeRobotModel qualifies. A remote OpenPI policy server is not supported (its protocol has no batched-request shape); run one agent per rollout against it instead. BatchedAgent also takes ownership of the agent you pass (it swaps in the batched model in place), so give it a dedicated instance rather than one you also use for unbatched runs.

Vectorized envs and evals

Grading N rollouts by booting N separate containers wastes a GPU sim’s whole advantage - each container pays for its own physics from scratch, when a sim like Isaac Sim could instead step all N copies together in one vectorized instance. Running an eval against a vectorized sim needs no changes to the agent or the env’s contract, only to how the eval provisions rollouts: wrap the runtime in Shared(provider, width=N) so group rollouts share that one substrate instead of provisioning N containers.

from hud.eval import Shared, DockerRuntime

job = await taskset.run(
    MyAgent(),
    runtime=Shared(DockerRuntime("hud-isaac-env"), width=8),
    group=8,
    max_concurrent=8,
)

group and max_concurrent are the ordinary Taskset.run parameters. A few things to note:

The agent doesn’t change. Each rollout still drives one ordinary WebSocket connection and calls reset / result as usual - the environment resolves each call to that rollout’s own slot.
Slots fill in order, then stop. The first reset claims slot 0; once every slot is taken, a further reset errors instead of queuing, so set max_concurrent to width so every slot fills.
EnvHub sims work unchanged. A LeRobot EnvHub package’s make_env often already returns a vectorized sim, so env.gym(make_env, num_envs=N) serves it directly.

Contract

Embodiments and policies disagree on cameras, state layout, action semantics, and control rate, so pairing a model with an env always needs a wiring step. The contract makes it explicit: a JSON document in the capability manifest that the agent reads back with RobotClient.spaces(), which splits features into an observation and an action space by each feature’s role - so a policy wires itself with no shared config. Here’s a proper minimal contract - one camera, a state vector, and an action:

{
  "control_rate": 10,
  "features": {
    "observation/image": {
      "role": "observation",
      "type": "rgb",
      "names": ["height", "width", "channel"]
    },
    "observation/state": {
      "role": "observation",
      "names": ["eef_x", "eef_y", "eef_z", "axis_x", "axis_y", "axis_z", "grip_l", "grip_r"]
    },
    "action": {
      "role": "action",
      "names": ["dx", "dy", "dz", "drx", "dry", "drz", "gripper"]
    }
  }
}

role and type wire the spaces; control_rate and names are what make the run legible on the platform, so a real contract should always set them:

role (observation / action) - spaces() splits the contract by it and the Adapter wires against that split. Required on every feature.
type on image observations - rgb/bgr/gray/depth is how the bundled adapter spots a camera; the first observation without an image type becomes the state. Omit it and your image is mistaken for the state. (On the state and action, type is descriptive.)
control_rate (top level, Hz) - the agent reads it as the playback fps for the recorded episode, so the trace viewer plays back at real speed and any saved dataset has the right frame rate. Set it to your real control rate.
names (per feature) - per-element labels the trace viewer shows on each state/action slice; without them the dimensions display unlabeled.

Feature keys are openpi flat slash-paths and must match verbatim the keys your bridge returns from get_observation (action is the single action feature). Everything else - robot_type, dtype, shape, stats - is descriptive and never enforced. Full list in the reference below.

Full field reference

Field	Where	Meaning
`robot_type`	top level	Embodiment id, shown in the trace viewer. Descriptive.
`control_rate`	top level	Control-loop frequency in Hz; the agent reads it as the playback/record fps. Recommended.
`features`	top level	Map of feature name → feature spec (rows below).
`role`	feature	`observation` or `action` - the only field that splits the spaces. Load-bearing.
`type`	feature	Representation tag. Observations: `rgb`/`bgr`/`gray`/`depth` mark an image (load-bearing for the bundled adapter); others (`ee_abs`, `ee_del`, `joint_pos`, …) are descriptive control/state modes.
`dtype`	feature	`image` for frames, else a numpy dtype (`float32`). Descriptive - not checked against your arrays.
`shape`	feature	Declared dims (`[H, W, 3]`, `[8]`). Descriptive; every feature is rank ≥ 1 (scalars are `[1]`).
`names`	feature	Per-element labels; what the trace viewer uses to label state/action slices.
`stats`	feature	Per-element `mean` / `std` / `min` / `max` for a custom adapter. The stock LeRobot path uses the checkpoint’s own normalization, so you can omit it.
`state_type` / `state_representation` / `frame`	feature	Closed-symbol embodiment metadata (EEF vs joint, quaternion vs axis-angle, world vs base frame). Descriptive.

The v0 schema is deliberately narrow: one embodiment, one observation space, one action space per contract. The framework never validates your arrays against shape / dtype; the full authoring spec - the closed symbol sets and known traps - lives outside the SDK alongside the contract corpus.

Recording datasets

Set agent.save = True (wire it to a --save flag on your runner) to also record every (observation, executed action) tick into a LeRobot v3 dataset - the rollouts you just ran, ready to finetune a policy on. Telemetry streams either way; saving is the opt-in extra. Recording is agent-side: it consumes the observations and actions the agent already sees, so it runs in your process, never the environment container. Only your machine needs pip install 'lerobot[dataset]' - the sim’s own dependency stack (e.g. Isaac/RoboLab) never enters the picture. All rollouts in a process record into one shared dataset: each buffers its episode and commits it whole, so concurrent runs (e.g. BatchedAgent) interleave safely, and the dataset finalizes at process exit. Destination and Hub push come from the environment:

Env var	Effect
`RECORD_DIR`	Dataset root (default `./data`, relative to where the rollout launched)
`HF_REPO`	Also push the finalized dataset to this HF namespace (needs `HF_TOKEN`)
`HF_PRIVATE`	Push the dataset private

The contract drives the schema with no extra wiring: image features become observation.images.<camera> (encoded to per-episode video), the lone state vector becomes observation.state, the action becomes action, and the task prompt rides along as each frame’s task.

API summary

Symbol	Where	Role
`env.gym(make_env)` / `Gym`	`hud.environment.robot`	Declarative sim: contract, capability, and serving derived from a gym factory
`wrap(env)`	`hud.environment.robot`	One-line trace streaming for any gym-style env you drive yourself
`RobotBridge` / `GymBridge`	`hud.environment.robot`	Env-side serve loop (slot barrier + scalar wire); subclass `RobotBridge` for a custom sim
`RobotEndpoint`	`hud.environment.robot`	Episode bookkeeping + results (`reset` → `{prompt, token}`, `result(token=...)`)
`RobotEndpoint.capability(...)`	`hud.environment.robot`	Build the `openpi/0` capability after `start()`
`Capability.robot(name, url, contract)`	`hud.capabilities`	Lower-level constructor (usually via `endpoint.capability`)
`RobotClient`	`hud.capabilities.robot`	Agent-side wire client (`connect(..., token=)`, `spaces`, `get_observation`, `send_action`)
`RobotAgent`	`hud.agents.robot`	The harness: one open-loop chunk queue per scalar connection
`Model` / `LeRobotModel`, `Adapter` / `LeRobotAdapter`	`hud.agents.robot`	Policy + space-translation seams
`BatchedAgent` / `BatchedModel`	`hud.agents.robot.batching`	Many concurrent rollouts against one shared, batched model
`Shared(provider, width=N)`	`hud.eval`	Refcounting provider: N concurrent rollouts share one substrate

Robot benchmark cookbook

LIBERO in Docker, driven by pi0.5, end to end.

Start here

Guides

Reference

Advanced

Cookbooks

More

Robots

Overview

Environment side

Gym-style envs

Custom bridges

Agent side

LeRobot integration

The Model

Batching

Vectorized envs and evals

Contract

Recording datasets

API summary

See also

Robot benchmark cookbook

Capabilities

​Overview

​Environment side

​Gym-style envs

​Custom bridges

​Agent side

​LeRobot integration

​The Model

​Batching

​Vectorized envs and evals

​Contract

​Recording datasets

​API summary

​See also

Robot benchmark cookbook

Capabilities

Overview

Environment side

Gym-style envs

Custom bridges

Agent side

LeRobot integration

The Model

Batching

Vectorized envs and evals

Contract

Recording datasets

API summary

See also