rollouts

Rollouts the caller can see — ones they created, or whose model repo they can write (owner, or a member of the owning org) — newest first. A rollout worker polls this with status=queued to find work; the model repo page lists a policy's past batteries with model_repo=owner/slug. No rollout is public. (M8)

Queues a sim rollout: load model_repo's policy, run it on robot_repo's MJCF in MuJoCo, render a video and trajectory. Cost is capped by horizon_s ≤ 30 (CPU-only in v0).

M8 — the gate. A rollout is a battery of command cells × seeds (battery: smoke | full | an inline spec). Each cell is scored by the source harness's own rule — PASS when every seed stays upright for the whole horizon AND every commanded axis tracks inside [0.85, 1.15] — and the result is written as a scorecard.json whose PASS rows the Training Coach reads as standing constraints. Name a run_id and the terminal report also attaches that scorecard to the run (Run.scorecard_key), so POST /v1/coach/advice on the run sees the gates without anyone copying anything. The 720p mp4 per cell is the by-product. Both repos must be visible to the caller and of the right kind (a model and a robot whose card names an mjcf_path); an invisible repo is 404, a wrong kind 409, an invisible run_id 422.

Nothing executes server-side: the rollout waits queued until a rollout worker (lucen rollout worker, or lucen rollout run --local which claims its own) takes it through claimRollout / presignRolloutArtifacts / reportRollout. A CPU rollout is unpriced (cost_usd: 0, priced: false).

C2 — the founder's harness, in the cloud. The battery may be one of the founder's (c_matrix, robust, recovery, watch) or carry his sim2sim conditions (conditions: delay, friction, kp/kd scale, push, power scale, heading, the bridge slew, jitter, a fallen start); every cell's seed 0 is rendered from his three fixed views. executor: modal has the hub spawn the deployed rollout function after the commit (modal_call_id; 503 naming MODAL_APP_NAME when the backend is not configured, with nothing recorded). checkpoint_step rolls out checkpoints/step_<N>/ instead of the root policy.

Hosted execution (L0). executor: modal runs on the hub's own Modal account, whose container claims the rollout with the operator's key; so it needs the model's owner in the deployment's HOSTED_OWNERS (default lucen; * = everyone) and write access to the model repo — reading a public model is enough for a worker rollout, never for a hosted one. Either refusal is 403 with errors[].type: forbidden.hosted_owner and records nothing.

Rollout status and, when succeeded, the storage keys of the rendered video and trajectory — plus, since M8, presigned 1-hour playback URLs (video_url, traj_url, scorecard_url, and one pair per cell), the per-cell verdicts and the battery that was run. Visibility is the creator's, or anyone who can write the model repo; anything else is 404.

Compare-and-set queued → running, exactly like claimRun: one caller wins, everyone else gets 409 naming the holder. Records claimed_by, claimed_at, started_at, heartbeat_at, the claiming credential (presignRolloutArtifacts and reportRollout are 409 from any other — one key per worker) and an audit row, and returns the rollout with its resolved battery_spec — the cells, seeds and horizon the worker must execute. The model and robot repos are what the worker pulls next (the policy ONNX and its io_contract, the MJCF and its meshes). key:train: a worker spends compute on the creator's behalf. No lease (M7-RL's rule): a dead worker leaves the rollout running until a human acts. (M8)

Bytes never stream through the API (docs/ARCHITECTURE.md), and a battery's videos are megabytes each — so a worker asks for presigned PUT URLs here and uploads straight to object storage, one object per artifact, under rollouts/{rollout_id}/… (outside the content-addressed blobs/ prefix M3's GC sweeps). Names are repo-style relative paths (cells/vx-plus-0p30/video.mp4, cells/vx-plus-0p30/trajectory.json, trajectory.json); the keys returned are what reportRollout then names in cells[]. Only a running rollout may presign, and only the credential that claimed it (409 otherwise); URLs live 1 h. (M8)

Storage (L0). The declared sizes count toward the rollout creator's storage quota (STORAGE_QUOTA_BYTES, the same 50 GiB as repo files): over it is 413 and no URL is issued.

The worker's one write path after claimRollout; the rollout must be running and the caller the credential that claimed it (409 otherwise). Every field is optional: log_chunk (≤ 256 KiB) is stored as one object under rollouts/{id}/logs/; cells[] carries the measurements — seeds run, seeds that stayed upright, the tracking ratio per commanded axis, the artifact keys presigned earlier — and warnings[] / engine say what the worker could not do (a missing renderer, an unsupported observation). The hub scores, not the worker: on a terminal succeeded report the hub applies the harness rule to the measurements, writes scorecard.json (gates in the shape coach/gates.py reads, names that survive its normalisation — vy+0.10 and vy-0.10 stay distinct), sets scorecard_key / video_key / traj_key, records cost_usd 0 (priced: false), and — when the rollout carries a run_id — copies the scorecard to runs/{run_id}/scorecard.json, sets Run.scorecard_key and writes a run.scorecard audit row. status may only move running → succeeded | failed. A key named in cells[] that is not under this rollout's prefix, or that holds no object, is 422. (M8)

A queued or running rollout becomes canceled in this request (C2). A modal rollout's call is terminated first (502 if Modal refuses, and nothing changes); a worker rollout's worker is told on its next report, which is 409 — a rollout keeps no lease, so there is no flag to wait on. Nothing is billed either way (CPU rollouts are unpriced). A terminal rollout is 409. Writes a rollout.cancel audit row.

L1 — a gate while training renders at most a poster per cell (video: poster or none); this queues the videos afterwards. The new rollout (purpose: render, render_of = this one) runs the same model repo, checkpoint, robot, battery conditions and seed with seeds: 1 and video: full (or poster), so seed 0 of each chosen cell is simulated again and rendered — the engine is deterministic for a seed, and the hub checks the replay against the measured episode (render_check). It carries the source's run_id, never counts as a gate, a scan or the run's scorecard, and is unpriced like any CPU rollout. The source must be succeeded (409 otherwise); a cell name the source battery does not have is 422. Same executor rules as createRollout.

Hosted execution (L0). executor: modal runs on the hub's own Modal account, whose container claims the rollout with the operator's key; so it needs the model's owner in the deployment's HOSTED_OWNERS (default lucen; * = everyone) and write access to the model repo — reading a public model is enough for a worker rollout, never for a hosted one. Either refusal is 403 with errors[].type: forbidden.hosted_owner and records nothing.

Step × cells × verdict for the gate rollouts the hub queued from the run's Recipe.gate (C2), oldest step first, plus every checkpoint step the output repo holds and the tripwire state — which rule fired, at which step, on which rollout. A run with no gate answers an empty table. L1: each row says what its gate rendered (video), carries a presigned poster per cell and one for the row when it rendered posters, and the latest renderRollout of it.

Step × cells × verdict for the run's latest band scan (or scan_id), oldest step first (C2). An empty table when the run was never scanned.

One rollout per checkpoints/step_<N>/ of the run's output model repo in the requested band, each battery × seeds (default watch × 20 — the founder's 20-seed rule, because a few seeds differ between machines), on robot_repo (default recipe.robot), all under one scan_id (C2). 409 when the run has no output repo or no checkpoint in the band; 422 for a band with more than 50 checkpoints. Same executor rules as createRollout.

Hosted execution (L0). executor: modal runs on the hub's own Modal account, whose container claims the rollout with the operator's key; so it needs the model's owner in the deployment's HOSTED_OWNERS (default lucen; * = everyone) and write access to the model repo — reading a public model is enough for a worker rollout, never for a hosted one. Either refusal is 403 with errors[].type: forbidden.hosted_owner and records nothing.

Why a rollout exists (C2): manual (someone asked; only these attach their scorecard to run_id), gate (queued by the hub for a checkpoint of a running run's Recipe.gate) or scan (one checkpoint of a band scan). L1 adds render — the videos of an earlier rollout rendered afterwards (renderRollout: seed 0 of its cells replayed with video: full); it never counts as a gate, a scan or the run's scorecard.

One progress or terminal report from a rollout worker. Every field is optional.

L1 — render an earlier rollout's videos now. The hub queues a purpose: render rollout of the same model repo, checkpoint, robot, battery and seed with seeds: 1 and video (default full), so seed 0 of each chosen cell is simulated again and rendered. The engine is deterministic for a seed, so the replay is the episode that was measured; the hub checks it — seed 0's upright / fall time / tracking against the source's — and says so in Rollout.render_check.

Step × cells × verdict (C2): the gates a running run's checkpoints got (getRunGates) or one band scan (scanRun / getRunScan), oldest step first.

A band scan (C2, the founder's 20-seed pick of 1–2 checkpoints): one rollout per checkpoints/step_<N>/ of the run's output model repo in [from_step, to_step] (or exactly steps), each battery × seeds.

A battery: cells × seeds × horizon. The named batteries: smoke (stand, vx+0.30, vy+0.10, vy-0.10, yaw+0.50 × 2 seeds × 5 s) and full (the source harness's 13-cell C-matrix × 20 seeds × 20 s). M17b: when the robot card's interface says base: fixed, velocity cells are meaningless, so smoke resolves to the fixed-base smoke battery instead — hold, jt+0.30, jt-0.30 (every joint offset by ±0.30 rad from the default pose) × 2 seeds × 5 s — and full is 422. The resolved cells are always returned as Rollout.battery_spec. C2 adds the founder's harness batteries (legged, floating base only — 422 on a fixed base), each measured under protocol: sim2sim: c_matrix (eval_c_matrix.py: the 13 cells × 20 seeds × 20 s at delay 2, joint jitter 0.05, the bridge slew), robust (eval_matrix.py: vx+0.25 gated on tracking, then ground μ 0.4 / 0.6 / 0.8 / 1.0, lateral pushes ±4 / ±6 N·s at 8 s and 12 s and the zero command, each gated on survival — 14 cells × 20 seeds × 20 s at the same delay/jitter/slew), recovery (the four fallen categories supine / prone / side / mid × 20 seeds × 10 s at delay 2 with the slew) and watch (watch_ckpt.py's smoke gate: vx+0.25 × 3 seeds × 20 s at delay 2 — what gating during training runs). C3 adds stand: one zero-command cell scored on posture (score: posture) × 20 seeds × 20 s under the same sim2sim conditions — what a stand campaign gates and scans on.

The founder's sim2sim conditions (C2), ported from tools/sim2sim.py flag by flag; absent keys are M8's plant. On a battery they apply to every cell; on a cell they override the battery's key by key; on RolloutCreate they override a named battery's own. The worker applies them; the hub still scores.

What the worker measured for one cell. status/detail are filled in by the hub when it scores the terminal report; a worker leaves them out.

L1 — does a render's replay of seed 0 match the episode the source rollout measured? Compared per cell on the hub, once the render finished: upright, the fall time (±1 control step) and each tracked axis (±0.5 percentage points).

The only transitions a worker may make — running → succeeded | failed.

P1 — the checkpoint a table's "best N / M at step" names. Ordered by: more cells passed; then more of the owning round's locked gates passed at that step; then the higher tracking on the round's declared axis (the mean over the cells commanding that axis, an overshoot counted as its mirror below 100 %); then the later step.

One command cell: by default (kind: velocity) the body-frame velocity command held for the whole horizon. vx/vy in m/s, yaw in rad/s. A cell with every command zero is the stand test; only non-zero axes are tracked. M17b adds kind — joint_target and reach cells hold a target instead, given as an OFFSET from the robot's default pose so one battery is portable across robots.

How a battery is measured (C2). m8 — M8's definitions: the robot's start pose, upright = never below 40% of the start height nor tilted past 60°, tracking the mean over every seed. sim2sim — the founder's harness: the standing keyframe with the policy's nominal joints, alive = the episode ends above 0.30 m and tilted under 25° (never below 0.25 m on the way; fractions of the standing height on another robot), the yaw rate from the net heading change, and tracking the median over the seeds that stayed alive (eval_c_matrix.py).

--push N_S AT_S: at at_s seconds, impulse / total mass is added to the base's world velocity. (C2)

--init-fallen: every seed starts fallen (C2). Seeds are allocated to the categories by their weights, stratified (20 seeds over {side_l: 1, side_r: 1} are exactly 10 + 10), and seed k of a category is the founder's sim2sim.py --init-fallen <category> --seed k: the category's roll/pitch ± jitter_deg, yaw anywhere, dropped from 0.28–0.40 m (scaled to the robot's standing height), joints uniform inside the soft limits, small random velocities. mid is a random-axis tilt of 50°–125°.

One seed of one cell, as measured.

What a cell commands, and so how it is scored (M17b). velocity — the default, and every cell before M17b — holds a body-frame velocity and PASSes on upright + tracking in [0.85, 1.15]. joint_target and reach hold a joint-space / end-effector target and PASS when every seed's settle error (mean over the last 20% of the horizon) is within tolerance AND no control step left the soft joint limits. On a fixed base upright is true by construction, so every kind — including velocity, where a zero command means "hold the default pose" — is scored on settle error and limit violations instead, and a non-zero velocity command cannot PASS. C2 adds recovery: every seed starts fallen (conditions.init_pose), the episode never ends early, and the cell PASSes on the founder's stand-up rule (RECOVERY_V0_SPEC §5): at least min_success of the seeds stand up (tilt < 15°, base above 0.85 of the standing height, both feet on the ground, |ω| < 1 rad/s, held 0.5 s), the median stand-up time is ≤ 5 s, ≤ 10% of those that stood fall again (the last 0.5 s must hold), and the median peak PD torque demand is ≤ 90% of each joint's effort limit.

M17b. What a joint_target / reach cell — or any cell on a fixed base — measured: settle_error (mean over the last 20% of the horizon), tracking_error (mean after a 1 s settle) and limit_violations (control steps with any joint outside its soft limit). Errors are the worst joint in radians, or metres for reach. A value is null when it could not be measured (the episode ended before the window) — never 0, which would read as perfect; the hub scores a null as FAIL. On a cell, errors are the worst seed and limit_violations the sum. C3 adds the posture metrics a posture cell (and every sim2sim episode on a floating base) measures: tilt_max_deg (the largest angle between the base's z axis and the world's over the episode), asym_max_deg (the worst mirrored joint pair's settled |mean(q_l − s·q_r)|, degrees), asym_pairs (how many pairs were found), drift_m (horizontal distance moved); on a cell, each is the median over the seeds. C3b adds, on an episode with a push: recover_s (seconds from the push until the base stayed within 3° of upright and under 5 cm/s across the floor to the end; the whole window left after the push when it fell or never settled) and recovered (1 when it settled, else 0).

L1 — the latest render of a gate rollout (renderRollout).

How a velocity cell on a floating base is scored (C2). track (the default) is M8's rule — every seed upright AND every commanded axis inside [0.85, 1.15]. survive is eval_matrix.py's rule for its friction, push and zero-command blocks — every seed alive, tracking reported but not gated. posture (C3) is the stand battery's rule — every seed upright AND, as the median over the seeds, the maximum base tilt, the worst mirrored joint pair's left/right asymmetry and the drift across the floor each within posture_limits (defaults 3.0°, 2.0°, 0.25 m): a stand is judged by its posture, never by survival.

C3 — a posture cell's bars, each on the median over the seeds. Null or absent means the hub's default. Set by the hub or the battery, never by the worker.

The founder's fallen start categories (sim2sim.py --init-fallen). (C2)