Skip to content
MouseMouse

Pre-register the ladder's risks and how each future result will be read - before training

A lesson the Coach cites as preregister-risks-and-fork-readings.
ReplicatedExperiment method

Before a training ladder or risky rung, write the risks, the stop rules, and how every plausible outcome will be interpreted - then do not edit them after seeing results.

Symptom

Without pre-registration, ladder results get rationalized after the fact; the team had already seen post-hoc reads go wrong and adopted written pre-commitment.

Context

The C-ladder execution sheet opens with three numbered pre-registered risks: (1) S2 plant robustness will not carry into omni - a full S2 redo is budgeted from the start; (2) the s1e recipe has a collapse valley at iter 1500+ (recorded twice), so every rung runs a watch_ckpt --every 100 smoke loop with stop-on-degradation; (3) the root's OOD survival edge may partly be "it is slower / commits less" - with the reading fixed in advance: if C1 training raises tracking while survival drops, the edge was bought with slowness, and root selection reopens. Later rungs went further, pre-registering a full result-to-conclusion table for the A/B arms ("预注册读法(事后不改)") and even pre-registering the author's own doubt that a level would fail and what its failure would prove.

Change

Standing practice: before each rung, write down (a) known risks with their mitigations, (b) the interpretation of each possible outcome, (c) stop criteria - all frozen before the run starts ("开训前写死,事后不许改").

Outcome

When arm-A/arm-B and redo results arrived, conclusions were read off the pre-registered table instead of argued; a predicted-likely-FAIL level (C4-redo3) was still run because its pre-registered value was eliminating the regularization hypothesis - which it did.

Mechanism

Pre-commitment converts each training run into a decisive experiment: outcomes falsify or confirm named hypotheses instead of being absorbed into a story; it also makes negative results valuable (a FAIL that eliminates a hypothesis advances the search).

Applies when

  • launching a multi-rung training ladder
  • running an A/B fork whose outcome will drive a fork/root decision
  • a rung is expected to fail but is run for its diagnostic value
“三条预注册风险 … 若 C1 训后跟踪提上去而存活掉下来,说明这条优势是速度买的、不是通用性 → 那时重开选根。”
train/C_LADDER_RUN.md § 0. 三条预注册风险

Principles that cite it