The experiment
Roster built itself with several agent setups, reviewed its own sessions, and promoted one setup. Before that review ran, we froze a backlog of new tasks and their acceptance tests. This page reports the forward run: every frozen task, run through both the demoted and the promoted setup, graded against those frozen tests. The hypotheses, tests and stop rules are fixed in PREREGISTRATION.md.
Pairs
| Task | Old setup | New setup | Old cost | New cost | Excluded |
|---|---|---|---|---|---|
| F02 | pass | pass | $0.15 | $0.02 | none |
| F01 | fail | pass | $0.20 | $0.01 | none |
| F03 | fail | pass | $0.17 | $0.03 | none |
| F04 | fail | fail | $0.27 | $0.03 | F6: inherited base test tests/agent-backend.test.ts (bd90b27), amendment 18 |
| F05 | fail | fail | $0.20 | $0.04 | F6: inherited base test tests/agent-backend.test.ts (bd90b27), amendment 18 |
| F06 | fail | fail | $0.17 | $0.03 | F6: inherited base test tests/agent-backend.test.ts (bd90b27), amendment 18 |
| F08 | fail | fail | $0.12 | $0.03 | F6: inherited base test tests/agent-backend.test.ts (bd90b27), amendment 18 |
| F09 | fail | fail | $0.27 | $0.04 | none |
| F10 | fail | fail | $0.37 | $0.12 | none |
| F15 | pass | pass | $0.17 | $0.03 | none |
| F12 | pass | pass | $0.13 | $0.03 | none |
| F16 | fail | pass | $0.19 | $0.04 | none |
| F17 | fail | pass | $0.14 | $0.04 | none |
| F18 | fail | pass | $0.21 | $0.03 | none |
| F19 | pass | pass | $0.09 | $0.02 | none |
| F20 | pass | pass | $0.11 | $0.02 | none |
| F21 | pass | pass | $0.23 | $0.02 | none |
| F22 | pass | pass | $0.18 | $0.04 | none |
| F23 | pass | pass | $0.11 | $0.02 | none |
| F24 | pass | pass | $0.15 | $0.02 | none |
| F25 | pass | pass | $0.13 | $0.02 | none |
| F26 | pass | pass | $0.12 | $0.01 | none |
| F27 | pass | pass | $0.18 | $0.02 | none |
| F28 | pass | pass | $0.15 | $0.03 | none |
| F29 | pass | pass | $0.11 | $0.01 | none |
| F30 | pass | pass | $0.12 | $0.02 | none |
| F14 | fail | fail | $0.22 | $0.04 | none |
| F07 | pass | pass | $0.15 | $0.02 | none |
| F31 | fail | pass | $0.10 | $0.01 | none |
| F32 | pass | pass | $0.09 | $0.01 | none |
| F33 | pass | pass | $0.16 | $0.01 | none |
| F34 | pass | fail | $0.18 | $0.04 | none |
| F36 | fail | pass | $0.13 | $0.01 | none |
| F35 | pass | pass | $0.10 | $0.02 | none |
| F37 | fail | pass | $0.16 | $0.02 | none |
| F38 | fail | pass | $0.12 | $0.01 | none |
| F40 | pass | pass | $0.11 | $0.02 | none |
| F39 | fail | pass | $0.11 | $0.02 | none |
| F13 | fail | pass | $0.23 | $0.02 | none |