Sweep pool size directly, and withdraw the claim that it mattered
The Epic 3 arms appeared to show that a small casting roster was the largest available win. Swept directly across 4, 8, 16 and 32 over twelve songs with three pool draws each, the differences are 0.003 to 0.007 against a run-to-run noise of +/-0.003 to +/-0.005. Pool size does nothing measurable. The arms varied two things at once — smaller pool AND the same pool for every song — and only the stages-versus-legacy comparison inside them held everything else constant. That one still stands at +14%. POOL_SIZE stays at 8, now on grounds the metric cannot see: about nine distinct scenes in a video rather than seventeen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -312,6 +312,33 @@ built from legacy scenes? Measure that on five stages before touching the other
|
||||
|
||||
---
|
||||
|
||||
## 9a. What the first slice actually measured
|
||||
|
||||
Four stages shipped, and the A/B ran: same songs, same instrument, same pool
|
||||
size, the only difference being whether the scenes draw the song's cast or their
|
||||
own content.
|
||||
|
||||
arm floor observed spread
|
||||
stages, four of them 0.0801 0.1133 +0.0332
|
||||
legacy scenes, four of them 0.0830 0.1122 +0.0292
|
||||
the unrestricted generator 0.1058 0.1146 +0.0088
|
||||
|
||||
The stages-versus-legacy comparison is the internally valid one — everything
|
||||
except the content sharing is held constant — and it says the inversion helps by
|
||||
about 14%. Real, in the predicted direction, and much smaller than hoped.
|
||||
|
||||
The third row looked at first like the headline: that a small roster, not the
|
||||
content sharing, was carrying the improvement. It is not. That arm varies two
|
||||
things at once — the pool is smaller AND it is the same pool for every song —
|
||||
and a direct sweep of pool size alone (§ `checks.html?sweep=1`, twelve songs,
|
||||
three draws each) finds differences of 0.003 to 0.007 against a run-to-run noise
|
||||
of ±0.003 to ±0.005. Pool size does nothing measurable between 4 and 32.
|
||||
|
||||
Two lessons worth keeping. Arms that differ in more than one way cannot be read
|
||||
as if they differed in one. And this metric's noise floor at seven songs is
|
||||
large enough to invent findings — anything under about 0.01 of spread needs
|
||||
repeats before it is believed.
|
||||
|
||||
## 10. The smallest experiment worth running first
|
||||
|
||||
One artifact, three stages, one measurement. Do not build the whole identity layer on a
|
||||
|
||||
Reference in New Issue
Block a user