Sweep pool size directly, and withdraw the claim that it mattered

The Epic 3 arms appeared to show that a small casting roster was the largest
available win. Swept directly across 4, 8, 16 and 32 over twelve songs with
three pool draws each, the differences are 0.003 to 0.007 against a run-to-run
noise of +/-0.003 to +/-0.005. Pool size does nothing measurable.

The arms varied two things at once — smaller pool AND the same pool for every
song — and only the stages-versus-legacy comparison inside them held everything
else constant. That one still stands at +14%.

POOL_SIZE stays at 8, now on grounds the metric cannot see: about nine distinct
scenes in a video rather than seventeen.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Dejvino
2026-08-17 21:27:58 +02:00
co-authored by Claude Opus 5
parent ec38def1a9
commit 577ec107f6
5 changed files with 153 additions and 8 deletions
+27
View File
@@ -312,6 +312,33 @@ built from legacy scenes? Measure that on five stages before touching the other
---
## 9a. What the first slice actually measured
Four stages shipped, and the A/B ran: same songs, same instrument, same pool
size, the only difference being whether the scenes draw the song's cast or their
own content.
arm floor observed spread
stages, four of them 0.0801 0.1133 +0.0332
legacy scenes, four of them 0.0830 0.1122 +0.0292
the unrestricted generator 0.1058 0.1146 +0.0088
The stages-versus-legacy comparison is the internally valid one — everything
except the content sharing is held constant — and it says the inversion helps by
about 14%. Real, in the predicted direction, and much smaller than hoped.
The third row looked at first like the headline: that a small roster, not the
content sharing, was carrying the improvement. It is not. That arm varies two
things at once — the pool is smaller AND it is the same pool for every song —
and a direct sweep of pool size alone (§ `checks.html?sweep=1`, twelve songs,
three draws each) finds differences of 0.003 to 0.007 against a run-to-run noise
of ±0.003 to ±0.005. Pool size does nothing measurable between 4 and 32.
Two lessons worth keeping. Arms that differ in more than one way cannot be read
as if they differed in one. And this metric's noise floor at seven songs is
large enough to invent findings — anything under about 0.01 of spread needs
repeats before it is believed.
## 10. The smallest experiment worth running first
One artifact, three stages, one measurement. Do not build the whole identity layer on a