89e05459c0
A section used to be one scene, and two thirds of the library is composable —
sparse by design, elements ON something. Cast as backgrounds anyway, they left
9 of 40 sampled frames under 20% painted, the darkest at 0.3%: a minute and a
half of a few bright things on black, invisible to every gate because every
gate on the stack was a limit rather than a floor.
Every section now stands on a GROUND: a canvas that fills the frame, cast per
section kind so a shot cut changes the shot and not the world. When the shot
fills the frame itself it IS the ground — two canvases stacked is two pictures
fighting. Above that, a coverage BUDGET: director appetite times the section's
energy times where the story is, capped at two frames' worth of material.
The measured facts move into the repo. scenes/metadata.json is generated from
the gallery — coverage as a shot, coverage as a bed, variety, the structural
profile — tracked in git, stamped with a fingerprint of the scenes and the
metric definitions, and refreshed from gallery.html. `surface` is derived from
it rather than declared; nine scenes claimed `canvas` while painting under a
third of the frame, and declaring it is now a lint error. The generator weights
every layering choice by measured structural distance, because family labels
and the render disagree: two `geometric` scenes can be 0.31 apart and a `flow`
and an `organic` scene 0.04.
The gallery's 0.1 red line is gone. It was right when a section was one scene
and wrong now — nineteen scenes were failing a bar for being consistent, which
is a virtue in an ingredient.
Chasing the numbers turned up four real faults:
* A scene that reads prev() cannot be a ground. It returns the whole
composited frame including the layers above it, so a datamosh under a shot
is eating it: the render stopped reproducing from a seek and two WebGL
contexts diverged by 91/255 against a tolerance of 4.
* Screen was the wrong operator for a shot over a bed. It lightens, so a
median quarter of every frame clipped to paper and whole sections rendered
100% white. Replaced by a lumakey — the shot's brightness is its alpha.
* Feedback was an accumulator: a still image settled at 2.3x its own
brightness. Fine over black, fatal over a filled ground. Normalised at 0.6,
plus a highlight shoulder so the top rolls off instead of clipping.
* useTrack never prewarmed, so a fresh Show's first frame differed from every
later render of it — the export-breaking hazard Compositor.prime documents.
Blazing is a decision now, not a side effect: directors declare an appetite for
it, a section must be loud and late in the story to earn one, and quiet kinds
never do. The ceiling gate matches that — a hard cap per section, and no more
than a fifth of them hot at all.
Rendered across twelve videos, middle of every section:
painted 51% mean, darkest 0.3% -> 87% mean, darkest 43%
clipped white 24% median, worst 100% -> 1% mean, worst 30%
separation 0.10 -> 0.44
Seven scenes can ground a section — five geometric, two organic — so every
quiet section of every video stands on one of two beds. That is the library's
largest hole and it is scene work: there is no minimal or flow canvas that
fills half the frame without reading prev().
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
321 lines
15 KiB
Markdown
321 lines
15 KiB
Markdown
# How to build a visualizer that varies
|
||
|
||
`HOWTO-visualizers.md` covers making a scene that *works*. This one covers making
|
||
a scene that looks different from one song to the next — a separate skill, and
|
||
the one the library is currently worst at.
|
||
|
||
Everything here is a measurement rather than an opinion, and the numbers are
|
||
quoted so a future change can contradict them. Several already contradict things
|
||
believed earlier in the same week. Living document: add what you learn, and
|
||
delete what stops being true.
|
||
|
||
---
|
||
|
||
## The one-line version
|
||
|
||
> A scene varies when the SONG can change what is on screen. It does not vary
|
||
> because its parameters moved.
|
||
|
||
Parameters mostly change the same picture. Content changes the picture.
|
||
|
||
---
|
||
|
||
## How to know if you succeeded
|
||
|
||
```
|
||
gallery.html your scene, six times, on six songs — with a score
|
||
checks.html?scene=X the per-scene gate, including `consumes:` lines
|
||
```
|
||
|
||
The gallery score is the mean structural distance between a scene's own six
|
||
frames, on the same descriptor the variety harness uses. High means the song
|
||
changes this scene a lot; low means it looks like itself wherever it is cast.
|
||
|
||
**There is no bar.** There used to be one — 0.1, drawn across the gallery as a
|
||
red line — and it was removed when composition arrived. A section is no longer
|
||
one scene: it is a ground, a shot standing on it, and sometimes a pass over
|
||
that, so what a viewer sees is a COMBINATION. A scene that is reliably itself is
|
||
a perfectly good ingredient in one, and the old line failed it for being
|
||
consistent.
|
||
|
||
What replaced it is a level up. The interesting quantity is how unalike the
|
||
scenes in one stack are, and the generator now weights its choices by exactly
|
||
that — see `structuralDistance` in `scenes/surface.js`. Your scene does not have
|
||
to be varied on its own. It has to be unlike the things it will be stacked with,
|
||
and that is measured for you.
|
||
|
||
Some reference points, measured across six songs:
|
||
|
||
```
|
||
0.324 Droste Feedback varies a lot
|
||
0.082 Plasma Bloom
|
||
0.062 Voronoi Shatter after a focus
|
||
0.062 Apollonian Gasket
|
||
0.047 Moiré Grid after a subject; 0.031 before
|
||
```
|
||
|
||
### The measured metadata
|
||
|
||
`src/scenes/metadata.json` is generated, tracked in git, and holds every
|
||
measured fact about every scene: coverage, variety, per-block scores and the
|
||
structural profile the distance above is computed from. Nothing in it is
|
||
hand-written, and two decisions read it:
|
||
|
||
* **`surface`** — canvas at 50% coverage or more, composable below. It used to
|
||
be declared per scene and it drifted; nine scenes claimed `canvas` while
|
||
painting under a third of the frame. Declaring `surface:` in a scene file is
|
||
now an error.
|
||
* **which scenes may be a GROUND** — a canvas that does not call `prev()`, fills the frame
|
||
when sampled as a bed, and does not vanish under any of the six identities. Coverage is
|
||
measured twice, once as a shot and once as a bed, because a scene paints wildly different
|
||
amounts at different parameters: Metaballs is 69% as a drop's shot and 6% as an intro's
|
||
ground.
|
||
|
||
Refresh it from **gallery.html → refresh metadata** after changing a shader or a
|
||
metric. The file carries a fingerprint of the scenes, the shader contract, the
|
||
identities and palettes, and the descriptor definitions; when that stops
|
||
matching, phase 12 fails and tells you to re-measure. The write goes through a
|
||
dev-only endpoint in `vite.config.js`, so it only works under `npm run dev`.
|
||
|
||
---
|
||
|
||
## What the descriptor can and cannot see
|
||
|
||
Design against this, because half of "why is my score low" is here.
|
||
|
||
**It is blind to:**
|
||
|
||
- **Brightness and contrast.** Frames are standardised to zero mean and unit
|
||
variance before anything is measured. A scene that only gets brighter has not
|
||
changed.
|
||
- **Colour.** Measured, reported, and excluded from the score. Six palettes
|
||
cannot disguise one image, which was the entire point.
|
||
- **Rotation**, on the scale, orientation and texture blocks.
|
||
- **Quality.** It measures change between songs, nothing else. Apollonian Gasket
|
||
scores 0.039 and looks great. Both facts are true and neither implies the
|
||
other.
|
||
|
||
**It sees:**
|
||
|
||
| block | what it measures | how to move it |
|
||
|---|---|---|
|
||
| `scale` | feature size — fine grain vs large forms | let the song set element size (`stageScale()`) |
|
||
| `orient` | grid vs radial vs stripes, rotation-blind | break or vary a regular lattice |
|
||
| `layout` | where in the frame structure sits | move the subject, change what is empty |
|
||
| `texture` | element count, sparsity, mirror and radial symmetry | vary how many things there are |
|
||
| `region` | how differently the parts of the frame behave from each other | give the frame a subject, so one region is unlike its neighbours |
|
||
| `motion` | what moves and where, not how much | vary the KIND of movement, not the rate |
|
||
|
||
**`region` is the block that rewards having something to look at.** Layout says
|
||
where the energy is, normalised, so a uniform field and a field with a subject in
|
||
it can normalise to nearly the same answer. Region characterises each part of the
|
||
frame in its own right — detail, direction, element count — and reports how far
|
||
each deviates from the frame's average. A pattern spread evenly deviates by
|
||
nothing everywhere, which is exactly why there is nothing to watch.
|
||
|
||
Measured when it was added: Apollonian Gasket 0.039 to 0.062 and Plasma Bloom to
|
||
0.082, both on region alone, because both have a subject the old blocks were not
|
||
crediting. It is the strongest single block in the descriptor.
|
||
|
||
**A known blind spot.** `layout` is nearly useless for full-frame fields. Voronoi
|
||
Shatter measures 0.004 there no matter what changes, because edge-to-edge cells
|
||
occupy the same frame however they fall. That is honest — there genuinely is no
|
||
arrangement — but it means a whole family is judged on four blocks instead of
|
||
five. If your scene fills the frame corner to corner, expect to earn your score
|
||
on `orient` and `texture`.
|
||
|
||
---
|
||
|
||
## What has actually worked
|
||
|
||
Ordered by measured effect.
|
||
|
||
### 1. Draw the song's content instead of your own
|
||
|
||
The largest single lever. Take `consumes: ['cast', 'ink', 'staging']` and draw
|
||
`castMain`/`castChorus` where you would have drawn your own primitive. Recipe in
|
||
`MIGRATION.md`.
|
||
|
||
Measured: identity is worth 117% of what the container is worth
|
||
(`checks.html?decompose=1`) — swapping the song's cast under a fixed scene moves
|
||
the picture more than swapping the scene under a fixed cast. A migrated library
|
||
scene also carries the identity better than a stage written from scratch to
|
||
carry it, which was a surprise and is why the whole library was migrated rather
|
||
than replaced.
|
||
|
||
### 1b. Give the subject a third dimension — payoff still unmeasured
|
||
|
||
`castSolid` / `castChorusSolid` / `castMarch`, and `consumes: ['form']`. The
|
||
song's protagonist as an assembly of solids instead of an outline, so its
|
||
silhouette CHANGES as the shot moves rather than merely rotating. Recipe in
|
||
`MIGRATION.md`, including the two guards that keep many instances affordable.
|
||
|
||
Three scenes carry it — Effigy, Floating Geometry, Swarm — which is 34.8% of
|
||
videos containing at least one, against 11.9% when only Effigy had it. That is
|
||
reach, not payoff.
|
||
|
||
The payoff is still unmeasured: the variety report has not been re-run against
|
||
a library with these in it, so nothing here says the videos are more varied.
|
||
What is measured is narrower — across four seeds, the lit area of Effigy's
|
||
subject varies 14-113% over one turn against Soloist's 3-51% for the same
|
||
rotation, which says the outline genuinely changes rather than merely spinning.
|
||
Whether that reaches the variety blocks is the open question, and it is the next
|
||
thing to run. Do not migrate a field scene onto it hoping for a win.
|
||
|
||
### 2. Let the song decide element SIZE
|
||
|
||
`stageScale()`. Not the size of your features relative to each other — the size
|
||
of the whole vocabulary. A song of six huge forms and a song of four hundred
|
||
tiny ones are different videos before anything else is decided.
|
||
|
||
Measured: adding `elementScale` to the identity moved stages from
|
||
+0.0111 ±0.0043 to +0.0143 ±0.0008, and the `scale` block from 0.033 to 0.041.
|
||
|
||
### 3. Displace a regular lattice
|
||
|
||
If your scene is a grid, let the song push things off it. Bounded — past about
|
||
half a cell the structure the grid was providing stops reading.
|
||
|
||
Measured: Isometric Blocks 0.034 → 0.049, with `orient` nearly doubling from
|
||
0.093 to 0.164 as the rigid lattice softened.
|
||
|
||
### 4. Give a full-frame field somewhere to be about
|
||
|
||
`focusWarp(p, amount)` and `focusField(p)`. The song picks one to three focal
|
||
points; tile in the warped coordinate so cells crowd toward them or pull away.
|
||
|
||
Measured: Voronoi Shatter 0.0463 → 0.0588. **Note what moved:** `orient` 0.140 →
|
||
0.194, while `layout` went 0.004 → 0.007, i.e. nowhere. Warping where cells sit
|
||
changes what they look like without changing where the frame's energy is. The
|
||
gain is real; the stated reason for it was wrong.
|
||
|
||
---
|
||
|
||
## What has measurably NOT worked
|
||
|
||
Recorded because the failures were more informative than the wins, and because
|
||
each of these looked obviously right beforehand.
|
||
|
||
### Effects as parameters
|
||
|
||
Forty-nine scenes had their own `glow`; every one had its own grain. An effect
|
||
knob sampled per song makes a scene look varied across parameter draws while its
|
||
structure never moves — it inflates the score without changing the picture, and
|
||
it fights the grade, which already does bloom, grain, chroma and vignette with
|
||
an envelope the scene cannot see.
|
||
|
||
**Do not add glow, bloom, grain, haze, chroma or trails to a scene.** The post
|
||
chain owns them. `npm run lint:scenes` does not catch this yet; reviewers should.
|
||
|
||
### Concentrating disturbance instead of adding it
|
||
|
||
Isometric Blocks, twice. Spending the scatter budget near the focal points and
|
||
calming the rest measured 0.049 → 0.0374 — most of the field went back onto the
|
||
rigid lattice and took the orientation variety with it. Making the focus
|
||
additive instead recovered nothing: 0.0365.
|
||
|
||
Two plausible diagnoses, both wrong. The lesson is narrow and worth keeping:
|
||
**a focal point has to be something the field gains, not something the rest of it
|
||
pays for** — and even that framing did not rescue it, so the real cause is still
|
||
unknown.
|
||
|
||
### Making the roster smaller
|
||
|
||
A whole afternoon went into the theory that a track drawing on fewer scenes
|
||
would look more like itself. Swept directly across pool sizes 4, 8, 16 and 32
|
||
over twelve songs with three draws each: differences of 0.003 to 0.007 against a
|
||
run-to-run noise of ±0.003 to ±0.005. **Nothing.** The theory came from an
|
||
experiment whose arms differed in two ways at once.
|
||
|
||
### Reading a form as a metric rather than drawing it
|
||
|
||
Spectrum Sculpture used `sigShape` as a radial distance rather than as a subject.
|
||
Migrating it to `castMain` rendered pure black, because the cast carries notches
|
||
and a hollow and an annulus used as a radius turns a sculpture inside out. If
|
||
your scene consults a form's geometry rather than drawing it, it wants
|
||
`sigShape`, and the cast is not for you.
|
||
|
||
---
|
||
|
||
## Scene shapes, and how hard each is to vary
|
||
|
||
- **Element-placing** (`metaballs`, `floating-geometry`, `firefly-drift`) —
|
||
easiest. Take the cast, place on `stageNode`, keep your motion. Nearly all the
|
||
wins above are this shape.
|
||
- **Single figure** (`slow-orb`, `eclipse-field`) — take the cast and
|
||
`stageScale()`. Composition is yours to vary; most such scenes centre their
|
||
subject and never move it, which is free `layout` left on the table.
|
||
- **Full-frame field** (`voronoi-shatter`, `plasma-bloom`, `curl-flow`) — hard.
|
||
`layout` is unavailable to you. Earn it on `orient` and `texture`, and consider
|
||
a focus.
|
||
- **Fixed geometry** (`truchet-fold`, `quasicrystal`, `apollonian-gasket`) —
|
||
hardest, and currently unsolved. The image *is* the maths, so the cast cannot
|
||
be pasted on top of it. The open idea is to make the cast the repeating UNIT
|
||
the tiling is built from, which is a real rewrite per scene rather than a
|
||
recipe step. Nobody has tried it yet.
|
||
- **Treatment** (`analog-wow`, `scan-tear`, `halftone-misprint`) — these are
|
||
effects wearing a scene's clothes. They score 0.031–0.040 and their whole
|
||
migration was one posterisation. They probably belong in the identity's
|
||
effects register rather than competing for screen time as subjects.
|
||
|
||
---
|
||
|
||
## Trusting the numbers
|
||
|
||
This harness has two classes of measurement and only one is safe to steer by.
|
||
|
||
**Direct render comparisons** — the gallery score, `checks.html?decompose=1`, the
|
||
per-scene gate — compare rendered frames to each other. Noise floor of exactly
|
||
zero. Trust these.
|
||
|
||
**Aggregate ratios** — chiefly `separation`, which divides one small difference
|
||
by another — swing wildly. Three runs of the same post-migration measurement gave
|
||
0.31, 0.56 and 0.07. **Four conclusions were drawn and withdrawn during this epic
|
||
for exactly that reason.** Anything under about 0.01 of spread needs
|
||
`&repeats=3` before it is believed, and a difference that changes sign with the
|
||
sample size is not a difference.
|
||
|
||
Two more traps, both of which have bitten:
|
||
|
||
- **A score of exactly 0.000 is a dead shader, not a boring scene.** The gallery
|
||
now says so outright — the row goes purple and carries the luminance and
|
||
variance — but the underlying trap is general: a broken render produces the
|
||
most boring possible numbers rather than an error.
|
||
- **A permanently-zero block looks like a property of your scene.** The gallery
|
||
built four of the descriptor's five blocks for weeks; `motion` came back 0.000
|
||
for every scene and depressed every score by a fifth. If a block is identical
|
||
across many unrelated scenes, suspect the instrument.
|
||
- **Arms that differ in more than one way cannot be read as if they differed in
|
||
one.** This produced the roster-size result above, and it was believed for a
|
||
day.
|
||
|
||
---
|
||
|
||
## Checklist for a new or reworked scene
|
||
|
||
1. `consumes` declared, and the `consumes:` lines pass in
|
||
`checks.html?scene=<name>`. A declared artifact that is ignored is worse than
|
||
one not declared, because the casting code believes it.
|
||
2. No glow, bloom, grain, haze, chroma or trail parameters.
|
||
3. Element size goes through `stageScale()`.
|
||
4. Placement goes through `stageNode()` unless the composition IS the scene.
|
||
5. Gallery score above 0.1, checked before and after.
|
||
6. The rest of the per-scene battery still passes — especially `distinct`. The
|
||
more scenes share a cast, the easier it is to become one of your neighbours.
|
||
|
||
---
|
||
|
||
## Open questions
|
||
|
||
- **Coupling is zero and has never moved.** Whether a song *looks* different in
|
||
proportion to how it *sounds* different measures -0.02 ±0.03. Every change in
|
||
this epic left it there. Nobody has attacked it directly.
|
||
- **Fixed-geometry scenes** have no known route to variety.
|
||
- **The `layout` blindness** for full-frame fields would need a new descriptor
|
||
block measuring cell statistics rather than where structure sits.
|
||
- **The bar is a target, not a description.** Most of the library does not clear
|
||
0.1 — that is the point of it. It has moved from 0.04 to 0.05 to 0.1
|
||
as the descriptor gained a block and lost a bug, both of which raised every
|
||
score. Expect to move it again whenever the descriptor changes, and re-read it
|
||
off a fresh gallery rather than carrying the old number forward.
|