music-video-gen/flow-state/PLAN.md
Dejvino 2806ef1386 Phase 10: variety, twelve scenes, and tooling to write the next one
Watching several finished tracks side by side turned up the problem neither
Phase 8 (too few cuts) nor Phase 9 (no through-line) addressed: the same
scene cast in two different videos looked like the same footage twice.
Section bias is nearly identical between two tracks' drops, so both sampled
their parameters around the same centre, and the library's own averageness
did the rest.

Three answers, none of them a new scene:

  Temperament — a per-track hand on every parameter dial: intensity, pace,
  detail, and an extremity that decides how far toward the ends of a range
  the track is willing to sample. Bias comes from the section and is shared
  between tracks; temperament comes from the track and is not.

  Overlays — sometimes a second full scene composited over the shot, from a
  different family, in a blend that preserves what is underneath and never
  above 0.6 opacity. Not always: a stack that always doubled up would read
  as permanently cluttered rather than as occasionally layered.

  A wider palette — hue now derives from SPECTRAL TILT, the log ratio of
  treble to body. The centroid is a number most masters sit in the middle
  of, and the plain body/(body+treble) fraction is worse: low frequencies
  carry most of the energy in all music, so it read 0.98-1.00 for
  everything and four different battery tracks came out within 0.02 of
  each other. The ratio is multiplicative, so its logarithm is what
  spreads — the same four measure -9.3, -5.0, -4.1, -3.8. Also both ways
  round the wheel (violet, magenta and pink were unreachable by
  construction), four new schemes, and seeded chroma profile and lightness
  curve. Closest battery pair went from 0.005 to 0.113.

Twelve scenes take the library to 36, six per family: Aurora Veil, Vortex
Drift, Tide Rings, Ink Bleed, Dust Chamber, Salt Flat, Cargo Belt, Gate
Corridor, Circuit Bloom, Truchet Fold, Signal Decay, Storm Rift. Weighted
toward the 'space' and 'shape' traits, which were thinnest and so the
signatures most likely to run a track out of cast — the Phase 9 casting
rule means the pool a track draws from is smaller than the library.

Also fixes a real one in shots.js: heavy LRU weighting was not enough to
make a section reach its whole roster, and a five-shot section still came
out 0,2,0,2,0 about a fifth of the time. An unseen companion now wins
outright; which one is still free, so only the coverage is guaranteed.

Block Mosh declared the camera trait, assigned sigCamera(p) to a p it then
never read, and passed the lint's evidence grep. The Phase 9 render gate
measured its response to the camera at exactly zero.

--- tooling ---

Adding a scene was mostly boilerplate and round-trips, which is expensive
in both senses. The irreducible cost is the shader body; everything around
it is now mechanical:

  npm run new:scene -- "Name" --family=... --traits=...

writes the module, registers it, and leaves a skeleton that already passes
every gate, with name-derived constants so two skeletons are not twins.

The lint grew the rules that previously needed a GPU to catch: the dead
camera above, prev() with no base image, and large loops with no early
break (with a `// lint: fixed-cost` opt-out for a genuinely fixed-cost
sampling loop). checks.html?scene=Name runs the per-scene acceptance
battery for one scene — ten lines and a verdict instead of rendering the
whole library to find out whether one shader is alive. The same procedure
is a repo skill under .claude/skills/build-visualizer/.

--- checks changed, with the measurements ---

P5 determinism compared two WebGL CONTEXTS, which is not what it is for.
Measured: one context is bit-exact over 40 frames with feedback at 0.6;
two contexts disagree by up to 2/255 whether feedback is on or off. It now
asserts generation is byte-identical (hard) and rendering within 2/255,
since feedback compounds single-level variance.

P10's cross-track comparison measures distance RELATIVE to how much image
there is. Most scenes are mostly dark, so two genuinely different renders
— 25 bars against 53 — scored under 0.02 absolute purely because the black
background agrees with itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 23:59:20 +02:00

34 KiB

flow-state — plan

An ambient/EDM music video generator. Drop in a track, get a full-length, non-story, music-reactive video that evolves through the song. No sourced footage: every frame is generated, and everything the look depends on is derived from the audio itself.

Name is a placeholder, easy to rename before Phase 0 lands.

Decisions taken

Question Decision
Render model Deterministic core. Realtime preview and offline frame-exact export, from the same code path.
Preview Required. Full transport, section jumping, live param editing, and segment test-renders — see §8.
Art direction Automatic from the audio, every derived parameter overridable and savable per track.
Scene architecture Hybrid compositor. Layers are usually fullscreen fragment shaders; three.js particle/geometry layers are also valid layers.
Cover art Not available. Palette is audio-derived. A PaletteSource seam is left so artwork can be added later without touching scenes.
Relationship to party-stage Fork and detach. Copy what's useful, then zero cross-imports — see §12.

The two problems this has to solve

They are separate and both are load-bearing:

  1. Sameness across tracks. Track #12 must not look like track #11. Solved by deriving the look from the track's own measured character plus a content-derived seed.
  2. Monotony within a track. A single shader gets dull around minute three. Solved by segmenting the song and driving scene changes, palette shifts and parameter motion off that structure.

Both have explicit validation gates in §11 — they are easy to believe you've solved and hard to actually solve.


1. The determinism spine

This is the single architectural constraint everything else obeys. It is what makes "realtime preview" and "4K60 offline export" the same program rather than two programs that drift apart.

Mechanism: the audio is fully analyzed before the first frame renders, into a frame-indexed table.

track file
   ↓ decode()            OfflineAudioContext → AudioBuffer (a 6-min track decodes in ~1-2s)
   ↓ analyze()           STFT at hop = 1/60 s, plus global passes
   ↓
FeatureTrack             a typed-array table, one row per output frame

Nothing in the engine ever reads a live AnalyserNode. Realtime preview maps audio.currentTime → frameIndex and reads row n. The offline exporter counts frameIndex directly and reads row n. Identical input to the visuals in both modes, so what you preview is exactly what you export.

This also unlocks the thing a causal analyser fundamentally cannot do:

Lookahead. Because the whole song is analyzed up front, a build can anticipate its drop — ramp density and warp toward the drop's target values over the eight bars before it, and hit the transition already at full tension. A causal analyser can never do this; it only knows the drop landed after it landed. This is the biggest single visual win in the whole plan.

Determinism rules (violating any of these breaks export)

  • No wall-clock anywhere in scene or layer code. No performance.now(), no Date, no clock.getElapsedTime(). A Timeline object injects { frame, time, dt, progress }.
  • No Math.random() in anything that animates. Seeded PRNG only (mulberry32, with the seed held per-instance rather than on a global).
  • dt is constant1/fps — even in realtime preview. If the browser drops a frame, preview shows a hitch; it does not change the animation. Frame 5400 is the same image every run.
  • Deterministic buffer initialization. Feedback and ping-pong targets must be explicitly cleared at start; never inherit whatever was in GPU memory.
  • Resolution independence. Scene math lives in normalized aspect-corrected coordinates. Anything genuinely pixel-sized takes a u_pixelScale = resolution / referenceResolution uniform. This is what lets a 720p preview match a 4K export.

Stated limit on the guarantee

Determinism is per-machine: same browser, same GPU, same driver → bit-identical frames. Across different GPUs, floating-point and fwidth derivative differences make bit-exactness unrealistic, and chasing it would be wasted effort. The cross-machine guarantee is perceptual: frames must match within a small diff threshold. The validation checks in §11 are written to that distinction, so the acceptance criteria are actually achievable.

Two refinements, both found by building it:

Programs must be primed before the first frame. Shader programs link asynchronously (KHR_parallel_shader_compile), and a draw issued against an unlinked program produces wrong output. Measured on the heaviest scene, the first ten frames rendered differently from every later render of the same frames. Preview hides this completely — the frames go past and the next pass is right — but an export renders each frame exactly once, so those frames would ship broken. Engine.prime() compiles every program and discards a warm frame, and the exporter calls it before encoding anything. Rendering a throwaway frame and reading it back is not sufficient; WebGLRenderer.compile() is.

Even same-machine, the heaviest shaders vary by one LSB. With priming in place most scenes reproduce byte-for-byte, but a few come back with scattered pixels differing by 1/255 — floating-point variance under differing GPU load. That is below any perceptual threshold and is not something the hardware offers to fix, so the per-scene criterion is a max channel delta of ≤ 1 rather than an identical hash. A real determinism bug scores in the tens or hundreds on that metric, so the check keeps its teeth.


2. Audio analysis

src/audio/. Pure functions over an AudioBuffer — no DOM, no three.js — so it's unit testable and can move to a worker without rework.

Per frame (hop 1/60 s, window 2048, Hann):

Feature Used for
rms, loudness overall intensity, quiet-passage detection
band energies (sub / low / mid / high / air) per-element reactivity
flux (spectral flux, half-wave rectified) onset strength, the beat signal
centroid perceived brightness → color temperature
flatness noisy vs tonal → distinguishes pads from percussion
stereoWidth width of the visual field

Global passes:

  • Tempo — autocorrelation over the onset envelope, octave-normalized to 90-180 BPM. Whole-song, so it's exact rather than converging. Yields a phase-locked beat grid and downbeats, hence exact beatPhase / barPhase / phrasePhase per frame.
  • Segmentation — novelty curve from a self-similarity matrix over the band-energy vectors, peak-picked into section boundaries.
  • Section classification — each section labelled by energy percentile, flux density and centroid: intro | build | drop | sustain | breakdown | outro. Deliberately simple heuristics over features we already have; this is the fuzziest part of the pipeline and is not worth an ML detour.

Lookahead fields, per frame: timeToNextSection, nextSectionKind, nextSectionEnergy, buildSlope. These are what the arc driver leans on.


3. Scene modules and the parameter schema

This is the decision that makes a large library affordable. Adding a scene must cost a shader and a schema block — no plumbing, no UI code, no wiring.

export const NebulaDrift = {
  name: 'Nebula Drift',
  family: 'organic',
  kind: 'fragment',            // or 'layer3d'
  shader: fragmentSource,

  params: {
    density:  { type:'float',   range:[0,1],  default:0.5, uniform:'u_density' },
    warp:     { type:'float',   range:[0,3],  default:1.0, uniform:'u_warp' },
    symmetry: { type:'int',     range:[1,8],  default:1,   uniform:'u_symmetry' },
    grain:    { type:'float',   range:[0,1],  default:0.2, uniform:'u_grain' },
    palette:  { type:'palette', count:5,                   uniform:'u_colors' },
  },

  // which audio features modulate which params, and how hard
  reactive: {
    density: { feature:'bandLow',  amount:0.3 },
    warp:    { feature:'beat',     amount:0.5, response:'spike' },
    grain:   { feature:'flatness', amount:0.2 },
  },
};

One schema, five payoffs:

  1. Uniforms bind automatically — no per-scene wiring.
  2. The config UI generates its own sliders, so override-ability is free for every new scene.
  3. LookGenerator samples the declared ranges with the track's seed → per-track variation with no hand-tuning.
  4. ArcDriver can animate any declared param across a section boundary generically.
  5. Presets are just serialized param sets, savable per track.

It also makes scenes machine-checkable, which is what keeps a thirty-scene library from rotting — see the schema lint in §10.

Library targets

Interest across songs comes from breadth of family, not raw count. Six families, growing to roughly 4-6 scenes each:

Family Character Fits
flow curl noise, fluid, drifting particles ambient, sustain
geometric tunnels, kaleidoscope, grids, synthwave drops
organic nebula, plasma, reaction-diffusion, metaballs sustain, breakdown
structural raymarched terrain, cities, architecture builds
minimal lines, waveform, spectrum sculpture, negative space intros, quiet passages
glitch feedback, datamosh, scanlines, chromatic tearing drops, transitions

The arc driver picks a family from section kind, then a scene within it from the seed — so a quiet passage never lands on a strobing glitch scene, and two tracks with similar structure still choose differently.

Seeded from the five that exist

party-stage's five visualizer shaders are the library's starting point. Porting each means: strip the LED-grid mask, replace hardcoded colors with palette lookups, pull magic numbers out into declared params, swap live uniforms for FeatureTrack reads. Note their u_resolution is not pixels — it's the LED grid cell count. Rename it u_gridSize on port so it doesn't collide with the real resolution uniform.


4. Compositor

src/engine/Compositor.js. A layer stack rendered to ping-pong render targets.

  • Layers: fragment-shader (a fullscreen quad) or 3D (a three.js sub-scene — particles, geometry, camera path). Both present the same render(target, timeline, features) face, so the stack doesn't care which it holds.
  • Blend modes per layer: normal, add, screen, multiply, overlay.
  • Feedback buffer — previous frame available to any layer as a texture, with configurable decay and warp. Cheap, and disproportionately responsible for things looking "alive".
  • Post chain: bloom, chromatic aberration, grain, vignette, final grade.

Typical stack: background scene → accent scene (screen blend, low opacity) → particles → feedback → post.


5. Look generation and the arc driver

LookGenerator runs once per track, after analysis:

seed = hash(decoded PCM)          → same track always renders identically
palette = f(centroid, flatness, energy distribution, seed)
sceneAssignments = per section: family from kind, scene from seed
paramSets = sampled from declared ranges, biased by track character

PaletteSource is an interface here, with an AudioPalette implementation. Adding a CoverArtPalette later is a new file and one line of config — no scene changes.

ArcDriver runs per frame:

  • Cross-fades scenes at section boundaries (compositor holds both briefly, opacity ramp).
  • Ramps params toward the next section's targets during builds, using the lookahead fields.
  • Applies slow seeded LFO drift to params within a section, so nothing sits still even during a four-minute sustain.
  • Applies the reactive mappings on top: beat spikes, band-energy modulation.

Three timescales stacked — per-frame reactivity, per-section drift, whole-song arc — is what keeps a six-minute track from reading as a loop.


6. Preview

Non-negotiable requirement: the composition is fully reviewable before anything is exported. Export is slow and committing to a six-minute render to discover a bad transition at 4:10 is unacceptable. Preview is not a debug view — it is the primary working surface, and export is the thing you do once you're happy.

What it must do

  • Same everything as export. Identical compositor, identical look spec, identical FeatureTrack. The only permitted differences are output resolution and warm-up state (below). Preview must never make a different look decision than export would.
  • Transport: play / pause / scrub, frame step, and jump to previous/next section boundary — because transitions are what you actually need to review.
  • Loop a section while tuning it.
  • Live param editing with instant feedback, no track restart. Edits apply to the current section's param set and are savable.
  • Timeline strip under the viewport: sections as colored blocks by kind, beat/bar ticks, playhead, and markers for scene changes. Segmentation problems are visible at a glance here.
  • Re-roll controls: new track seed, re-roll just this section's scene, lock a section so further re-rolls leave it alone.
  • Quality toggle: draft (half resolution, post chain simplified) for smooth scrubbing on heavy stacks, versus full-quality preview at export settings but reduced resolution.
  • Segment test-render: export ~20 seconds around the playhead at full export quality. This is the bridge between "looks right in preview" and "commit to the full render", and it is the thing that makes a bad 4K export a rare event rather than a routine one.

The seek problem, honestly

Stateful layers — feedback buffers, particle systems — mean frame N depends on the frames before it. Scrubbing to an arbitrary frame therefore can't be exact for free. Three-part resolution:

  1. Warm-up. On seek, render K frames off-screen before displaying. K is computed from the feedback decay rather than fixed — the residual after n frames is decay^n, so log(0.001)/log(decay) frames converges to a tenth of a percent (96 frames at decay 0.93). A light-feedback look seeks almost instantly; a heavy one still lands.
  2. Export is always sequential, so it is exact everywhere by construction.

Correction, made during Phase 4. The original plan claimed section-boundary seeks would be frame-exact with no warm-up, because layer state is re-seeded there. That is wrong: layer state is only half the story, and the compositor's feedback buffer is global — it carries straight across a boundary like any other frame. Making boundary seeks exact would mean clearing feedback at every transition, which trades a cheap warm-up for a visible flash on every scene change. Warm-up is the better trade, and it applies everywhere rather than only mid-section. Measured: converges to a 0.00000 mean difference. With feedback disabled, seeks are bit-exact anywhere, which is what proves nothing else in the pipeline is carrying state.

Documented consequence: scrubbing shows a converged, not bit-exact, image whenever feedback is enabled — which is visually indistinguishable, and exact once feedback is off.


7. Export

Same engine, Timeline in fixed-step mode, rendering to an offscreen target at export resolution.

  • Primary: WebCodecs VideoEncoder. Hardware-accelerated H.264/VP9 straight from the render target, muxed with the decoded audio. No ffmpeg.wasm payload, and fast.
  • Fallback: frame dump + ffmpeg. A small vite dev-server middleware accepts POSTed frames and pipes them to a local ffmpeg. Slower, trivially debuggable.
  • Title card and fade in/out at the edges.
  • Progress reporting and cancel, since a 4K render is minutes not seconds.

8. Validation tooling

Built early (most of it in Phases 0-1) because every later phase depends on it. This is the difference between "seems fine" and "verified".

Tool What it catches
Frame hash log Renders N frames headless, hashes each (FNV over readPixels), writes JSON. Diffing two runs is the determinism regression test.
Click track Mixes an audible click on the detected beat grid into the track. You can hear whether tempo detection is right — far more reliable than watching visuals and guessing.
Section timeline overlay Detected sections and labels drawn over the transport. Bad segmentation is obvious immediately.
Feature scope Oscilloscope plot of selected features with playhead. Catches dead, constant, or saturated features — e.g. a band that's always 0 because the split was wrong.
Dual-resolution diff Renders one frame at 720p and 4K, downsamples, diffs. Catches resolution dependence, which is otherwise silent until export looks wrong.
Seed contact sheet Same frame across 16 seeds as a grid. Makes look-space collapse visible instantly — this is the primary defense against "sameness across tracks".
Schema lint Parses each scene's shader for uniform declarations and cross-checks against its params block, both directions. Catches typos and orphans; the thing that keeps a 30-scene library maintainable.
Param range sweep Renders each param at several points across its declared range, asserts no NaN, no all-black, no all-white frames.
Perf HUD Frame time, GPU time per layer, frame index, active section, live param values.
Flash-rate meter Counts light-dark cycles per second against the WCAG 2.3.1 / Harding ceiling of three. Added during Phase 5 and not in the original plan — this generates beat-reactive video for publication, an unsupervised generator finds unsafe states on its own, and nobody watches every frame of every export. It caught a scene running at 7-8 flashes/s at every output resolution.
First-render check Renders a frame in a fresh engine and compares against the same frame rendered later. Catches anything that is correct on repeat but wrong the first time — invisible to fresh-vs-fresh comparison, and wrong in every export, which renders each frame exactly once. Caught two separate bugs (a reused feature row, and unlinked shader programs).

The track battery

A fixed set of 6 of your own tracks spanning ambient → mid-tempo → hard EDM, checked in as the standing regression corpus (paths only, not the audio). Every phase gate below is evaluated against the same battery, so improvements and regressions are comparable across time. Pick these before Phase 1 — the analysis work is much easier to judge against real material you know well.


9. Phases, with gates

Each phase has an explicit gate. Don't start the next phase until the gate passes; these are cheap to check now and expensive to retrofit later.

Phase 0 — skeleton and determinism spine

Timeline, Renderer, Compositor with a single layer, seeded RNG, fullscreen quad. One ported shader with the LED grid stripped. Frame hash tool.

Gate:

  • 300 frames rendered twice on the same machine → bit-identical hashes.
  • Artificially stalling the render loop (simulated dropped frames) changes nothing in the output hashes — proves dt is truly fixed.
  • Dual-resolution diff on 5 sample frames passes the perceptual threshold.
  • Grep gate: zero occurrences of Math.random, performance.now, Date.now under src/scenes/ and src/engine/ outside of explicitly allowed spots.

Phase 1 — offline audio pipeline

decode, FeatureTrack, tempo and beat grid. The ported scene reads features by frame index instead of a live analyser. Click track, feature scope.

Gate:

  • Click track lines up by ear on all 6 battery tracks. If tempo is wrong anywhere, stop and fix it here — every downstream timing artifact traces back to this.
  • Feature sanity: no NaN or Inf anywhere in the table; every feature's realized range covers a meaningful span (flag any that are constant or pinned at an extreme).
  • Realtime-driven playback and fixed-step render of frames 1000-1100 → identical hashes.
  • Analysis of a 6-minute track completes in under ~3s.
  • Seeking to frame N after warm-up matches sequential playback to N within threshold.

Phase 2 — parameter schema, automatic binding, generated UI

Remaining four shaders ported with schemas. Schema lint, param range sweep.

Gate:

  • Schema lint clean on every scene, both directions.
  • Range sweep clean: no scene produces NaN, black or white frames anywhere in its declared ranges.
  • Every declared param appears in the generated UI, edits take visible effect, and values round-trip through save/load unchanged.
  • Ported scenes reviewed side-by-side against the party-stage originals — confirm nothing was lost in the port.

Phase 3 — look generation ("A" complete)

LookGenerator, PaletteSource, content-derived seed. Seed contact sheet.

Gate:

  • Same file loaded twice → identical seed → identical look spec (JSON compare).
  • Seed contact sheet shows genuine spread: mean pairwise perceptual distance across 16 seeds above a set threshold. This is the "sameness" gate — a collapsed look space fails here.
  • All 6 battery tracks produce meaningfully different look specs (different scene selections, distinguishable palettes).
  • Palettes pass a contrast/luminance-spread check — no muddy, low-separation sets.
  • Every scene survives every generated param set across the battery without crashing.

At this gate the tool is genuinely useful: drop a track in, get an uploadable video with no input.

Phase 4 — segmentation and arc driver

Sections, classification, cross-fades, lookahead ramps, intra-section drift.

Gate:

  • Hand-label section boundaries on the battery; detected boundaries score an acceptable F1 within ±2s tolerance. Record the number — it's the baseline for future tuning.
  • Transition check: frame-to-frame perceptual delta across the whole track, flagging spikes. No black frames or pops at boundaries except where intended.
  • Param traces logged over a build section show monotonic ramps into the following drop — proves lookahead is actually wired, not just present in the data.
  • Watch all 6 battery tracks end to end. Unavoidable and not substitutable. Budget the time; the failure mode this catches — subtle monotony — is invisible to every automated check.

Phase 5 — compositing depth ("C" complete)

Multi-layer, blend modes, feedback, post chain, 3D particle layers.

Gate:

  • Full stack holds 60fps at preview resolution; per-layer GPU cost recorded against a budget.
  • Feedback stability: 10,000-frame run neither saturates to white nor decays to black.
  • Determinism still passes with feedback and particles active — this is where deterministic buffer initialization gets tested for real.
  • Every layer renders correctly in isolation when soloed.

Phase 6 — export

WebCodecs path, title card, fades, progress and cancel.

Gate:

  • Preview/export parity: hashes of exported frames match preview-rendered frames for the same range.
  • A/V sync measured at start, middle and end of a 6-minute export — click track against video frames, drift within one frame.
  • ffprobe frame count and duration exactly as expected; no dropped or duplicated frames.
  • Exported audio matches the source.
  • Plays correctly in VLC and in a browser, and survives a real upload to the video site you actually publish on. Test this once, early, with a short file — container quirks are much cheaper to find now than after a 4K render.

Phase 7 — grow the library

Toward six families. Each new scene is a shader plus a schema block.

Per-scene checklist (the gate is per scene, not per phase):

  • Schema lint clean; range sweep clean; determinism hashes stable.
  • Within GPU budget at 4K.
  • Sits correctly in its declared family — reviewed in both a quiet and a loud section.
  • Library-wide regression contact sheet at a fixed seed, diffed against last known good, so shared-code changes can't silently break existing scenes.

Phase 8 — shots

Not in the original plan. It exists because the manual gate in Phase 4 caught exactly what it was written to catch: watching whole tracks, long stretches went by without the image changing. Sections are the song's stages, and a stage can run ninety seconds; one scene held for ninety seconds reads as a still image with a wobble on it, however reactive the wobble is.

So a third level sits between section and frame. Each section KIND gets a roster of three or four stage visuals instead of one scene, and each section is cut into shots that rotate between them on phrase lines — four to eight bars in a drop, eight to sixteen in an intro, never longer than twenty-two seconds. The roster stays per kind, so all of a track's drops still cut between the same visuals and the video keeps its identity; the first entry is the anchor, opens every section of that kind, and the rotation keeps returning to it. When a companion is due it is the least recently shown one, so a long section reaches its whole roster instead of ping-ponging between two images.

Mechanically this is one change: the arc driver stopped working in sections and started working in cues, one per shot, so a shot cut and a section change take the same code path and differ only in transition length. The default transition is a slow dissolve — two bars on calm material, one on loud, capped at 40% of the shot. A straight cut is reserved for genuinely high-energy sections: below the energy threshold nothing ever cuts, because on calm material a cut reads as a glitch rather than as an edit.

Gate:

  • No image held longer than the ceiling, and none shorter than the floor. Both measured across several seeds; the first is the complaint that started the phase, expressed as a number.
  • Intra-section cuts land within a quarter-bar of a downbeat.
  • Identity holds: a kind's roster is stable across its sections, the anchor opens each section and stays within one showing of the most-shown visual, and no variant repeats back to back.
  • A section with four or more shots reaches at least three distinct visuals — otherwise the rotation has collapsed back to A/B and the roster is decoration.
  • Dissolves outnumber cuts, and no section below the energy threshold cuts at all.
  • No black frame or discontinuity at a cut; the flash-rate sweep runs per shot rather than per section, so the visuals that only appear mid-section are measured too.
  • Watch the battery again. The phase exists because of a manual finding and the fix is a pacing judgement, which no delta threshold can make.

Phase 9 — production design

Also not in the original plan, and for the same reason as Phase 8: watching real tracks. With cuts every fifteen seconds the next problem became obvious — the images being cut between had nothing in common but the palette. That is a slideshow, not a music video.

What a music video actually shares across its shots is a location, a cast, a camera operator and an art direction. So each track now generates a personality in four traits, once, off the look seed:

trait what it fixes how a scene expresses it
shape the cast a signature form — round, or an n-gon at a tilt — stamped wherever a scene draws elements
camera the operator one slow returning pan, sway, roll and bar-locked breath, applied to the coordinate a scene works in
space the location a shared horizon height, depth falloff and background wash
style the art direction line weight, edge softness, surface grain, fold count

The traits reach shaders as ordinary uniforms plus four helper functions in the contract (sigShape, sigCamera, sigAir, sigEdge/sigGrain), so a scene honours a trait in its own way — the rings of Classic Wave become hexagonal, Metaballs merge as hexagons, Floating Geometry stops choosing between a box and a circle because the production already decided.

The part that makes it a design rather than a filter: each scene DECLARES which traits it honours, each track is built on one or two of them, and a scene that does not honour all of them is not cast in that track. The library shrinks per track on purpose. A scene with no way to draw a hexagon should not appear in the hexagon video.

Gate:

  • Personality reproduces exactly from the seed, and differs between seeds.
  • Every trait has at least six scenes, or a track built on it could not fill its rosters.
  • No scene is ever cast in a track it does not honour, and no signature starves a track below three distinct scenes (the fallback to a one-trait signature exists for this).
  • Every declared trait visibly changes the scene that declares it. The lint proves a scene mentions a trait; only rendering proves it matters. Measured as a frame delta against a one-LSB floor — the tolerance the determinism checks already call noise.
  • Changing the personality moves every scene in a track's cast, not just one.
  • A layer with no personality renders exactly what it rendered before the phase existed, which is what keeps every earlier sweep and regression valid.
  • Watch the battery. The target is that a viewer could name the through-line on a second viewing — not the first. No metric expresses that.

Phase 10 — variety

The third finding from watching real tracks, after Phase 8 (too few cuts) and Phase 9 (no through-line): the same scene cast in two different videos looked like the same footage twice. Section bias is nearly identical between two tracks' drops, so both sampled their parameters around the same centre, and the library's own averageness did the rest.

Three answers, none of them a new scene:

  • Temperament. A per-track hand on every parameter dial — intensity, pace, detail, and an extremity that decides how far toward the ends of a range the track is willing to sample. Bias comes from the section and is shared; temperament comes from the track and is not.
  • Overlays. Sometimes a second full scene is composited over the shot at partial opacity, from a different family, in a blend that preserves what is underneath. Not always — a stack that always doubled up would read as permanently cluttered rather than as occasionally layered.
  • A wider palette. Hue derives from spectral tilt — the log ratio of treble to body — rather than the centroid or a plain body fraction. Both of those collapse: the centroid is a number most masters sit in the middle of, and low frequencies carry most of the energy in all music, so the plain fraction read 0.98-1.00 for everything and four different battery tracks came out within 0.02 of each other. The ratio is multiplicative, so its logarithm is what spreads. Plus: both ways round the wheel (violet, magenta and pink were previously unreachable by construction), four new schemes, and seeded chroma profile and lightness curve.

The library also grew to 36 scenes, six per family — depth matters more than it looks, because the Phase 9 casting rule means the pool a given track draws from is smaller than the library.

Gate:

  • Temperaments spread across a battery rather than collapsing to one value.
  • One scene rendered under two tracks' parameters differs — measured RELATIVE to how much image there is, because most scenes are mostly dark and an absolute frame distance scores two genuinely different sparse renders as nearly identical.
  • Overlays occur on 5-55% of stacks, never at normal blend, never above 0.6 opacity.
  • Every sixth of the colour wheel is reachable across a sampled population, and two tracks that sound different do not get the same palette.

Tooling, added with Phase 10

Adding a scene was mostly boilerplate and round-trips, which is expensive in both senses.

  • npm run new:scene -- "Name" --family=… --traits=… writes the module, registers it, and leaves a skeleton that already passes every gate (with name-derived constants, so two skeletons are not identical to each other).
  • The lint grew the rules that used to need a GPU to catch: a dead camera (p = sigCamera(p) and then nothing reads p — a real scene shipped like that and the Phase 9 render gate measured its response at exactly zero), prev() with no base image, and large loops with no early break (with a // lint: fixed-cost opt-out).
  • checks.html?scene=Name runs the per-scene acceptance battery for ONE scene: ten lines and a verdict, instead of rendering the whole library to find out whether one shader is alive.
  • .claude/skills/build-visualizer/ is the same procedure as a repo skill.

10. Detachment from party-stage

Fork, copy what's useful, then detach completely. This matches how the rest of the repo works — each generator stands alone — and here it's also technically right: party-stage is causal and realtime by design, this is two-pass and offline, and a shared module would serve neither well.

Copied, then owned outright (edit freely, no upstream obligation):

  • the mulberry32 seeded PRNG
  • MediaStorage, the IndexedDB track persistence
  • config-UI patterns
  • the postprocessing setup
  • the five visualizer shaders

Referenced as prior art, not copied: music-visualizer.js. Its three-detector onset logic and band splits are a good specification for what to measure; the offline analyzer computes all of it better, so none of the code carries over.

Deliberately not ported: SceneFeature / SceneFeatureManager. Layer + Compositor is the right shape for a layer stack, and retrofitting the old pattern would fight the design.

Enforced: zero imports crossing the directory boundary, in either direction. A grep for party-stage under flow-state/src/ returns nothing. party-stage keeps working exactly as it does today and is never touched by this project.


11. Risks

  • Segmentation quality is the fuzziest component. Mitigation: energy-novelty heuristics only, tuned by ear against the battery; a misjudged section means a scene change in a slightly odd place, not a broken video. The Phase 4 F1 number keeps it honest.
  • Shader compile time grows with the library. Mitigation: compile lazily, only the scenes the look actually selected.
  • 4K export speed. Mitigation: WebCodecs, plus the fact that it's offline — slow is fine as long as preview stays fast.
  • Resolution-independence discipline is easy to violate silently. Mitigation: the dual-resolution diff, run every phase gate, not just Phase 0.
  • Monotony can survive every automated check. Only watching full tracks catches it. This is why Phase 4's gate includes end-to-end viewing and why the battery exists — and it's the risk most likely to be the one that actually bites.