Grain was in every video. It was added twice unconditionally — every scene called sigGrain, and the grade added its own on top — so the only thing that varied between two tracks was how much of it there was. That makes grain the renderer's fingerprint rather than a decision about one video. It is now described rather than dialled (look/grain.js): a mode (off / constant / swell / sections / transient), a cell size in pixels, a refresh rate in frames, a mask (uniform, shadows, highlights, edges, bands) and a chroma amount. Roughly 45% of tracks get none at all. The non-constant modes carry a per-frame envelope computed in Show._postAt from frame and features only, so preview and export still agree. Scene-side grain is gated the same way, and a module can decline it outright with `texture: 0` — crisp line work should stay crisp. The post tab grew a real grain block so any of it can be forced per track. Slow songs got fast scenes. `motion` bias was mostly section energy with tempo as a small correction, so a 70bpm track's drop asked for nearly as much speed as a 150bpm one. Motion is now tempo-dominated, and every `rate: true` param is additionally scaled by a per-track rateScale — measured, 84bpm now samples its rate params at 0.276 of range against 148bpm's 0.571. Parameter sampling also commits harder: extremity starts at 0.45 rather than 0.25 and shapes the draw more aggressively. This was first pushed to 0.82 and backed off to 0.72, because the gates caught the overshoot — seeds began collapsing onto the same range ends and a sparse scene sampled at its low end rendered effectively black. Fallout worth recording: turning the default grade grain off exposed two scenes that were never really animating. Dust Chamber and Eclipse Field passed the Phase 7 movement gate only because per-pixel noise was moving underneath them; both now breathe on their own fixed clock, and Dust Chamber needed a brightness floor as well. Four scenes (Classic Wave, Silk Ribbon, Kaleido Tunnel, Slow Orb) express the style trait ONLY through grain and so cannot opt out yet; they hold a reduced share at 0.35 pending real edge and softness response. Phase 3's look-space check now measures its closest pair relative to image brightness, the same correction Phase 10 already documents for sparse scenes: the absolute number was being propped up by grain rather than by look-space width. Five new Phase 10 checks cover grain distribution, treatment variety, envelope range, the texture opt-out, and tempo. 93/93 pass. Two transport bugs, both stale state surfacing in the UI: - Loading a track replaces audio.src, which stops playback silently, so state.playing stayed true and the play button stayed on pause — the first click after a track change only flipped the flag back. Added stopPlayback(). - The controls row wrapped mid-song because two readouts that change length while playing were sized by their content: the clock crossing ten minutes and the section label whenever a scene name is long or a crossfade appears. The clock is now fixed-width with tabular figures and the section label is the row's only flexible item, laying out at zero width and ellipsising into whatever space is left. The two spacers competed with it for that space and are gone; its own text-align does their job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
654 lines
36 KiB
Markdown
654 lines
36 KiB
Markdown
# flow-state — plan
|
|
|
|
An ambient/EDM music video generator. Drop in a track, get a full-length, non-story,
|
|
music-reactive video that evolves through the song. No sourced footage: every frame is
|
|
generated, and everything the look depends on is derived from the audio itself.
|
|
|
|
Name is a placeholder, easy to rename before Phase 0 lands.
|
|
|
|
## Decisions taken
|
|
|
|
| Question | Decision |
|
|
|---|---|
|
|
| Render model | Deterministic core. Realtime preview **and** offline frame-exact export, from the same code path. |
|
|
| Preview | **Required.** Full transport, section jumping, live param editing, and segment test-renders — see §8. |
|
|
| Art direction | Automatic from the audio, every derived parameter overridable and savable per track. |
|
|
| Scene architecture | Hybrid compositor. Layers are usually fullscreen fragment shaders; three.js particle/geometry layers are also valid layers. |
|
|
| Cover art | Not available. Palette is audio-derived. A `PaletteSource` seam is left so artwork can be added later without touching scenes. |
|
|
| Relationship to `party-stage` | Fork and **detach**. Copy what's useful, then zero cross-imports — see §12. |
|
|
|
|
## The two problems this has to solve
|
|
|
|
They are separate and both are load-bearing:
|
|
|
|
1. **Sameness across tracks.** Track #12 must not look like track #11. Solved by deriving
|
|
the look from the track's own measured character plus a content-derived seed.
|
|
2. **Monotony within a track.** A single shader gets dull around minute three. Solved by
|
|
segmenting the song and driving scene changes, palette shifts and parameter motion off
|
|
that structure.
|
|
|
|
Both have explicit validation gates in §11 — they are easy to *believe* you've solved and
|
|
hard to actually solve.
|
|
|
|
---
|
|
|
|
## 1. The determinism spine
|
|
|
|
This is the single architectural constraint everything else obeys. It is what makes
|
|
"realtime preview" and "4K60 offline export" the same program rather than two programs
|
|
that drift apart.
|
|
|
|
**Mechanism: the audio is fully analyzed before the first frame renders, into a
|
|
frame-indexed table.**
|
|
|
|
```
|
|
track file
|
|
↓ decode() OfflineAudioContext → AudioBuffer (a 6-min track decodes in ~1-2s)
|
|
↓ analyze() STFT at hop = 1/60 s, plus global passes
|
|
↓
|
|
FeatureTrack a typed-array table, one row per output frame
|
|
```
|
|
|
|
Nothing in the engine ever reads a live `AnalyserNode`. Realtime preview maps
|
|
`audio.currentTime → frameIndex` and reads row *n*. The offline exporter counts
|
|
`frameIndex` directly and reads row *n*. **Identical input to the visuals in both modes**,
|
|
so what you preview is exactly what you export.
|
|
|
|
This also unlocks the thing a causal analyser fundamentally cannot do:
|
|
|
|
> **Lookahead.** Because the whole song is analyzed up front, a build can *anticipate* its
|
|
> drop — ramp density and warp toward the drop's target values over the eight bars before
|
|
> it, and hit the transition already at full tension. A causal analyser can never do this;
|
|
> it only knows the drop landed after it landed. This is the biggest single visual win in
|
|
> the whole plan.
|
|
|
|
### Determinism rules (violating any of these breaks export)
|
|
|
|
- **No wall-clock anywhere in scene or layer code.** No `performance.now()`, no `Date`, no
|
|
`clock.getElapsedTime()`. A `Timeline` object injects `{ frame, time, dt, progress }`.
|
|
- **No `Math.random()` in anything that animates.** Seeded PRNG only (mulberry32, with the
|
|
seed held per-instance rather than on a global).
|
|
- **`dt` is constant** — `1/fps` — even in realtime preview. If the browser drops a frame,
|
|
preview shows a hitch; it does not change the animation. Frame 5400 is the same image
|
|
every run.
|
|
- **Deterministic buffer initialization.** Feedback and ping-pong targets must be explicitly
|
|
cleared at start; never inherit whatever was in GPU memory.
|
|
- **Resolution independence.** Scene math lives in normalized aspect-corrected coordinates.
|
|
Anything genuinely pixel-sized takes a `u_pixelScale = resolution / referenceResolution`
|
|
uniform. This is what lets a 720p preview match a 4K export.
|
|
|
|
### Stated limit on the guarantee
|
|
|
|
Determinism is **per-machine**: same browser, same GPU, same driver → bit-identical frames.
|
|
Across different GPUs, floating-point and `fwidth` derivative differences make bit-exactness
|
|
unrealistic, and chasing it would be wasted effort. The cross-machine guarantee is
|
|
*perceptual*: frames must match within a small diff threshold. The validation checks in §11
|
|
are written to that distinction, so the acceptance criteria are actually achievable.
|
|
|
|
Two refinements, both found by building it:
|
|
|
|
**Programs must be primed before the first frame.** Shader programs link asynchronously
|
|
(`KHR_parallel_shader_compile`), and a draw issued against an unlinked program produces
|
|
wrong output. Measured on the heaviest scene, the first *ten* frames rendered differently
|
|
from every later render of the same frames. Preview hides this completely — the frames go
|
|
past and the next pass is right — but an export renders each frame exactly once, so those
|
|
frames would ship broken. `Engine.prime()` compiles every program and discards a warm frame,
|
|
and the exporter calls it before encoding anything. Rendering a throwaway frame and reading
|
|
it back is *not* sufficient; `WebGLRenderer.compile()` is.
|
|
|
|
**Even same-machine, the heaviest shaders vary by one LSB.** With priming in place most
|
|
scenes reproduce byte-for-byte, but a few come back with scattered pixels differing by 1/255
|
|
— floating-point variance under differing GPU load. That is below any perceptual threshold
|
|
and is not something the hardware offers to fix, so the per-scene criterion is a max
|
|
channel delta of ≤ 1 rather than an identical hash. A real determinism bug scores in the
|
|
tens or hundreds on that metric, so the check keeps its teeth.
|
|
|
|
---
|
|
|
|
## 2. Audio analysis
|
|
|
|
`src/audio/`. Pure functions over an `AudioBuffer` — no DOM, no three.js — so it's unit
|
|
testable and can move to a worker without rework.
|
|
|
|
**Per frame** (hop 1/60 s, window 2048, Hann):
|
|
|
|
| Feature | Used for |
|
|
|---|---|
|
|
| `rms`, `loudness` | overall intensity, quiet-passage detection |
|
|
| band energies (sub / low / mid / high / air) | per-element reactivity |
|
|
| `flux` (spectral flux, half-wave rectified) | onset strength, the beat signal |
|
|
| `centroid` | perceived brightness → color temperature |
|
|
| `flatness` | noisy vs tonal → distinguishes pads from percussion |
|
|
| `stereoWidth` | width of the visual field |
|
|
|
|
**Global passes:**
|
|
|
|
- **Tempo** — autocorrelation over the onset envelope, octave-normalized to 90-180 BPM.
|
|
Whole-song, so it's exact rather than converging. Yields a phase-locked **beat grid** and
|
|
downbeats, hence exact `beatPhase` / `barPhase` / `phrasePhase` per frame.
|
|
- **Segmentation** — novelty curve from a self-similarity matrix over the band-energy
|
|
vectors, peak-picked into section boundaries.
|
|
- **Section classification** — each section labelled by energy percentile, flux density and
|
|
centroid: `intro | build | drop | sustain | breakdown | outro`. Deliberately simple
|
|
heuristics over features we already have; this is the fuzziest part of the pipeline and is
|
|
not worth an ML detour.
|
|
|
|
**Lookahead fields, per frame:** `timeToNextSection`, `nextSectionKind`, `nextSectionEnergy`,
|
|
`buildSlope`. These are what the arc driver leans on.
|
|
|
|
---
|
|
|
|
## 3. Scene modules and the parameter schema
|
|
|
|
**This is the decision that makes a large library affordable.** Adding a scene must cost a
|
|
shader and a schema block — no plumbing, no UI code, no wiring.
|
|
|
|
```js
|
|
export const NebulaDrift = {
|
|
name: 'Nebula Drift',
|
|
family: 'organic',
|
|
kind: 'fragment', // or 'layer3d'
|
|
shader: fragmentSource,
|
|
|
|
params: {
|
|
density: { type:'float', range:[0,1], default:0.5, uniform:'u_density' },
|
|
warp: { type:'float', range:[0,3], default:1.0, uniform:'u_warp' },
|
|
symmetry: { type:'int', range:[1,8], default:1, uniform:'u_symmetry' },
|
|
grain: { type:'float', range:[0,1], default:0.2, uniform:'u_grain' },
|
|
palette: { type:'palette', count:5, uniform:'u_colors' },
|
|
},
|
|
|
|
// which audio features modulate which params, and how hard
|
|
reactive: {
|
|
density: { feature:'bandLow', amount:0.3 },
|
|
warp: { feature:'beat', amount:0.5, response:'spike' },
|
|
grain: { feature:'flatness', amount:0.2 },
|
|
},
|
|
};
|
|
```
|
|
|
|
One schema, five payoffs:
|
|
|
|
1. Uniforms bind automatically — no per-scene wiring.
|
|
2. The config UI generates its own sliders, so override-ability is free for every new scene.
|
|
3. `LookGenerator` samples the declared ranges with the track's seed → per-track variation
|
|
with no hand-tuning.
|
|
4. `ArcDriver` can animate any declared param across a section boundary generically.
|
|
5. Presets are just serialized param sets, savable per track.
|
|
|
|
It also makes scenes **machine-checkable**, which is what keeps a thirty-scene library from
|
|
rotting — see the schema lint in §10.
|
|
|
|
### Library targets
|
|
|
|
Interest across songs comes from breadth of *family*, not raw count. Six families, growing
|
|
to roughly 4-6 scenes each:
|
|
|
|
| Family | Character | Fits |
|
|
|---|---|---|
|
|
| **flow** | curl noise, fluid, drifting particles | ambient, sustain |
|
|
| **geometric** | tunnels, kaleidoscope, grids, synthwave | drops |
|
|
| **organic** | nebula, plasma, reaction-diffusion, metaballs | sustain, breakdown |
|
|
| **structural** | raymarched terrain, cities, architecture | builds |
|
|
| **minimal** | lines, waveform, spectrum sculpture, negative space | intros, quiet passages |
|
|
| **glitch** | feedback, datamosh, scanlines, chromatic tearing | drops, transitions |
|
|
|
|
The arc driver picks a *family* from section kind, then a scene within it from the seed — so
|
|
a quiet passage never lands on a strobing glitch scene, and two tracks with similar structure
|
|
still choose differently.
|
|
|
|
### Seeded from the five that exist
|
|
|
|
`party-stage`'s five visualizer shaders are the library's starting point. Porting each means:
|
|
strip the LED-grid mask, replace hardcoded colors with palette lookups, pull magic numbers
|
|
out into declared params, swap live uniforms for `FeatureTrack` reads. Note their
|
|
`u_resolution` is not pixels — it's the LED grid cell count. Rename it `u_gridSize` on port
|
|
so it doesn't collide with the real resolution uniform.
|
|
|
|
---
|
|
|
|
## 4. Compositor
|
|
|
|
`src/engine/Compositor.js`. A layer stack rendered to ping-pong render targets.
|
|
|
|
- **Layers**: fragment-shader (a fullscreen quad) or 3D (a three.js sub-scene — particles,
|
|
geometry, camera path). Both present the same `render(target, timeline, features)` face,
|
|
so the stack doesn't care which it holds.
|
|
- **Blend modes** per layer: normal, add, screen, multiply, overlay.
|
|
- **Feedback buffer** — previous frame available to any layer as a texture, with configurable
|
|
decay and warp. Cheap, and disproportionately responsible for things looking "alive".
|
|
- **Post chain**: bloom, chromatic aberration, grain, vignette, final grade.
|
|
|
|
Typical stack: background scene → accent scene (screen blend, low opacity) → particles →
|
|
feedback → post.
|
|
|
|
---
|
|
|
|
## 5. Look generation and the arc driver
|
|
|
|
**`LookGenerator`** runs once per track, after analysis:
|
|
|
|
```
|
|
seed = hash(decoded PCM) → same track always renders identically
|
|
palette = f(centroid, flatness, energy distribution, seed)
|
|
sceneAssignments = per section: family from kind, scene from seed
|
|
paramSets = sampled from declared ranges, biased by track character
|
|
```
|
|
|
|
`PaletteSource` is an interface here, with an `AudioPalette` implementation. Adding a
|
|
`CoverArtPalette` later is a new file and one line of config — no scene changes.
|
|
|
|
**`ArcDriver`** runs per frame:
|
|
|
|
- Cross-fades scenes at section boundaries (compositor holds both briefly, opacity ramp).
|
|
- Ramps params toward the *next* section's targets during builds, using the lookahead fields.
|
|
- Applies slow seeded LFO drift to params within a section, so nothing sits still even
|
|
during a four-minute sustain.
|
|
- Applies the `reactive` mappings on top: beat spikes, band-energy modulation.
|
|
|
|
Three timescales stacked — per-frame reactivity, per-section drift, whole-song arc — is what
|
|
keeps a six-minute track from reading as a loop.
|
|
|
|
---
|
|
|
|
## 6. Preview
|
|
|
|
**Non-negotiable requirement: the composition is fully reviewable before anything is
|
|
exported.** Export is slow and committing to a six-minute render to discover a bad transition
|
|
at 4:10 is unacceptable. Preview is not a debug view — it is the primary working surface, and
|
|
export is the thing you do once you're happy.
|
|
|
|
### What it must do
|
|
|
|
- **Same everything as export.** Identical compositor, identical look spec, identical
|
|
`FeatureTrack`. The *only* permitted differences are output resolution and warm-up state
|
|
(below). Preview must never make a different look decision than export would.
|
|
- **Transport**: play / pause / scrub, frame step, and **jump to previous/next section
|
|
boundary** — because transitions are what you actually need to review.
|
|
- **Loop a section** while tuning it.
|
|
- **Live param editing** with instant feedback, no track restart. Edits apply to the current
|
|
section's param set and are savable.
|
|
- **Timeline strip** under the viewport: sections as colored blocks by kind, beat/bar ticks,
|
|
playhead, and markers for scene changes. Segmentation problems are visible at a glance here.
|
|
- **Re-roll controls**: new track seed, re-roll just this section's scene, **lock** a section
|
|
so further re-rolls leave it alone.
|
|
- **Quality toggle**: draft (half resolution, post chain simplified) for smooth scrubbing on
|
|
heavy stacks, versus full-quality preview at export settings but reduced resolution.
|
|
- **Segment test-render**: export ~20 seconds around the playhead at full export quality.
|
|
This is the bridge between "looks right in preview" and "commit to the full render", and
|
|
it is the thing that makes a bad 4K export a rare event rather than a routine one.
|
|
|
|
### The seek problem, honestly
|
|
|
|
Stateful layers — feedback buffers, particle systems — mean frame *N* depends on the frames
|
|
before it. Scrubbing to an arbitrary frame therefore can't be exact for free. Three-part
|
|
resolution:
|
|
|
|
1. **Warm-up.** On seek, render *K* frames off-screen before displaying. *K* is computed
|
|
from the feedback decay rather than fixed — the residual after *n* frames is `decay^n`,
|
|
so `log(0.001)/log(decay)` frames converges to a tenth of a percent (96 frames at
|
|
decay 0.93). A light-feedback look seeks almost instantly; a heavy one still lands.
|
|
2. **Export is always sequential**, so it is exact everywhere by construction.
|
|
|
|
> **Correction, made during Phase 4.** The original plan claimed section-boundary seeks
|
|
> would be frame-exact with no warm-up, because layer state is re-seeded there. That is
|
|
> wrong: layer state is only half the story, and the compositor's feedback buffer is
|
|
> *global* — it carries straight across a boundary like any other frame. Making boundary
|
|
> seeks exact would mean clearing feedback at every transition, which trades a cheap
|
|
> warm-up for a visible flash on every scene change. Warm-up is the better trade, and it
|
|
> applies everywhere rather than only mid-section. Measured: converges to a 0.00000 mean
|
|
> difference. With feedback disabled, seeks are bit-exact anywhere, which is what proves
|
|
> nothing *else* in the pipeline is carrying state.
|
|
|
|
Documented consequence: scrubbing shows a *converged*, not bit-exact, image whenever
|
|
feedback is enabled — which is visually indistinguishable, and exact once feedback is off.
|
|
|
|
---
|
|
|
|
## 7. Export
|
|
|
|
Same engine, `Timeline` in fixed-step mode, rendering to an offscreen target at export
|
|
resolution.
|
|
|
|
- **Primary: WebCodecs `VideoEncoder`.** Hardware-accelerated H.264/VP9 straight from the
|
|
render target, muxed with the decoded audio. No ffmpeg.wasm payload, and fast.
|
|
- **Fallback: frame dump + ffmpeg.** A small vite dev-server middleware accepts POSTed frames
|
|
and pipes them to a local `ffmpeg`. Slower, trivially debuggable.
|
|
- Title card and fade in/out at the edges.
|
|
- Progress reporting and cancel, since a 4K render is minutes not seconds.
|
|
|
|
---
|
|
|
|
## 8. Validation tooling
|
|
|
|
Built early (most of it in Phases 0-1) because every later phase depends on it. This is the
|
|
difference between "seems fine" and "verified".
|
|
|
|
| Tool | What it catches |
|
|
|---|---|
|
|
| **Frame hash log** | Renders *N* frames headless, hashes each (FNV over `readPixels`), writes JSON. Diffing two runs is the determinism regression test. |
|
|
| **Click track** | Mixes an audible click on the detected beat grid into the track. You can *hear* whether tempo detection is right — far more reliable than watching visuals and guessing. |
|
|
| **Section timeline overlay** | Detected sections and labels drawn over the transport. Bad segmentation is obvious immediately. |
|
|
| **Feature scope** | Oscilloscope plot of selected features with playhead. Catches dead, constant, or saturated features — e.g. a band that's always 0 because the split was wrong. |
|
|
| **Dual-resolution diff** | Renders one frame at 720p and 4K, downsamples, diffs. Catches resolution dependence, which is otherwise silent until export looks wrong. |
|
|
| **Seed contact sheet** | Same frame across 16 seeds as a grid. Makes look-space collapse visible instantly — this is the primary defense against "sameness across tracks". |
|
|
| **Schema lint** | Parses each scene's shader for `uniform` declarations and cross-checks against its `params` block, both directions. Catches typos and orphans; the thing that keeps a 30-scene library maintainable. |
|
|
| **Param range sweep** | Renders each param at several points across its declared range, asserts no NaN, no all-black, no all-white frames. |
|
|
| **Perf HUD** | Frame time, GPU time per layer, frame index, active section, live param values. |
|
|
| **Flash-rate meter** | Counts light-dark cycles per second against the WCAG 2.3.1 / Harding ceiling of three. Added during Phase 5 and not in the original plan — this generates beat-reactive video for publication, an unsupervised generator finds unsafe states on its own, and nobody watches every frame of every export. It caught a scene running at 7-8 flashes/s at *every* output resolution. |
|
|
| **First-render check** | Renders a frame in a fresh engine and compares against the same frame rendered later. Catches anything that is correct on repeat but wrong the first time — invisible to fresh-vs-fresh comparison, and wrong in every export, which renders each frame exactly once. Caught two separate bugs (a reused feature row, and unlinked shader programs). |
|
|
|
|
### The track battery
|
|
|
|
A fixed set of **6 of your own tracks** spanning ambient → mid-tempo → hard EDM, checked in
|
|
as the standing regression corpus (paths only, not the audio). Every phase gate below is
|
|
evaluated against the same battery, so improvements and regressions are comparable across
|
|
time. Pick these before Phase 1 — the analysis work is much easier to judge against real
|
|
material you know well.
|
|
|
|
---
|
|
|
|
## 9. Phases, with gates
|
|
|
|
Each phase has an explicit gate. Don't start the next phase until the gate passes; these are
|
|
cheap to check now and expensive to retrofit later.
|
|
|
|
### Phase 0 — skeleton and determinism spine
|
|
`Timeline`, `Renderer`, `Compositor` with a single layer, seeded RNG, fullscreen quad. One
|
|
ported shader with the LED grid stripped. Frame hash tool.
|
|
|
|
**Gate:**
|
|
- 300 frames rendered twice on the same machine → **bit-identical hashes**.
|
|
- Artificially stalling the render loop (simulated dropped frames) changes nothing in the
|
|
output hashes — proves `dt` is truly fixed.
|
|
- Dual-resolution diff on 5 sample frames passes the perceptual threshold.
|
|
- Grep gate: zero occurrences of `Math.random`, `performance.now`, `Date.now` under
|
|
`src/scenes/` and `src/engine/` outside of explicitly allowed spots.
|
|
|
|
### Phase 1 — offline audio pipeline
|
|
`decode`, `FeatureTrack`, tempo and beat grid. The ported scene reads features by frame index
|
|
instead of a live analyser. Click track, feature scope.
|
|
|
|
**Gate:**
|
|
- **Click track lines up by ear on all 6 battery tracks.** If tempo is wrong anywhere, stop
|
|
and fix it here — every downstream timing artifact traces back to this.
|
|
- Feature sanity: no NaN or Inf anywhere in the table; every feature's realized range covers
|
|
a meaningful span (flag any that are constant or pinned at an extreme).
|
|
- Realtime-driven playback and fixed-step render of frames 1000-1100 → identical hashes.
|
|
- Analysis of a 6-minute track completes in under ~3s.
|
|
- Seeking to frame *N* after warm-up matches sequential playback to *N* within threshold.
|
|
|
|
### Phase 2 — parameter schema, automatic binding, generated UI
|
|
Remaining four shaders ported with schemas. Schema lint, param range sweep.
|
|
|
|
**Gate:**
|
|
- Schema lint clean on every scene, both directions.
|
|
- Range sweep clean: no scene produces NaN, black or white frames anywhere in its declared
|
|
ranges.
|
|
- Every declared param appears in the generated UI, edits take visible effect, and values
|
|
round-trip through save/load unchanged.
|
|
- Ported scenes reviewed side-by-side against the `party-stage` originals — confirm nothing
|
|
was lost in the port.
|
|
|
|
### Phase 3 — look generation ("A" complete)
|
|
`LookGenerator`, `PaletteSource`, content-derived seed. Seed contact sheet.
|
|
|
|
**Gate:**
|
|
- Same file loaded twice → identical seed → identical look spec (JSON compare).
|
|
- **Seed contact sheet shows genuine spread**: mean pairwise perceptual distance across 16
|
|
seeds above a set threshold. This is the "sameness" gate — a collapsed look space fails here.
|
|
- All 6 battery tracks produce meaningfully different look specs (different scene selections,
|
|
distinguishable palettes).
|
|
- Palettes pass a contrast/luminance-spread check — no muddy, low-separation sets.
|
|
- Every scene survives every generated param set across the battery without crashing.
|
|
|
|
*At this gate the tool is genuinely useful: drop a track in, get an uploadable video with no
|
|
input.*
|
|
|
|
### Phase 4 — segmentation and arc driver
|
|
Sections, classification, cross-fades, lookahead ramps, intra-section drift.
|
|
|
|
**Gate:**
|
|
- Hand-label section boundaries on the battery; detected boundaries score an acceptable F1
|
|
within ±2s tolerance. Record the number — it's the baseline for future tuning.
|
|
- Transition check: frame-to-frame perceptual delta across the whole track, flagging spikes.
|
|
No black frames or pops at boundaries except where intended.
|
|
- Param traces logged over a build section show monotonic ramps into the following drop —
|
|
proves lookahead is actually wired, not just present in the data.
|
|
- **Watch all 6 battery tracks end to end.** Unavoidable and not substitutable. Budget the
|
|
time; the failure mode this catches — subtle monotony — is invisible to every automated check.
|
|
|
|
### Phase 5 — compositing depth ("C" complete)
|
|
Multi-layer, blend modes, feedback, post chain, 3D particle layers.
|
|
|
|
**Gate:**
|
|
- Full stack holds 60fps at preview resolution; per-layer GPU cost recorded against a budget.
|
|
- Feedback stability: 10,000-frame run neither saturates to white nor decays to black.
|
|
- Determinism still passes with feedback and particles active — this is where deterministic
|
|
buffer initialization gets tested for real.
|
|
- Every layer renders correctly in isolation when soloed.
|
|
|
|
### Phase 6 — export
|
|
WebCodecs path, title card, fades, progress and cancel.
|
|
|
|
**Gate:**
|
|
- **Preview/export parity**: hashes of exported frames match preview-rendered frames for the
|
|
same range.
|
|
- A/V sync measured at start, middle and end of a 6-minute export — click track against video
|
|
frames, drift within one frame.
|
|
- `ffprobe` frame count and duration exactly as expected; no dropped or duplicated frames.
|
|
- Exported audio matches the source.
|
|
- Plays correctly in VLC and in a browser, **and survives a real upload** to the video site
|
|
you actually publish on. Test this once, early, with a short file — container quirks are
|
|
much cheaper to find now than after a 4K render.
|
|
|
|
### Phase 7 — grow the library
|
|
Toward six families. Each new scene is a shader plus a schema block.
|
|
|
|
**Per-scene checklist** (the gate is per scene, not per phase):
|
|
- Schema lint clean; range sweep clean; determinism hashes stable.
|
|
- Within GPU budget at 4K.
|
|
- Sits correctly in its declared family — reviewed in both a quiet and a loud section.
|
|
- Library-wide regression contact sheet at a fixed seed, diffed against last known good, so
|
|
shared-code changes can't silently break existing scenes.
|
|
|
|
### Phase 8 — shots
|
|
Not in the original plan. It exists because the manual gate in Phase 4 caught exactly what it
|
|
was written to catch: watching whole tracks, long stretches went by without the image
|
|
changing. Sections are the song's *stages*, and a stage can run ninety seconds; one scene held
|
|
for ninety seconds reads as a still image with a wobble on it, however reactive the wobble is.
|
|
|
|
So a third level sits between section and frame. Each section KIND gets a roster of three or
|
|
four **stage visuals** instead of one scene, and each section is cut into **shots** that
|
|
rotate between them on phrase lines — four to eight bars in a drop, eight to sixteen in an
|
|
intro, never longer than twenty-two seconds. The roster stays per kind, so all of a track's
|
|
drops still cut between the same visuals and the video keeps its identity; the first entry is
|
|
the anchor, opens every section of that kind, and the rotation keeps returning to it. When a
|
|
companion is due it is the least recently shown one, so a long section reaches its whole
|
|
roster instead of ping-ponging between two images.
|
|
|
|
Mechanically this is one change: the arc driver stopped working in sections and started
|
|
working in **cues**, one per shot, so a shot cut and a section change take the same code path
|
|
and differ only in transition length. The default transition is a slow dissolve — two bars on
|
|
calm material, one on loud, capped at 40% of the shot. A straight cut is reserved for genuinely
|
|
high-energy sections: below the energy threshold nothing ever cuts, because on calm material a
|
|
cut reads as a glitch rather than as an edit.
|
|
|
|
**Gate:**
|
|
- No image held longer than the ceiling, and none shorter than the floor. Both measured across
|
|
several seeds; the first is the complaint that started the phase, expressed as a number.
|
|
- Intra-section cuts land within a quarter-bar of a downbeat.
|
|
- Identity holds: a kind's roster is stable across its sections, the anchor opens each section
|
|
and stays within one showing of the most-shown visual, and no variant repeats back to back.
|
|
- A section with four or more shots reaches at least three distinct visuals — otherwise the
|
|
rotation has collapsed back to A/B and the roster is decoration.
|
|
- Dissolves outnumber cuts, and no section below the energy threshold cuts at all.
|
|
- No black frame or discontinuity at a cut; the flash-rate sweep runs per shot rather than per
|
|
section, so the visuals that only appear mid-section are measured too.
|
|
- **Watch the battery again.** The phase exists because of a manual finding and the fix is a
|
|
pacing judgement, which no delta threshold can make.
|
|
|
|
### Phase 9 — production design
|
|
Also not in the original plan, and for the same reason as Phase 8: watching real tracks. With
|
|
cuts every fifteen seconds the next problem became obvious — the images being cut between had
|
|
nothing in common but the palette. That is a slideshow, not a music video.
|
|
|
|
What a music video actually shares across its shots is a location, a cast, a camera operator
|
|
and an art direction. So each track now generates a **personality** in four traits, once, off
|
|
the look seed:
|
|
|
|
| trait | what it fixes | how a scene expresses it |
|
|
|---|---|---|
|
|
| `shape` | the cast | a signature form — round, or an n-gon at a tilt — stamped wherever a scene draws elements |
|
|
| `camera` | the operator | one slow returning pan, sway, roll and bar-locked breath, applied to the coordinate a scene works in |
|
|
| `space` | the location | a shared horizon height, depth falloff and background wash |
|
|
| `style` | the art direction | line weight, edge softness, surface grain, fold count |
|
|
|
|
The traits reach shaders as ordinary uniforms plus four helper functions in the contract
|
|
(`sigShape`, `sigCamera`, `sigAir`, `sigEdge`/`sigGrain`), so a scene honours a trait in its own
|
|
way — the rings of Classic Wave become hexagonal, Metaballs merge as hexagons, Floating
|
|
Geometry stops choosing between a box and a circle because the production already decided.
|
|
|
|
The part that makes it a design rather than a filter: each scene DECLARES which traits it
|
|
honours, each track is built on one or two of them, and **a scene that does not honour all of
|
|
them is not cast in that track**. The library shrinks per track on purpose. A scene with no way
|
|
to draw a hexagon should not appear in the hexagon video.
|
|
|
|
**Gate:**
|
|
- Personality reproduces exactly from the seed, and differs between seeds.
|
|
- Every trait has at least six scenes, or a track built on it could not fill its rosters.
|
|
- No scene is ever cast in a track it does not honour, and no signature starves a track below
|
|
three distinct scenes (the fallback to a one-trait signature exists for this).
|
|
- **Every declared trait visibly changes the scene that declares it.** The lint proves a scene
|
|
mentions a trait; only rendering proves it matters. Measured as a frame delta against a
|
|
one-LSB floor — the tolerance the determinism checks already call noise.
|
|
- Changing the personality moves every scene in a track's cast, not just one.
|
|
- A layer with no personality renders exactly what it rendered before the phase existed, which
|
|
is what keeps every earlier sweep and regression valid.
|
|
- **Watch the battery.** The target is that a viewer could name the through-line on a second
|
|
viewing — not the first. No metric expresses that.
|
|
|
|
### Phase 10 — variety
|
|
The third finding from watching real tracks, after Phase 8 (too few cuts) and Phase 9 (no
|
|
through-line): **the same scene cast in two different videos looked like the same footage
|
|
twice.** Section bias is nearly identical between two tracks' drops, so both sampled their
|
|
parameters around the same centre, and the library's own averageness did the rest.
|
|
|
|
Three answers, none of them a new scene:
|
|
|
|
- **Temperament.** A per-track hand on every parameter dial — intensity, pace, detail, and an
|
|
*extremity* that decides how far toward the ends of a range the track is willing to sample.
|
|
Bias comes from the section and is shared; temperament comes from the track and is not.
|
|
- **Overlays.** Sometimes a second full scene is composited over the shot at partial opacity,
|
|
from a different family, in a blend that preserves what is underneath. Not always — a stack
|
|
that always doubled up would read as permanently cluttered rather than as occasionally
|
|
layered.
|
|
- **A wider palette.** Hue derives from **spectral tilt** — the log ratio of treble to body —
|
|
rather than the centroid or a plain body fraction. Both of those collapse: the centroid is a
|
|
number most masters sit in the middle of, and low frequencies carry most of the energy in
|
|
all music, so the plain fraction read 0.98-1.00 for everything and four different battery
|
|
tracks came out within 0.02 of each other. The ratio is multiplicative, so its logarithm is
|
|
what spreads. Plus: both ways round the wheel (violet, magenta and pink were previously
|
|
unreachable by construction), four new schemes, and seeded chroma profile and lightness
|
|
curve.
|
|
|
|
The library also grew to **36 scenes, six per family** — depth matters more than it looks,
|
|
because the Phase 9 casting rule means the pool a given track draws from is smaller than the
|
|
library.
|
|
|
|
**Gate:**
|
|
- Temperaments spread across a battery rather than collapsing to one value.
|
|
- One scene rendered under two tracks' parameters differs — measured RELATIVE to how much
|
|
image there is, because most scenes are mostly dark and an absolute frame distance scores
|
|
two genuinely different sparse renders as nearly identical.
|
|
- Overlays occur on 5-55% of stacks, never at `normal` blend, never above 0.6 opacity.
|
|
- Every sixth of the colour wheel is reachable across a sampled population, and two tracks
|
|
that sound different do not get the same palette.
|
|
|
|
### Tooling, added with Phase 10
|
|
Adding a scene was mostly boilerplate and round-trips, which is expensive in both senses.
|
|
|
|
- `npm run new:scene -- "Name" --family=… --traits=…` writes the module, registers it, and
|
|
leaves a skeleton that already passes every gate (with name-derived constants, so two
|
|
skeletons are not identical to each other).
|
|
- The lint grew the rules that used to need a GPU to catch: a **dead camera**
|
|
(`p = sigCamera(p)` and then nothing reads `p` — a real scene shipped like that and the
|
|
Phase 9 render gate measured its response at exactly zero), `prev()` with no base image, and
|
|
large loops with no early break (with a `// lint: fixed-cost` opt-out).
|
|
- `checks.html?scene=Name` runs the per-scene acceptance battery for ONE scene: ten lines and
|
|
a verdict, instead of rendering the whole library to find out whether one shader is alive.
|
|
- `.claude/skills/build-visualizer/` is the same procedure as a repo skill.
|
|
|
|
### Phase 10b — grain as a treatment, tempo as a governor
|
|
|
|
Watching the library again surfaced two things no gate was asking about.
|
|
|
|
**Grain was in every video.** It was added twice unconditionally — every scene called
|
|
`sigGrain`, and the grade added its own on top — so the only thing that varied between two
|
|
tracks was how much. That makes grain the renderer's fingerprint rather than a decision about
|
|
one video. It is now DESCRIBED (`look/grain.js`): a mode (`off` / `constant` / `swell` /
|
|
`sections` / `transient`), a cell size in pixels, a refresh rate in frames, a mask (uniform,
|
|
shadows, highlights, edges, bands) and a chroma amount. Roughly 45% of tracks get none at all;
|
|
the non-constant modes have a per-frame envelope computed in `Show._postAt` from frame and
|
|
features only, so preview and export still agree. Scene-side grain is gated the same way, and
|
|
a module can decline it outright with `texture: 0` — crisp line work should stay crisp.
|
|
|
|
**Slow songs got fast scenes.** `motion` bias was mostly section energy with tempo as a small
|
|
correction, so a 70bpm track's drop asked for nearly as much speed as a 150bpm one. Motion is
|
|
now tempo-dominated, and every `rate: true` param is additionally scaled by a per-track
|
|
`rateScale`, so absolute animation speed follows the song rather than only the sampled
|
|
position in a range.
|
|
|
|
Parameter sampling also commits harder: `extremity` starts at 0.5 rather than 0.25 and shapes
|
|
the draw more aggressively, because a range is the scene author's statement of what the scene
|
|
can survive and a library that samples the middle of every range shows every scene's default.
|
|
|
|
**Gate:** a quarter to two thirds of tracks have no grain and at least four modes appear;
|
|
thirty grainy tracks produce mostly-distinct treatments; every time-varying grain reaches both
|
|
zero and full; a `texture: 0` scene gets nothing even from a maximally gritty track; and a
|
|
84bpm track's rate params sample materially below a 148bpm track's.
|
|
|
|
---
|
|
|
|
## 10. Detachment from `party-stage`
|
|
|
|
Fork, copy what's useful, then **detach completely**. This matches how the rest of the repo
|
|
works — each generator stands alone — and here it's also technically right: `party-stage` is
|
|
causal and realtime by design, this is two-pass and offline, and a shared module would serve
|
|
neither well.
|
|
|
|
**Copied, then owned outright** (edit freely, no upstream obligation):
|
|
- the mulberry32 seeded PRNG
|
|
- `MediaStorage`, the IndexedDB track persistence
|
|
- config-UI patterns
|
|
- the postprocessing setup
|
|
- the five visualizer shaders
|
|
|
|
**Referenced as prior art, not copied**: `music-visualizer.js`. Its three-detector onset logic
|
|
and band splits are a good specification for *what to measure*; the offline analyzer computes
|
|
all of it better, so none of the code carries over.
|
|
|
|
**Deliberately not ported**: `SceneFeature` / `SceneFeatureManager`. `Layer` + `Compositor` is
|
|
the right shape for a layer stack, and retrofitting the old pattern would fight the design.
|
|
|
|
**Enforced**: zero imports crossing the directory boundary, in either direction. A grep for
|
|
`party-stage` under `flow-state/src/` returns nothing. `party-stage` keeps working exactly as
|
|
it does today and is never touched by this project.
|
|
|
|
---
|
|
|
|
## 11. Risks
|
|
|
|
- **Segmentation quality** is the fuzziest component. Mitigation: energy-novelty heuristics
|
|
only, tuned by ear against the battery; a misjudged section means a scene change in a
|
|
slightly odd place, not a broken video. The Phase 4 F1 number keeps it honest.
|
|
- **Shader compile time** grows with the library. Mitigation: compile lazily, only the scenes
|
|
the look actually selected.
|
|
- **4K export speed.** Mitigation: WebCodecs, plus the fact that it's offline — slow is fine
|
|
as long as preview stays fast.
|
|
- **Resolution-independence discipline** is easy to violate silently. Mitigation: the
|
|
dual-resolution diff, run every phase gate, not just Phase 0.
|
|
- **Monotony can survive every automated check.** Only watching full tracks catches it. This
|
|
is why Phase 4's gate includes end-to-end viewing and why the battery exists — and it's the
|
|
risk most likely to be the one that actually bites.
|