From e8fb647f1128b6e87de5d3b194bc48de57d31c86 Mon Sep 17 00:00:00 2001 From: Dejvino Date: Wed, 5 Aug 2026 10:49:14 +0200 Subject: [PATCH] Plan: flow-state ambient/EDM video generator Design document for a new generator that produces full-length, non-story music videos from a track alone, with no sourced footage. Core decisions: - Deterministic engine: audio is analyzed up front into a frame-indexed feature table, so realtime preview and offline export render from the same code path and produce the same frames. - Whole-song analysis enables lookahead, so builds can anticipate drops. - Scenes declare their parameters in a schema, which drives uniform binding, generated UI, seeded per-track variation and arc automation. - Hybrid compositor: fragment-shader and three.js layers in one stack. - Preview is a first-class surface with section jumping, live param editing and segment test-renders. - Forked from party-stage by copy, then fully detached. Includes per-phase gates and the validation tooling they depend on. Co-Authored-By: Claude Opus 5 --- flow-state/PLAN.md | 468 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 468 insertions(+) create mode 100644 flow-state/PLAN.md diff --git a/flow-state/PLAN.md b/flow-state/PLAN.md new file mode 100644 index 0000000..700fe0a --- /dev/null +++ b/flow-state/PLAN.md @@ -0,0 +1,468 @@ +# flow-state — plan + +An ambient/EDM music video generator. Drop in a track, get a full-length, non-story, +music-reactive video that evolves through the song. No sourced footage: every frame is +generated, and everything the look depends on is derived from the audio itself. + +Name is a placeholder, easy to rename before Phase 0 lands. + +## Decisions taken + +| Question | Decision | +|---|---| +| Render model | Deterministic core. Realtime preview **and** offline frame-exact export, from the same code path. | +| Preview | **Required.** Full transport, section jumping, live param editing, and segment test-renders — see §8. | +| Art direction | Automatic from the audio, every derived parameter overridable and savable per track. | +| Scene architecture | Hybrid compositor. Layers are usually fullscreen fragment shaders; three.js particle/geometry layers are also valid layers. | +| Cover art | Not available. Palette is audio-derived. A `PaletteSource` seam is left so artwork can be added later without touching scenes. | +| Relationship to `party-stage` | Fork and **detach**. Copy what's useful, then zero cross-imports — see §12. | + +## The two problems this has to solve + +They are separate and both are load-bearing: + +1. **Sameness across tracks.** Track #12 must not look like track #11. Solved by deriving + the look from the track's own measured character plus a content-derived seed. +2. **Monotony within a track.** A single shader gets dull around minute three. Solved by + segmenting the song and driving scene changes, palette shifts and parameter motion off + that structure. + +Both have explicit validation gates in §11 — they are easy to *believe* you've solved and +hard to actually solve. + +--- + +## 1. The determinism spine + +This is the single architectural constraint everything else obeys. It is what makes +"realtime preview" and "4K60 offline export" the same program rather than two programs +that drift apart. + +**Mechanism: the audio is fully analyzed before the first frame renders, into a +frame-indexed table.** + +``` +track file + ↓ decode() OfflineAudioContext → AudioBuffer (a 6-min track decodes in ~1-2s) + ↓ analyze() STFT at hop = 1/60 s, plus global passes + ↓ +FeatureTrack a typed-array table, one row per output frame +``` + +Nothing in the engine ever reads a live `AnalyserNode`. Realtime preview maps +`audio.currentTime → frameIndex` and reads row *n*. The offline exporter counts +`frameIndex` directly and reads row *n*. **Identical input to the visuals in both modes**, +so what you preview is exactly what you export. + +This also unlocks the thing a causal analyser fundamentally cannot do: + +> **Lookahead.** Because the whole song is analyzed up front, a build can *anticipate* its +> drop — ramp density and warp toward the drop's target values over the eight bars before +> it, and hit the transition already at full tension. A causal analyser can never do this; +> it only knows the drop landed after it landed. This is the biggest single visual win in +> the whole plan. + +### Determinism rules (violating any of these breaks export) + +- **No wall-clock anywhere in scene or layer code.** No `performance.now()`, no `Date`, no + `clock.getElapsedTime()`. A `Timeline` object injects `{ frame, time, dt, progress }`. +- **No `Math.random()` in anything that animates.** Seeded PRNG only (mulberry32, with the + seed held per-instance rather than on a global). +- **`dt` is constant** — `1/fps` — even in realtime preview. If the browser drops a frame, + preview shows a hitch; it does not change the animation. Frame 5400 is the same image + every run. +- **Deterministic buffer initialization.** Feedback and ping-pong targets must be explicitly + cleared at start; never inherit whatever was in GPU memory. +- **Resolution independence.** Scene math lives in normalized aspect-corrected coordinates. + Anything genuinely pixel-sized takes a `u_pixelScale = resolution / referenceResolution` + uniform. This is what lets a 720p preview match a 4K export. + +### Stated limit on the guarantee + +Determinism is **per-machine**: same browser, same GPU, same driver → bit-identical frames. +Across different GPUs, floating-point and `fwidth` derivative differences make bit-exactness +unrealistic, and chasing it would be wasted effort. The cross-machine guarantee is +*perceptual*: frames must match within a small diff threshold. The validation checks in §11 +are written to that distinction, so the acceptance criteria are actually achievable. + +--- + +## 2. Audio analysis + +`src/audio/`. Pure functions over an `AudioBuffer` — no DOM, no three.js — so it's unit +testable and can move to a worker without rework. + +**Per frame** (hop 1/60 s, window 2048, Hann): + +| Feature | Used for | +|---|---| +| `rms`, `loudness` | overall intensity, quiet-passage detection | +| band energies (sub / low / mid / high / air) | per-element reactivity | +| `flux` (spectral flux, half-wave rectified) | onset strength, the beat signal | +| `centroid` | perceived brightness → color temperature | +| `flatness` | noisy vs tonal → distinguishes pads from percussion | +| `stereoWidth` | width of the visual field | + +**Global passes:** + +- **Tempo** — autocorrelation over the onset envelope, octave-normalized to 90-180 BPM. + Whole-song, so it's exact rather than converging. Yields a phase-locked **beat grid** and + downbeats, hence exact `beatPhase` / `barPhase` / `phrasePhase` per frame. +- **Segmentation** — novelty curve from a self-similarity matrix over the band-energy + vectors, peak-picked into section boundaries. +- **Section classification** — each section labelled by energy percentile, flux density and + centroid: `intro | build | drop | sustain | breakdown | outro`. Deliberately simple + heuristics over features we already have; this is the fuzziest part of the pipeline and is + not worth an ML detour. + +**Lookahead fields, per frame:** `timeToNextSection`, `nextSectionKind`, `nextSectionEnergy`, +`buildSlope`. These are what the arc driver leans on. + +--- + +## 3. Scene modules and the parameter schema + +**This is the decision that makes a large library affordable.** Adding a scene must cost a +shader and a schema block — no plumbing, no UI code, no wiring. + +```js +export const NebulaDrift = { + name: 'Nebula Drift', + family: 'organic', + kind: 'fragment', // or 'layer3d' + shader: fragmentSource, + + params: { + density: { type:'float', range:[0,1], default:0.5, uniform:'u_density' }, + warp: { type:'float', range:[0,3], default:1.0, uniform:'u_warp' }, + symmetry: { type:'int', range:[1,8], default:1, uniform:'u_symmetry' }, + grain: { type:'float', range:[0,1], default:0.2, uniform:'u_grain' }, + palette: { type:'palette', count:5, uniform:'u_colors' }, + }, + + // which audio features modulate which params, and how hard + reactive: { + density: { feature:'bandLow', amount:0.3 }, + warp: { feature:'beat', amount:0.5, response:'spike' }, + grain: { feature:'flatness', amount:0.2 }, + }, +}; +``` + +One schema, five payoffs: + +1. Uniforms bind automatically — no per-scene wiring. +2. The config UI generates its own sliders, so override-ability is free for every new scene. +3. `LookGenerator` samples the declared ranges with the track's seed → per-track variation + with no hand-tuning. +4. `ArcDriver` can animate any declared param across a section boundary generically. +5. Presets are just serialized param sets, savable per track. + +It also makes scenes **machine-checkable**, which is what keeps a thirty-scene library from +rotting — see the schema lint in §10. + +### Library targets + +Interest across songs comes from breadth of *family*, not raw count. Six families, growing +to roughly 4-6 scenes each: + +| Family | Character | Fits | +|---|---|---| +| **flow** | curl noise, fluid, drifting particles | ambient, sustain | +| **geometric** | tunnels, kaleidoscope, grids, synthwave | drops | +| **organic** | nebula, plasma, reaction-diffusion, metaballs | sustain, breakdown | +| **structural** | raymarched terrain, cities, architecture | builds | +| **minimal** | lines, waveform, spectrum sculpture, negative space | intros, quiet passages | +| **glitch** | feedback, datamosh, scanlines, chromatic tearing | drops, transitions | + +The arc driver picks a *family* from section kind, then a scene within it from the seed — so +a quiet passage never lands on a strobing glitch scene, and two tracks with similar structure +still choose differently. + +### Seeded from the five that exist + +`party-stage`'s five visualizer shaders are the library's starting point. Porting each means: +strip the LED-grid mask, replace hardcoded colors with palette lookups, pull magic numbers +out into declared params, swap live uniforms for `FeatureTrack` reads. Note their +`u_resolution` is not pixels — it's the LED grid cell count. Rename it `u_gridSize` on port +so it doesn't collide with the real resolution uniform. + +--- + +## 4. Compositor + +`src/engine/Compositor.js`. A layer stack rendered to ping-pong render targets. + +- **Layers**: fragment-shader (a fullscreen quad) or 3D (a three.js sub-scene — particles, + geometry, camera path). Both present the same `render(target, timeline, features)` face, + so the stack doesn't care which it holds. +- **Blend modes** per layer: normal, add, screen, multiply, overlay. +- **Feedback buffer** — previous frame available to any layer as a texture, with configurable + decay and warp. Cheap, and disproportionately responsible for things looking "alive". +- **Post chain**: bloom, chromatic aberration, grain, vignette, final grade. + +Typical stack: background scene → accent scene (screen blend, low opacity) → particles → +feedback → post. + +--- + +## 5. Look generation and the arc driver + +**`LookGenerator`** runs once per track, after analysis: + +``` +seed = hash(decoded PCM) → same track always renders identically +palette = f(centroid, flatness, energy distribution, seed) +sceneAssignments = per section: family from kind, scene from seed +paramSets = sampled from declared ranges, biased by track character +``` + +`PaletteSource` is an interface here, with an `AudioPalette` implementation. Adding a +`CoverArtPalette` later is a new file and one line of config — no scene changes. + +**`ArcDriver`** runs per frame: + +- Cross-fades scenes at section boundaries (compositor holds both briefly, opacity ramp). +- Ramps params toward the *next* section's targets during builds, using the lookahead fields. +- Applies slow seeded LFO drift to params within a section, so nothing sits still even + during a four-minute sustain. +- Applies the `reactive` mappings on top: beat spikes, band-energy modulation. + +Three timescales stacked — per-frame reactivity, per-section drift, whole-song arc — is what +keeps a six-minute track from reading as a loop. + +--- + +## 6. Preview + +**Non-negotiable requirement: the composition is fully reviewable before anything is +exported.** Export is slow and committing to a six-minute render to discover a bad transition +at 4:10 is unacceptable. Preview is not a debug view — it is the primary working surface, and +export is the thing you do once you're happy. + +### What it must do + +- **Same everything as export.** Identical compositor, identical look spec, identical + `FeatureTrack`. The *only* permitted differences are output resolution and warm-up state + (below). Preview must never make a different look decision than export would. +- **Transport**: play / pause / scrub, frame step, and **jump to previous/next section + boundary** — because transitions are what you actually need to review. +- **Loop a section** while tuning it. +- **Live param editing** with instant feedback, no track restart. Edits apply to the current + section's param set and are savable. +- **Timeline strip** under the viewport: sections as colored blocks by kind, beat/bar ticks, + playhead, and markers for scene changes. Segmentation problems are visible at a glance here. +- **Re-roll controls**: new track seed, re-roll just this section's scene, **lock** a section + so further re-rolls leave it alone. +- **Quality toggle**: draft (half resolution, post chain simplified) for smooth scrubbing on + heavy stacks, versus full-quality preview at export settings but reduced resolution. +- **Segment test-render**: export ~20 seconds around the playhead at full export quality. + This is the bridge between "looks right in preview" and "commit to the full render", and + it is the thing that makes a bad 4K export a rare event rather than a routine one. + +### The seek problem, honestly + +Stateful layers — feedback buffers, particle systems — mean frame *N* depends on the frames +before it. Scrubbing to an arbitrary frame therefore can't be exact for free. Three-part +resolution: + +1. **Warm-up.** On seek, render *K* frames (default ≈120, about 2s) as fast as possible + off-screen before displaying. Converges feedback state to visually correct. +2. **Section-boundary seeks are exact.** Particle systems and layer state are re-seeded + deterministically at each section boundary, so jumping to a boundary — the common review + action — needs no warm-up and is frame-exact. +3. **Export is always sequential**, so it is exact everywhere by construction. + +Documented consequence: scrubbing to a random mid-section frame shows a *converged*, not +bit-exact, image. Reviewing a transition never hits this, because transitions are boundaries. + +--- + +## 7. Export + +Same engine, `Timeline` in fixed-step mode, rendering to an offscreen target at export +resolution. + +- **Primary: WebCodecs `VideoEncoder`.** Hardware-accelerated H.264/VP9 straight from the + render target, muxed with the decoded audio. No ffmpeg.wasm payload, and fast. +- **Fallback: frame dump + ffmpeg.** A small vite dev-server middleware accepts POSTed frames + and pipes them to a local `ffmpeg`. Slower, trivially debuggable. +- Title card and fade in/out at the edges. +- Progress reporting and cancel, since a 4K render is minutes not seconds. + +--- + +## 8. Validation tooling + +Built early (most of it in Phases 0-1) because every later phase depends on it. This is the +difference between "seems fine" and "verified". + +| Tool | What it catches | +|---|---| +| **Frame hash log** | Renders *N* frames headless, hashes each (FNV over `readPixels`), writes JSON. Diffing two runs is the determinism regression test. | +| **Click track** | Mixes an audible click on the detected beat grid into the track. You can *hear* whether tempo detection is right — far more reliable than watching visuals and guessing. | +| **Section timeline overlay** | Detected sections and labels drawn over the transport. Bad segmentation is obvious immediately. | +| **Feature scope** | Oscilloscope plot of selected features with playhead. Catches dead, constant, or saturated features — e.g. a band that's always 0 because the split was wrong. | +| **Dual-resolution diff** | Renders one frame at 720p and 4K, downsamples, diffs. Catches resolution dependence, which is otherwise silent until export looks wrong. | +| **Seed contact sheet** | Same frame across 16 seeds as a grid. Makes look-space collapse visible instantly — this is the primary defense against "sameness across tracks". | +| **Schema lint** | Parses each scene's shader for `uniform` declarations and cross-checks against its `params` block, both directions. Catches typos and orphans; the thing that keeps a 30-scene library maintainable. | +| **Param range sweep** | Renders each param at several points across its declared range, asserts no NaN, no all-black, no all-white frames. | +| **Perf HUD** | Frame time, GPU time per layer, frame index, active section, live param values. | + +### The track battery + +A fixed set of **6 of your own tracks** spanning ambient → mid-tempo → hard EDM, checked in +as the standing regression corpus (paths only, not the audio). Every phase gate below is +evaluated against the same battery, so improvements and regressions are comparable across +time. Pick these before Phase 1 — the analysis work is much easier to judge against real +material you know well. + +--- + +## 9. Phases, with gates + +Each phase has an explicit gate. Don't start the next phase until the gate passes; these are +cheap to check now and expensive to retrofit later. + +### Phase 0 — skeleton and determinism spine +`Timeline`, `Renderer`, `Compositor` with a single layer, seeded RNG, fullscreen quad. One +ported shader with the LED grid stripped. Frame hash tool. + +**Gate:** +- 300 frames rendered twice on the same machine → **bit-identical hashes**. +- Artificially stalling the render loop (simulated dropped frames) changes nothing in the + output hashes — proves `dt` is truly fixed. +- Dual-resolution diff on 5 sample frames passes the perceptual threshold. +- Grep gate: zero occurrences of `Math.random`, `performance.now`, `Date.now` under + `src/scenes/` and `src/engine/` outside of explicitly allowed spots. + +### Phase 1 — offline audio pipeline +`decode`, `FeatureTrack`, tempo and beat grid. The ported scene reads features by frame index +instead of a live analyser. Click track, feature scope. + +**Gate:** +- **Click track lines up by ear on all 6 battery tracks.** If tempo is wrong anywhere, stop + and fix it here — every downstream timing artifact traces back to this. +- Feature sanity: no NaN or Inf anywhere in the table; every feature's realized range covers + a meaningful span (flag any that are constant or pinned at an extreme). +- Realtime-driven playback and fixed-step render of frames 1000-1100 → identical hashes. +- Analysis of a 6-minute track completes in under ~3s. +- Seeking to frame *N* after warm-up matches sequential playback to *N* within threshold. + +### Phase 2 — parameter schema, automatic binding, generated UI +Remaining four shaders ported with schemas. Schema lint, param range sweep. + +**Gate:** +- Schema lint clean on every scene, both directions. +- Range sweep clean: no scene produces NaN, black or white frames anywhere in its declared + ranges. +- Every declared param appears in the generated UI, edits take visible effect, and values + round-trip through save/load unchanged. +- Ported scenes reviewed side-by-side against the `party-stage` originals — confirm nothing + was lost in the port. + +### Phase 3 — look generation ("A" complete) +`LookGenerator`, `PaletteSource`, content-derived seed. Seed contact sheet. + +**Gate:** +- Same file loaded twice → identical seed → identical look spec (JSON compare). +- **Seed contact sheet shows genuine spread**: mean pairwise perceptual distance across 16 + seeds above a set threshold. This is the "sameness" gate — a collapsed look space fails here. +- All 6 battery tracks produce meaningfully different look specs (different scene selections, + distinguishable palettes). +- Palettes pass a contrast/luminance-spread check — no muddy, low-separation sets. +- Every scene survives every generated param set across the battery without crashing. + +*At this gate the tool is genuinely useful: drop a track in, get an uploadable video with no +input.* + +### Phase 4 — segmentation and arc driver +Sections, classification, cross-fades, lookahead ramps, intra-section drift. + +**Gate:** +- Hand-label section boundaries on the battery; detected boundaries score an acceptable F1 + within ±2s tolerance. Record the number — it's the baseline for future tuning. +- Transition check: frame-to-frame perceptual delta across the whole track, flagging spikes. + No black frames or pops at boundaries except where intended. +- Param traces logged over a build section show monotonic ramps into the following drop — + proves lookahead is actually wired, not just present in the data. +- **Watch all 6 battery tracks end to end.** Unavoidable and not substitutable. Budget the + time; the failure mode this catches — subtle monotony — is invisible to every automated check. + +### Phase 5 — compositing depth ("C" complete) +Multi-layer, blend modes, feedback, post chain, 3D particle layers. + +**Gate:** +- Full stack holds 60fps at preview resolution; per-layer GPU cost recorded against a budget. +- Feedback stability: 10,000-frame run neither saturates to white nor decays to black. +- Determinism still passes with feedback and particles active — this is where deterministic + buffer initialization gets tested for real. +- Every layer renders correctly in isolation when soloed. + +### Phase 6 — export +WebCodecs path, title card, fades, progress and cancel. + +**Gate:** +- **Preview/export parity**: hashes of exported frames match preview-rendered frames for the + same range. +- A/V sync measured at start, middle and end of a 6-minute export — click track against video + frames, drift within one frame. +- `ffprobe` frame count and duration exactly as expected; no dropped or duplicated frames. +- Exported audio matches the source. +- Plays correctly in VLC and in a browser, **and survives a real upload** to the video site + you actually publish on. Test this once, early, with a short file — container quirks are + much cheaper to find now than after a 4K render. + +### Phase 7 — grow the library +Toward six families. Each new scene is a shader plus a schema block. + +**Per-scene checklist** (the gate is per scene, not per phase): +- Schema lint clean; range sweep clean; determinism hashes stable. +- Within GPU budget at 4K. +- Sits correctly in its declared family — reviewed in both a quiet and a loud section. +- Library-wide regression contact sheet at a fixed seed, diffed against last known good, so + shared-code changes can't silently break existing scenes. + +--- + +## 10. Detachment from `party-stage` + +Fork, copy what's useful, then **detach completely**. This matches how the rest of the repo +works — each generator stands alone — and here it's also technically right: `party-stage` is +causal and realtime by design, this is two-pass and offline, and a shared module would serve +neither well. + +**Copied, then owned outright** (edit freely, no upstream obligation): +- the mulberry32 seeded PRNG +- `MediaStorage`, the IndexedDB track persistence +- config-UI patterns +- the postprocessing setup +- the five visualizer shaders + +**Referenced as prior art, not copied**: `music-visualizer.js`. Its three-detector onset logic +and band splits are a good specification for *what to measure*; the offline analyzer computes +all of it better, so none of the code carries over. + +**Deliberately not ported**: `SceneFeature` / `SceneFeatureManager`. `Layer` + `Compositor` is +the right shape for a layer stack, and retrofitting the old pattern would fight the design. + +**Enforced**: zero imports crossing the directory boundary, in either direction. A grep for +`party-stage` under `flow-state/src/` returns nothing. `party-stage` keeps working exactly as +it does today and is never touched by this project. + +--- + +## 11. Risks + +- **Segmentation quality** is the fuzziest component. Mitigation: energy-novelty heuristics + only, tuned by ear against the battery; a misjudged section means a scene change in a + slightly odd place, not a broken video. The Phase 4 F1 number keeps it honest. +- **Shader compile time** grows with the library. Mitigation: compile lazily, only the scenes + the look actually selected. +- **4K export speed.** Mitigation: WebCodecs, plus the fact that it's offline — slow is fine + as long as preview stays fast. +- **Resolution-independence discipline** is easy to violate silently. Mitigation: the + dual-resolution diff, run every phase gate, not just Phase 0. +- **Monotony can survive every automated check.** Only watching full tracks catches it. This + is why Phase 4's gate includes end-to-end viewing and why the battery exists — and it's the + risk most likely to be the one that actually bites.