music-video-gen/flow-state/PLAN.md

# flow-state — plan

An ambient/EDM music video generator. Drop in a track, get a full-length, non-story,
music-reactive video that evolves through the song. No sourced footage: every frame is
generated, and everything the look depends on is derived from the audio itself.

Name is a placeholder, easy to rename before Phase 0 lands.

## Decisions taken

| Question | Decision |
|---|---|
| Render model | Deterministic core. Realtime preview **and** offline frame-exact export, from the same code path. |
| Preview | **Required.** Full transport, section jumping, live param editing, and segment test-renders — see §8. |
| Art direction | Automatic from the audio, every derived parameter overridable and savable per track. |
| Scene architecture | Hybrid compositor. Layers are usually fullscreen fragment shaders; three.js particle/geometry layers are also valid layers. |
| Cover art | Not available. Palette is audio-derived. A `PaletteSource` seam is left so artwork can be added later without touching scenes. |
| Relationship to `party-stage` | Fork and **detach**. Copy what's useful, then zero cross-imports — see §12. |

## The two problems this has to solve

They are separate and both are load-bearing:

1. **Sameness across tracks.** Track #12 must not look like track #11. Solved by deriving
   the look from the track's own measured character plus a content-derived seed.
2. **Monotony within a track.** A single shader gets dull around minute three. Solved by
   segmenting the song and driving scene changes, palette shifts and parameter motion off
   that structure.

Both have explicit validation gates in §11 — they are easy to *believe* you've solved and
hard to actually solve.

---

## 1. The determinism spine

This is the single architectural constraint everything else obeys. It is what makes
"realtime preview" and "4K60 offline export" the same program rather than two programs
that drift apart.

**Mechanism: the audio is fully analyzed before the first frame renders, into a
frame-indexed table.**

```
track file
   ↓ decode()            OfflineAudioContext → AudioBuffer (a 6-min track decodes in ~1-2s)
   ↓ analyze()           STFT at hop = 1/60 s, plus global passes
   ↓
FeatureTrack             a typed-array table, one row per output frame
```

Nothing in the engine ever reads a live `AnalyserNode`. Realtime preview maps
`audio.currentTime → frameIndex` and reads row *n*. The offline exporter counts
`frameIndex` directly and reads row *n*. **Identical input to the visuals in both modes**,
so what you preview is exactly what you export.

This also unlocks the thing a causal analyser fundamentally cannot do:

> **Lookahead.** Because the whole song is analyzed up front, a build can *anticipate* its
> drop — ramp density and warp toward the drop's target values over the eight bars before
> it, and hit the transition already at full tension. A causal analyser can never do this;
> it only knows the drop landed after it landed. This is the biggest single visual win in
> the whole plan.

### Determinism rules (violating any of these breaks export)

- **No wall-clock anywhere in scene or layer code.** No `performance.now()`, no `Date`, no
  `clock.getElapsedTime()`. A `Timeline` object injects `{ frame, time, dt, progress }`.
- **No `Math.random()` in anything that animates.** Seeded PRNG only (mulberry32, with the
  seed held per-instance rather than on a global).
- **`dt` is constant** — `1/fps` — even in realtime preview. If the browser drops a frame,
  preview shows a hitch; it does not change the animation. Frame 5400 is the same image
  every run.
- **Deterministic buffer initialization.** Feedback and ping-pong targets must be explicitly
  cleared at start; never inherit whatever was in GPU memory.
- **Resolution independence.** Scene math lives in normalized aspect-corrected coordinates.
  Anything genuinely pixel-sized takes a `u_pixelScale = resolution / referenceResolution`
  uniform. This is what lets a 720p preview match a 4K export.

### Stated limit on the guarantee

Determinism is **per-machine**: same browser, same GPU, same driver → bit-identical frames.
Across different GPUs, floating-point and `fwidth` derivative differences make bit-exactness
unrealistic, and chasing it would be wasted effort. The cross-machine guarantee is
*perceptual*: frames must match within a small diff threshold. The validation checks in §11
are written to that distinction, so the acceptance criteria are actually achievable.

---

## 2. Audio analysis

`src/audio/`. Pure functions over an `AudioBuffer` — no DOM, no three.js — so it's unit
testable and can move to a worker without rework.

**Per frame** (hop 1/60 s, window 2048, Hann):

| Feature | Used for |
|---|---|
| `rms`, `loudness` | overall intensity, quiet-passage detection |
| band energies (sub / low / mid / high / air) | per-element reactivity |
| `flux` (spectral flux, half-wave rectified) | onset strength, the beat signal |
| `centroid` | perceived brightness → color temperature |
| `flatness` | noisy vs tonal → distinguishes pads from percussion |
| `stereoWidth` | width of the visual field |

**Global passes:**

- **Tempo** — autocorrelation over the onset envelope, octave-normalized to 90-180 BPM.
  Whole-song, so it's exact rather than converging. Yields a phase-locked **beat grid** and
  downbeats, hence exact `beatPhase` / `barPhase` / `phrasePhase` per frame.
- **Segmentation** — novelty curve from a self-similarity matrix over the band-energy
  vectors, peak-picked into section boundaries.
- **Section classification** — each section labelled by energy percentile, flux density and
  centroid: `intro | build | drop | sustain | breakdown | outro`. Deliberately simple
  heuristics over features we already have; this is the fuzziest part of the pipeline and is
  not worth an ML detour.

**Lookahead fields, per frame:** `timeToNextSection`, `nextSectionKind`, `nextSectionEnergy`,
`buildSlope`. These are what the arc driver leans on.

---

## 3. Scene modules and the parameter schema

**This is the decision that makes a large library affordable.** Adding a scene must cost a
shader and a schema block — no plumbing, no UI code, no wiring.

```js
export const NebulaDrift = {
  name: 'Nebula Drift',
  family: 'organic',
  kind: 'fragment',            // or 'layer3d'
  shader: fragmentSource,

  params: {
    density:  { type:'float',   range:[0,1],  default:0.5, uniform:'u_density' },
    warp:     { type:'float',   range:[0,3],  default:1.0, uniform:'u_warp' },
    symmetry: { type:'int',     range:[1,8],  default:1,   uniform:'u_symmetry' },
    grain:    { type:'float',   range:[0,1],  default:0.2, uniform:'u_grain' },
    palette:  { type:'palette', count:5,                   uniform:'u_colors' },
  },

  // which audio features modulate which params, and how hard
  reactive: {
    density: { feature:'bandLow',  amount:0.3 },
    warp:    { feature:'beat',     amount:0.5, response:'spike' },
    grain:   { feature:'flatness', amount:0.2 },
  },
};
```

One schema, five payoffs:

1. Uniforms bind automatically — no per-scene wiring.
2. The config UI generates its own sliders, so override-ability is free for every new scene.
3. `LookGenerator` samples the declared ranges with the track's seed → per-track variation
   with no hand-tuning.
4. `ArcDriver` can animate any declared param across a section boundary generically.
5. Presets are just serialized param sets, savable per track.

It also makes scenes **machine-checkable**, which is what keeps a thirty-scene library from
rotting — see the schema lint in §10.

### Library targets

Interest across songs comes from breadth of *family*, not raw count. Six families, growing
to roughly 4-6 scenes each:

| Family | Character | Fits |
|---|---|---|
| **flow** | curl noise, fluid, drifting particles | ambient, sustain |
| **geometric** | tunnels, kaleidoscope, grids, synthwave | drops |
| **organic** | nebula, plasma, reaction-diffusion, metaballs | sustain, breakdown |
| **structural** | raymarched terrain, cities, architecture | builds |
| **minimal** | lines, waveform, spectrum sculpture, negative space | intros, quiet passages |
| **glitch** | feedback, datamosh, scanlines, chromatic tearing | drops, transitions |

The arc driver picks a *family* from section kind, then a scene within it from the seed — so
a quiet passage never lands on a strobing glitch scene, and two tracks with similar structure
still choose differently.

### Seeded from the five that exist

`party-stage`'s five visualizer shaders are the library's starting point. Porting each means:
strip the LED-grid mask, replace hardcoded colors with palette lookups, pull magic numbers
out into declared params, swap live uniforms for `FeatureTrack` reads. Note their
`u_resolution` is not pixels — it's the LED grid cell count. Rename it `u_gridSize` on port
so it doesn't collide with the real resolution uniform.

---

## 4. Compositor

`src/engine/Compositor.js`. A layer stack rendered to ping-pong render targets.

- **Layers**: fragment-shader (a fullscreen quad) or 3D (a three.js sub-scene — particles,
  geometry, camera path). Both present the same `render(target, timeline, features)` face,
  so the stack doesn't care which it holds.
- **Blend modes** per layer: normal, add, screen, multiply, overlay.
- **Feedback buffer** — previous frame available to any layer as a texture, with configurable
  decay and warp. Cheap, and disproportionately responsible for things looking "alive".
- **Post chain**: bloom, chromatic aberration, grain, vignette, final grade.

Typical stack: background scene → accent scene (screen blend, low opacity) → particles →
feedback → post.

---

## 5. Look generation and the arc driver

**`LookGenerator`** runs once per track, after analysis:

```
seed = hash(decoded PCM)          → same track always renders identically
palette = f(centroid, flatness, energy distribution, seed)
sceneAssignments = per section: family from kind, scene from seed
paramSets = sampled from declared ranges, biased by track character
```

`PaletteSource` is an interface here, with an `AudioPalette` implementation. Adding a
`CoverArtPalette` later is a new file and one line of config — no scene changes.

**`ArcDriver`** runs per frame:

- Cross-fades scenes at section boundaries (compositor holds both briefly, opacity ramp).
- Ramps params toward the *next* section's targets during builds, using the lookahead fields.
- Applies slow seeded LFO drift to params within a section, so nothing sits still even
  during a four-minute sustain.
- Applies the `reactive` mappings on top: beat spikes, band-energy modulation.

Three timescales stacked — per-frame reactivity, per-section drift, whole-song arc — is what
keeps a six-minute track from reading as a loop.

---

## 6. Preview

**Non-negotiable requirement: the composition is fully reviewable before anything is
exported.** Export is slow and committing to a six-minute render to discover a bad transition
at 4:10 is unacceptable. Preview is not a debug view — it is the primary working surface, and
export is the thing you do once you're happy.

### What it must do

- **Same everything as export.** Identical compositor, identical look spec, identical
  `FeatureTrack`. The *only* permitted differences are output resolution and warm-up state
  (below). Preview must never make a different look decision than export would.
- **Transport**: play / pause / scrub, frame step, and **jump to previous/next section
  boundary** — because transitions are what you actually need to review.
- **Loop a section** while tuning it.
- **Live param editing** with instant feedback, no track restart. Edits apply to the current
  section's param set and are savable.
- **Timeline strip** under the viewport: sections as colored blocks by kind, beat/bar ticks,
  playhead, and markers for scene changes. Segmentation problems are visible at a glance here.
- **Re-roll controls**: new track seed, re-roll just this section's scene, **lock** a section
  so further re-rolls leave it alone.
- **Quality toggle**: draft (half resolution, post chain simplified) for smooth scrubbing on
  heavy stacks, versus full-quality preview at export settings but reduced resolution.
- **Segment test-render**: export ~20 seconds around the playhead at full export quality.
  This is the bridge between "looks right in preview" and "commit to the full render", and
  it is the thing that makes a bad 4K export a rare event rather than a routine one.

### The seek problem, honestly

Stateful layers — feedback buffers, particle systems — mean frame *N* depends on the frames
before it. Scrubbing to an arbitrary frame therefore can't be exact for free. Three-part
resolution:

1. **Warm-up.** On seek, render *K* frames (default ≈120, about 2s) as fast as possible
   off-screen before displaying. Converges feedback state to visually correct.
2. **Section-boundary seeks are exact.** Particle systems and layer state are re-seeded
   deterministically at each section boundary, so jumping to a boundary — the common review
   action — needs no warm-up and is frame-exact.
3. **Export is always sequential**, so it is exact everywhere by construction.

Documented consequence: scrubbing to a random mid-section frame shows a *converged*, not
bit-exact, image. Reviewing a transition never hits this, because transitions are boundaries.

---

## 7. Export

Same engine, `Timeline` in fixed-step mode, rendering to an offscreen target at export
resolution.

- **Primary: WebCodecs `VideoEncoder`.** Hardware-accelerated H.264/VP9 straight from the
  render target, muxed with the decoded audio. No ffmpeg.wasm payload, and fast.
- **Fallback: frame dump + ffmpeg.** A small vite dev-server middleware accepts POSTed frames
  and pipes them to a local `ffmpeg`. Slower, trivially debuggable.
- Title card and fade in/out at the edges.
- Progress reporting and cancel, since a 4K render is minutes not seconds.

---

## 8. Validation tooling

Built early (most of it in Phases 0-1) because every later phase depends on it. This is the
difference between "seems fine" and "verified".

| Tool | What it catches |
|---|---|
| **Frame hash log** | Renders *N* frames headless, hashes each (FNV over `readPixels`), writes JSON. Diffing two runs is the determinism regression test. |
| **Click track** | Mixes an audible click on the detected beat grid into the track. You can *hear* whether tempo detection is right — far more reliable than watching visuals and guessing. |
| **Section timeline overlay** | Detected sections and labels drawn over the transport. Bad segmentation is obvious immediately. |
| **Feature scope** | Oscilloscope plot of selected features with playhead. Catches dead, constant, or saturated features — e.g. a band that's always 0 because the split was wrong. |
| **Dual-resolution diff** | Renders one frame at 720p and 4K, downsamples, diffs. Catches resolution dependence, which is otherwise silent until export looks wrong. |
| **Seed contact sheet** | Same frame across 16 seeds as a grid. Makes look-space collapse visible instantly — this is the primary defense against "sameness across tracks". |
| **Schema lint** | Parses each scene's shader for `uniform` declarations and cross-checks against its `params` block, both directions. Catches typos and orphans; the thing that keeps a 30-scene library maintainable. |
| **Param range sweep** | Renders each param at several points across its declared range, asserts no NaN, no all-black, no all-white frames. |
| **Perf HUD** | Frame time, GPU time per layer, frame index, active section, live param values. |

### The track battery

A fixed set of **6 of your own tracks** spanning ambient → mid-tempo → hard EDM, checked in
as the standing regression corpus (paths only, not the audio). Every phase gate below is
evaluated against the same battery, so improvements and regressions are comparable across
time. Pick these before Phase 1 — the analysis work is much easier to judge against real
material you know well.

---

## 9. Phases, with gates

Each phase has an explicit gate. Don't start the next phase until the gate passes; these are
cheap to check now and expensive to retrofit later.

### Phase 0 — skeleton and determinism spine
`Timeline`, `Renderer`, `Compositor` with a single layer, seeded RNG, fullscreen quad. One
ported shader with the LED grid stripped. Frame hash tool.

**Gate:**
- 300 frames rendered twice on the same machine → **bit-identical hashes**.
- Artificially stalling the render loop (simulated dropped frames) changes nothing in the
  output hashes — proves `dt` is truly fixed.
- Dual-resolution diff on 5 sample frames passes the perceptual threshold.
- Grep gate: zero occurrences of `Math.random`, `performance.now`, `Date.now` under
  `src/scenes/` and `src/engine/` outside of explicitly allowed spots.

### Phase 1 — offline audio pipeline
`decode`, `FeatureTrack`, tempo and beat grid. The ported scene reads features by frame index
instead of a live analyser. Click track, feature scope.

**Gate:**
- **Click track lines up by ear on all 6 battery tracks.** If tempo is wrong anywhere, stop
  and fix it here — every downstream timing artifact traces back to this.
- Feature sanity: no NaN or Inf anywhere in the table; every feature's realized range covers
  a meaningful span (flag any that are constant or pinned at an extreme).
- Realtime-driven playback and fixed-step render of frames 1000-1100 → identical hashes.
- Analysis of a 6-minute track completes in under ~3s.
- Seeking to frame *N* after warm-up matches sequential playback to *N* within threshold.

### Phase 2 — parameter schema, automatic binding, generated UI
Remaining four shaders ported with schemas. Schema lint, param range sweep.

**Gate:**
- Schema lint clean on every scene, both directions.
- Range sweep clean: no scene produces NaN, black or white frames anywhere in its declared
  ranges.
- Every declared param appears in the generated UI, edits take visible effect, and values
  round-trip through save/load unchanged.
- Ported scenes reviewed side-by-side against the `party-stage` originals — confirm nothing
  was lost in the port.

### Phase 3 — look generation ("A" complete)
`LookGenerator`, `PaletteSource`, content-derived seed. Seed contact sheet.

**Gate:**
- Same file loaded twice → identical seed → identical look spec (JSON compare).
- **Seed contact sheet shows genuine spread**: mean pairwise perceptual distance across 16
  seeds above a set threshold. This is the "sameness" gate — a collapsed look space fails here.
- All 6 battery tracks produce meaningfully different look specs (different scene selections,
  distinguishable palettes).
- Palettes pass a contrast/luminance-spread check — no muddy, low-separation sets.
- Every scene survives every generated param set across the battery without crashing.

*At this gate the tool is genuinely useful: drop a track in, get an uploadable video with no
input.*

### Phase 4 — segmentation and arc driver
Sections, classification, cross-fades, lookahead ramps, intra-section drift.

**Gate:**
- Hand-label section boundaries on the battery; detected boundaries score an acceptable F1
  within ±2s tolerance. Record the number — it's the baseline for future tuning.
- Transition check: frame-to-frame perceptual delta across the whole track, flagging spikes.
  No black frames or pops at boundaries except where intended.
- Param traces logged over a build section show monotonic ramps into the following drop —
  proves lookahead is actually wired, not just present in the data.
- **Watch all 6 battery tracks end to end.** Unavoidable and not substitutable. Budget the
  time; the failure mode this catches — subtle monotony — is invisible to every automated check.

### Phase 5 — compositing depth ("C" complete)
Multi-layer, blend modes, feedback, post chain, 3D particle layers.

**Gate:**
- Full stack holds 60fps at preview resolution; per-layer GPU cost recorded against a budget.
- Feedback stability: 10,000-frame run neither saturates to white nor decays to black.
- Determinism still passes with feedback and particles active — this is where deterministic
  buffer initialization gets tested for real.
- Every layer renders correctly in isolation when soloed.

### Phase 6 — export
WebCodecs path, title card, fades, progress and cancel.

**Gate:**
- **Preview/export parity**: hashes of exported frames match preview-rendered frames for the
  same range.
- A/V sync measured at start, middle and end of a 6-minute export — click track against video
  frames, drift within one frame.
- `ffprobe` frame count and duration exactly as expected; no dropped or duplicated frames.
- Exported audio matches the source.
- Plays correctly in VLC and in a browser, **and survives a real upload** to the video site
  you actually publish on. Test this once, early, with a short file — container quirks are
  much cheaper to find now than after a 4K render.

### Phase 7 — grow the library
Toward six families. Each new scene is a shader plus a schema block.

**Per-scene checklist** (the gate is per scene, not per phase):
- Schema lint clean; range sweep clean; determinism hashes stable.
- Within GPU budget at 4K.
- Sits correctly in its declared family — reviewed in both a quiet and a loud section.
- Library-wide regression contact sheet at a fixed seed, diffed against last known good, so
  shared-code changes can't silently break existing scenes.

---

## 10. Detachment from `party-stage`

Fork, copy what's useful, then **detach completely**. This matches how the rest of the repo
works — each generator stands alone — and here it's also technically right: `party-stage` is
causal and realtime by design, this is two-pass and offline, and a shared module would serve
neither well.

**Copied, then owned outright** (edit freely, no upstream obligation):
- the mulberry32 seeded PRNG
- `MediaStorage`, the IndexedDB track persistence
- config-UI patterns
- the postprocessing setup
- the five visualizer shaders

**Referenced as prior art, not copied**: `music-visualizer.js`. Its three-detector onset logic
and band splits are a good specification for *what to measure*; the offline analyzer computes
all of it better, so none of the code carries over.

**Deliberately not ported**: `SceneFeature` / `SceneFeatureManager`. `Layer` + `Compositor` is
the right shape for a layer stack, and retrofitting the old pattern would fight the design.

**Enforced**: zero imports crossing the directory boundary, in either direction. A grep for
`party-stage` under `flow-state/src/` returns nothing. `party-stage` keeps working exactly as
it does today and is never touched by this project.

---

## 11. Risks

- **Segmentation quality** is the fuzziest component. Mitigation: energy-novelty heuristics
  only, tuned by ear against the battery; a misjudged section means a scene change in a
  slightly odd place, not a broken video. The Phase 4 F1 number keeps it honest.
- **Shader compile time** grows with the library. Mitigation: compile lazily, only the scenes
  the look actually selected.
- **4K export speed.** Mitigation: WebCodecs, plus the fact that it's offline — slow is fine
  as long as preview stays fast.
- **Resolution-independence discipline** is easy to violate silently. Mitigation: the
  dual-resolution diff, run every phase gate, not just Phase 0.
- **Monotony can survive every automated check.** Only watching full tracks catches it. This
  is why Phase 4's gate includes end-to-end viewing and why the battery exists — and it's the
  risk most likely to be the one that actually bites.