music-video-gen/flow-state/DESIGN.md

flow-state — design decisions

The decisions this project has taken, and why. New work must obey them; changing one means updating this file and the gate that enforces it.

Doc map: PLAN.md is the full design, EPIC-2/3/4/5-3D.md are the epic designs, HOWTO-visualizers.md / HOWTO-variety.md are the scene-author references, MIGRATION.md is the shader→stage migration recipe.

1. Determinism spine

• The audio is fully analysed before the first frame renders, into a frame-indexed FeatureTrack table. Nothing reads a live AnalyserNode. Preview maps currentTime → frameIndex; export counts frames. Same rows, same image.
• No wall-clock (performance.now, Date) and no Math.random in engine/scenes/look/audio/params/actors/venues. Seeded Rng only; motion is analytic f(t, seed) — seek === playback, preview === export.
• dt is constant (1/fps); dropped preview frames hitch, never diverge.
• Feedback/ping-pong targets are explicitly cleared; shader programs are primed (Engine.prime) before frame-exact rendering — unlinked programs rendered the first ~10 frames wrong on the heaviest scene.
• Scene math lives in normalized aspect-corrected coords (uv, p); only u_pixelScale carries pixels, so 720p preview matches 4K export.
• Guarantee is per-machine bit-identical; cross-machine is perceptual. Same-machine allows max channel delta ≤ 1 (GPU load varies one LSB under heavy shaders); real bugs score in the tens.
• Seeks need warm-up (log(0.001)/log(decay) frames) whenever feedback is on. Section boundaries are NOT exact for free — the compositor's feedback buffer is global and carries across them. Clearing it per boundary would flash every transition; warm-up is the better trade.
• Previews link to their exact app frame (src/ui/preview.js: song + seed
  • frame + venue/viz pins + the probe's warm-up length, both sides primed). Exactness takes four agreements: the bank probes analyse the bank PCM (peak-normalised 16-bit mono — audio/songbank.js bankPcm), file decode is fixed at 44.1 kHz with bit-exact WAV parsing (never the context rate), pins run before any layer builds (rebuilding mid-context renders differently), and materials built mid-update compile synchronously (needsCompile) so no warm-up frame draws against an unlinked program.

2. Lookahead

Offline analysis is what buys anticipation: buildSlope ramps through the bars before a higher-energy section, so visuals arrive at the drop already at tension. A causal analyser can never do this. The outgoing layer of a crossfade keeps its pre-boundary slope — feeding it the new section's features collapses its ramp in one frame (a visible pop at the edit).

3. Scenes: shader + schema, nothing else

• A scene is a fragment shader (or model build/update) plus a params block. Uniform binding, UI sliders, seeded sampling, arc drift and phase gates are all derived from the schema. No per-scene wiring.
• Six families (flow geometric organic structural minimal glitch) chosen by section kind — a breakdown never lands on a strobing glitch scene.
• Personality traits (shape camera space style) are a contract: each track is built on 1–2 traits and a scene honouring fewer is never cast. The library intentionally shrinks per track. Declared traits need machine-checked evidence (lint grep + render-delta gate); a dead camera (sigCamera assigned, result ignored) shipped once and is now linted.
• rate: true is mandatory for anything multiplying u_time. Phase is elapsed × rate; modulating a rate jumps phase by elapsed × Δrate — measured as 6 flashes/s, twice the WCAG 2.3.1 ceiling.
• Whole-frame luminance never pulses on the beat (3-flashes/s ceiling). Pulse something local; quantise glitches to the bar grid.
• A scene reading prev() must paint a base image first (black first frames, seek-fragile otherwise). prev() readers can never be grounds.
• No contract-uniform collisions, no hardcoded saturated colours — the look generator recolours through pal()/palRamp() only.
• Loops need constant bounds with an early break on the deciding param (// lint: fixed-cost opt-out for genuinely fixed work).

4. Continuity: nothing disappears mid-shot

Actors and objects change only at camera cuts:

• Discrete params (int/bool, plus floats marked discrete: true that drive loop bounds / instance counts) hold constant for the whole cue. No drift, no lookahead ramp, no reactive modulation — those would pop members in/out mid-shot. They still vary per section/variant sample, so drops can be denser than intros; the change lands on a cut.
• A discrete param MAY declare slowAxis: the driver steps its travel on the section-held journey instead of sliding it.
• Reactive on a discrete param and (unstated) drift of one are lint errors.
• Venue furniture never hard-toggles on interpolated state: chamber junk and stage pylons fade (ride-crew pattern); ride roadside density is arrangement-only. Re-hangs happen only on story jumps, always under a camera cut (venueJumpAt); the venue layer itself is persistent and never dissolved (old-room/new-room ghosting read as a glitch).

5. Time structure: cues, cuts, camera

• The arc driver works in cues (one per shot): a shot cut and a section change take the same code path, differing only in transition length.
• Default transition is a slow dissolve (2 bars calm, 1 bar loud, capped at 40% of the shot). Hard cuts only above the energy threshold — on calm material a cut reads as a glitch. Even hard cuts are 2-frame ramps (a 1-frame swap of bright scenes is a flash).
• Framing lives at the shot level (size constant per shot, gaze live); the outgoing shot keeps travelling as it fades — freezing it stops the old image dead before it disappears.
• Sharp transitions cut colour instantly with the image and always reframe (a bigger reframe than a dissolve earns).
• Story variables are staged, not ramped: flat inside a section, blended ~1 bar after boundaries. Moments (arrival/turn/climax/resolution) are found in the audio; the story never overrides the song (bounded bias terms). All scenes share one axis sign so the video travels one way.

6. Colour and variety

• Palettes in OKLCH; hue from spectral tilt (log treble/body ratio — centroid and plain body-fraction both collapsed across masters); both ways round the wheel; directors choose 1–4 palettes, blended in OKLCH across dissolves.
• Per-track temperament (intensity/pace/detail/extremity) moves every param dial so one scene cast in two videos isn't the same footage twice.
• Overlays on 5–55% of stacks, never normal blend, never above 0.6 opacity — occasionally layered, never permanently cluttered.
• A scene varies when the SONG changes what's on screen: draw castMain/castChorus (form in 3D), size via stageScale(), place via stageNode(). No effect-knobs (glow/grain/chroma belong to post).
• Grain is a per-track treatment (mode/cell/rate/mask/chroma, ~45% of tracks get none); scene texture: 0..2 scales only its share.
• Absolute animation speed follows the song: motion bias is tempo-dominated and every rate: true param scales by per-track rateScale — slow songs got fast scenes before this.

7. Grounds and composition

• surface is measured (metadata.json), never declared — nine scenes claimed canvas while painting under a third of the frame.
• A ground fills the frame on average and never vanishes under any identity (worst case counts, not the mean); hollow grounds are flattened via groundPersonalityFrom.
• Model layers share one Compositor perspective/depth rig; shader depth stays faked. Venue rooms are furnished with the track's monolith body so the room reads as this track's place.
• With a room present the viz plays ON its screens (window/wall/radio), never composited over it; model viz additionally stand in the shared world for shared-rig rooms. A directCamera room (ride) owns only its own rig — the shared rig keeps the subject grammar, or the viz is filmed through a room camera meant for another world and vanishes.

8. 3D on the same terms

• ActorSpec is data, not scene graph (serialisable, hashable); geometry via actorToGeometry from the same Identity.form the shaders march.
• Model motion is analytic, instanced, seeded; kit deformation reuses the song's silhouette params; castSolid impostors remain the fallback.
• New actors are a file plus a registry entry, gated like scenes.

9. Export and review

• WebCodecs primary, frame-dump+ffmpeg fallback; title card, edge fades, progress/cancel; A/V sync within one frame; survives a real upload test.
• Preview is the primary surface: transport, section jump/loop, live param edit, timeline strip, re-roll + locks, draft/full quality, 20s segment test-renders. Never commit to a full render blind.
• Every phase gate is measured; the battery (6 tracks ambient→hard EDM) is watched end to end — monotony is invisible to automation.

10. History

• Forked from party-stage, then detached: zero cross-imports.
• Corrections kept: boundary seeks need warm-up (not exact for free); layout is blind on full-frame fields; aggregate ratios (e.g. separation) swing run-to-run — steer by direct render comparisons.