dejvino/music-video-gen
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Phases 8 and 9: shots, and a production design per track

Browse Source 
Both phases come out of the manual gate — watching whole tracks — and both
fix something no automated check was looking for.

Phase 8: shots. A section is a STAGE of the song and can run ninety
seconds; one scene held that long reads as a still image with a wobble on
it. Each section kind now gets a roster of three or four stage visuals
instead of one scene, and each section is cut into shots that rotate
between them on phrase lines, never holding past 22s. The roster stays per
kind, so a track's drops still cut between the same images and the video
keeps its identity; the anchor opens each section and the rotation returns
to it, and when a companion is due it is the least recently shown one.

The arc driver stopped working in sections and started working in cues, one
per shot, so a shot cut and a section change take the same code path and
differ only in transition length. The default transition is a slow
dissolve — two bars calm, one loud; a straight cut is reserved for
sections above the energy threshold, because on calm material a cut reads
as a glitch rather than as an edit.

Phase 9: production design. With cuts every fifteen seconds the next
problem was that the images being cut between shared nothing but the
palette. What a music video actually shares across shots is a location, a
cast, a camera operator and an art direction, so each track now generates a
personality in four traits (shape, camera, space, style) off the look seed.
The traits reach shaders as uniforms plus four helpers in the contract, and
each scene expresses them its own way: Classic Wave's rings take the
signature polygon, Metaballs merge as one, Floating Geometry no longer
picks between a box and a circle because the production already decided.

The part that makes it a design rather than a filter: scenes DECLARE which
traits they honour, a track is built on one or two, and a scene that does
not honour all of them is not cast in that track. The library shrinks per
track on purpose.

Two gates keep the declaration honest — lint greps each shader for evidence
of every trait it claims, and a render check measures that each declared
trait actually moves the image (41 scene/trait pairs, weakest response 64
of 255). A layer with no personality renders bit-identically to before,
which is what keeps every earlier sweep and regression valid.

Checks changed rather than added:
  - P4 scene-change and drift checks now measure per shot, not per section;
    the crossfade check reads its length off the cue.
  - P5 flash sweep runs per shot, so the visuals that only appear
    mid-section are measured too.
  - P6 preview/export parity primes first (as both real paths do) and
    compares at the one-LSB tolerance Phase 7 already uses. Measured over
    four consecutive shows: 3 frames at delta 1, then bit-exact — GPU
    variance on first render, not a divergence.
  - P2's contract-uniform list is derived from the contract instead of
    retyped, so the signature uniforms cannot fall out of sync.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Dejvino
2026-08-05 20:04:33 +02:00
co-authored by Claude Opus 5
parent 9d15c3cf49
commit ca5a68eb84
39 changed files with 1795 additions and 209 deletions
Download Patch File Download Diff File Expand all files Collapse all files
flow-state/src/look/ArcDriver.js
+184 -67
View File
@@ -5,16 +5,23 @@ import { clampValue } from '../params/schema.js';
/**
* Drives the look across the song.
*
* Three timescales are stacked here, and it takes all three to keep six minutes
* Four timescales are stacked here, and it takes all four to keep six minutes
* from reading as a loop:
*
* per frame — reactive mappings (handled in Layer, from the feature row)
* per shot — cuts between the section's stage visuals, on phrase lines
* per section — seeded LFO drift, so nothing sits still during a long sustain
* whole song — scene changes at real boundaries, plus lookahead ramps that
* build INTO a drop rather than reacting after it lands
*
* Layer instances are created once per section and reused. Rebuilding them per
* frame would recompile shaders and is the obvious way to make this unusably slow.
* The shot level is what stops a ninety-second sustain from being one held
* image. Everything below works in CUES — a flat list of (section, shot) spans
* built from the look, so a shot cut and a section change take exactly the same
* code path and differ only in how long the transition is. See look/shots.js.
*
* Layer instances are created once per (section, variant) and reused across
* every shot that shows that variant. Rebuilding them per shot would recompile
* shaders at every cut, which is the obvious way to make this unusably slow.
*/
export class ArcDriver {
constructor(look, track, { crossfadeBars = 1, driftAmount = 0.09 } = {}) {
@@ -24,11 +31,13 @@ export class ArcDriver {
const barSeconds = (track.tempo.period * track.tempo.beatsPerBar) / track.fps;
this.crossfadeFrames = Math.max(12, Math.round(barSeconds * crossfadeBars * track.fps));
this.barFrames = Math.max(1, Math.round(barSeconds * track.fps));
this.layerCache = new Map();
this.driftPlans = new Map();
this.activeLayers = [];
this.state = { sectionIndex: 0, crossfade: 0, incoming: null };
this.cues = this._buildCues();
this.state = { sectionIndex: 0, shotIndex: 0, crossfade: 0, incoming: null };
}
dispose() {
@@ -36,12 +45,85 @@ export class ArcDriver {
this.layerCache.clear();
}
/** One Layer per (section, layer) slot, built lazily and kept. */
_layerFor(sectionIndex, slot = 0) {
const key = `${sectionIndex}:${slot}`;
/**
* Flatten the look into cues: one per shot, in playback order.
*
* A look generated before shots existed (or hand-built by a check) has no
* `shots` array; it degrades to exactly one cue per section, which is the
* old behaviour.
*/
_buildCues() {
const cues = [];
this.look.sections.forEach((section, sectionIndex) => {
const shots = (section.shots && section.shots.length) ? section.shots : [{
startFrame: section.startFrame, endFrame: section.endFrame,
variant: 0, hardCut: false,
}];
const energy = (section.bias && section.bias.energy) || 0;
shots.forEach((shot, shotIndex) => {
const atSectionStart = shotIndex === 0;
const span = shot.endFrame - shot.startFrame;
cues.push({
index: cues.length,
sectionIndex,
shotIndex,
variant: shot.variant || 0,
startFrame: shot.startFrame,
endFrame: shot.endFrame,
atSectionStart,
fadeFrames: shot.hardCut && !atSectionStart
? Math.max(2, Math.round(this.track.fps * 0.06))
: this._dissolveFrames(energy, span),
});
});
});
return cues;
}
/**
* How long a dissolve takes: the default transition, and deliberately slow.
*
* Two bars on calm material, one on loud — a long dissolve between two
* quiet scenes reads as the image evolving, while the same length under a
* drop reads as mush, because both images are moving too fast to overlay.
* Capped at 40% of the incoming shot so a transition never occupies most of
* the shot it is transitioning into.
*/
_dissolveFrames(energy, spanFrames) {
const bars = energy > 0.6 ? 1 : 2;
const wanted = Math.max(this.crossfadeFrames, this.barFrames * bars);
return Math.max(12, Math.round(Math.min(wanted, spanFrames * 0.4)));
}
/** Cue covering a frame. Binary search — a seek can land anywhere. */
_cueIndexAt(frame) {
const cues = this.cues;
let lo = 0;
let hi = cues.length - 1;
while (lo < hi) {
const mid = (lo + hi + 1) >> 1;
if (cues[mid].startFrame <= frame) lo = mid; else hi = mid - 1;
}
return lo;
}
/** The cue on screen at a frame. For the UI and the checks. */
cueAt(frame) {
return this.cues[this._cueIndexAt(frame)];
}
_specFor(sectionIndex, variant, slot) {
const section = this.look.sections[sectionIndex];
const stack = (section.variants && section.variants[variant]) || section.layers;
return stack[slot] || null;
}
/** One Layer per (section, variant, layer slot), built lazily and kept. */
_layerFor(sectionIndex, variant, slot = 0) {
const key = `${sectionIndex}:${variant}:${slot}`;
let layer = this.layerCache.get(key);
if (!layer) {
const spec = this.look.sections[sectionIndex].layers[slot];
const spec = this._specFor(sectionIndex, variant, slot);
layer = createLayer(spec.module, {
params: spec.params,
seed: spec.seed,
@@ -49,6 +131,7 @@ export class ArcDriver {
blend: spec.blend,
});
layer.setPalette(this.look.palette);
layer.setPersonality(this.look.personality);
this.layerCache.set(key, layer);
}
return layer;
@@ -58,12 +141,12 @@ export class ArcDriver {
* Per-param LFO plan for a section: amplitude, period and phase, all seeded.
* Slow enough to read as evolution rather than wobble — 20 to 70 seconds.
*/
_driftPlan(sectionIndex, slot = 0) {
const key = `${sectionIndex}:${slot}`;
_driftPlan(sectionIndex, variant, slot = 0) {
const key = `${sectionIndex}:${variant}:${slot}`;
let plan = this.driftPlans.get(key);
if (plan) return plan;
const spec = this.look.sections[sectionIndex].layers[slot];
const spec = this._specFor(sectionIndex, variant, slot);
const rng = new Rng(spec.seed ^ 0x5bf03635);
plan = [];
for (const [name, def] of Object.entries(spec.module.params || {})) {
@@ -86,11 +169,11 @@ export class ArcDriver {
* Base params for a section at a given time: the look's sampled values, plus
* drift, plus the lookahead ramp toward whatever comes next.
*/
_paramsAt(sectionIndex, slot, time, features) {
const spec = this.look.sections[sectionIndex].layers[slot];
_paramsAt(cue, slot, time, features) {
const spec = this._specFor(cue.sectionIndex, cue.variant, slot);
const out = { ...spec.params };
for (const item of this._driftPlan(sectionIndex, slot)) {
for (const item of this._driftPlan(cue.sectionIndex, cue.variant, slot)) {
const base = out[item.name];
if (typeof base !== 'number') continue;
const wave = Math.sin(2 * Math.PI * (time / item.period + item.phase));
@@ -103,10 +186,13 @@ export class ArcDriver {
// already at tension instead of catching up afterwards.
const slope = features ? features.buildSlope || 0 : 0;
if (slope > 0.001) {
const next = this.look.sections[sectionIndex + 1];
if (next && next.layers[slot] && next.layers[slot].module === spec.module) {
const nextCue = this.cues[cue.index + 1];
const next = nextCue
? this._specFor(nextCue.sectionIndex, nextCue.variant, slot)
: null;
if (next && next.module === spec.module) {
// Same scene either side: ramp the actual target values.
const target = next.layers[slot].params;
const target = next.params;
for (const [name, def] of Object.entries(spec.module.params || {})) {
if (def.type === 'palette' || typeof out[name] !== 'number') continue;
if (typeof target[name] !== 'number') continue;
@@ -138,83 +224,85 @@ export class ArcDriver {
* that was correct on every repeat but wrong the first time through, which is
* exactly the kind of fault the determinism checks exist to surface.
*/
_boundarySlope(sectionIndex) {
_boundarySlope(cue) {
if (!this._slopeCache) this._slopeCache = new Map();
if (this._slopeCache.has(sectionIndex)) return this._slopeCache.get(sectionIndex);
if (this._slopeCache.has(cue.index)) return this._slopeCache.get(cue.index);
const section = this.look.sections[sectionIndex];
const frame = Math.max(0, section.startFrame - 1);
const frame = Math.max(0, cue.startFrame - 1);
const value = this.track.tracks.buildSlope[frame] || 0;
this._slopeCache.set(sectionIndex, value);
this._slopeCache.set(cue.index, value);
return value;
}
/**
* Compute the active layer stack for a frame.
*
* The crossfade runs FORWARD from a boundary: the outgoing scene holds at
* full opacity while the incoming one fades in over it. That keeps the
* boundary frame itself a clean state, which is what makes a boundary seek
* exact without warm-up.
* The transition runs FORWARD from a cue: the outgoing image holds at full
* opacity while the incoming one fades in over it. That keeps the cue frame
* itself a clean state, which is what makes a boundary seek exact without
* warm-up. A hard cut is the same path with a two-frame fade — it is still
* a ramp rather than a jump, because a single-frame swap of two bright
* scenes is a flash, and the flash meter is not decorative.
*/
update(frame, features) {
const track = this.track;
const time = frame / track.fps;
const sectionIndex = track.sectionIndexAt(frame);
const section = this.look.sections[sectionIndex];
if (!section) return this.activeLayers;
const time = frame / this.track.fps;
const cueIndex = this._cueIndexAt(frame);
const cue = this.cues[cueIndex];
if (!cue) return this.activeLayers;
const section = this.look.sections[cue.sectionIndex];
const framesIntoSection = frame - section.startFrame;
const fading = sectionIndex > 0 && framesIntoSection < this.crossfadeFrames;
const t = fading ? framesIntoSection / this.crossfadeFrames : 1;
const framesIntoCue = frame - cue.startFrame;
const previous = cueIndex > 0 ? this.cues[cueIndex - 1] : null;
const fading = !!previous && framesIntoCue < cue.fadeFrames;
const t = fading ? framesIntoCue / cue.fadeFrames : 1;
const eased = t * t * (3 - 2 * t);
const layers = [];
if (fading) {
const previousIndex = sectionIndex - 1;
const outgoing = this._layerFor(previousIndex);
// buildSlope is discontinuous at a section boundary by construction:
// it ramps to ~1 through the bars before the change and is 0
// immediately after. The outgoing layer is still on screen when that
// happens, so feeding it the new section's features collapses its
// lookahead ramp in a single frame — a visible pop precisely at the
// transition. Hold the slope it had going into the boundary; it
// finished its build, and it stays there while it fades out. Within a
// section the slope is continuous, so the live value is correct there.
const outgoingFeatures = cue.atSectionStart
? { ...features, buildSlope: this._boundarySlope(cue) }
: features;
// buildSlope is discontinuous at a boundary by construction: it ramps
// to ~1 through the bars before the change and is 0 immediately after.
// The outgoing layer is still on screen when that happens, so feeding
// it the new section's features collapses its lookahead ramp in a
// single frame — a visible pop precisely at the transition. Hold the
// slope it had going into the boundary; it finished its build, and it
// stays there while it fades out.
outgoing.setParams(this._paramsAt(previousIndex, 0, time, {
...features,
buildSlope: this._boundarySlope(sectionIndex),
}));
outgoing.opacity = 1;
outgoing.blend = 'normal';
outgoing.setPalette(this.look.palette);
layers.push(outgoing);
for (let slot = 0; slot < this._stackSize(previous); slot++) {
const spec = this._specFor(previous.sectionIndex, previous.variant, slot);
const layer = this._layerFor(previous.sectionIndex, previous.variant, slot);
layer.setParams(this._paramsAt(previous, slot, time, outgoingFeatures));
layer.opacity = slot === 0 ? 1 : spec.opacity;
layer.blend = slot === 0 ? 'normal' : spec.blend;
layer.setPalette(this.look.palette);
layer.setPersonality(this.look.personality);
layers.push(layer);
}
}
const current = this._layerFor(sectionIndex);
current.setParams(this._paramsAt(sectionIndex, 0, time, features));
current.opacity = fading ? eased : 1;
current.blend = 'normal';
current.setPalette(this.look.palette);
layers.push(current);
// Extra composited layers declared on the section (Phase 5 stacks).
for (let slot = 1; slot < section.layers.length; slot++) {
const spec = section.layers[slot];
const layer = this._layerFor(sectionIndex, slot);
layer.setParams(this._paramsAt(sectionIndex, slot, time, features));
layer.opacity = spec.opacity * (fading ? eased : 1);
layer.blend = spec.blend;
for (let slot = 0; slot < this._stackSize(cue); slot++) {
const spec = this._specFor(cue.sectionIndex, cue.variant, slot);
const layer = this._layerFor(cue.sectionIndex, cue.variant, slot);
layer.setParams(this._paramsAt(cue, slot, time, features));
layer.opacity = (slot === 0 ? 1 : spec.opacity) * (fading ? eased : 1);
layer.blend = slot === 0 ? 'normal' : spec.blend;
layer.setPalette(this.look.palette);
layer.setPersonality(this.look.personality);
layers.push(layer);
}
this.state = {
sectionIndex,
sectionIndex: cue.sectionIndex,
shotIndex: cue.shotIndex,
shotCount: section.shots ? section.shots.length : 1,
variant: cue.variant,
kind: section.kind,
crossfade: fading ? eased : 0,
sceneName: section.layers[0].module.name,
sceneName: this._specFor(cue.sectionIndex, cue.variant, 0).module.name,
buildSlope: features ? features.buildSlope || 0 : 0,
};
@@ -222,6 +310,12 @@ export class ArcDriver {
return layers;
}
_stackSize(cue) {
const section = this.look.sections[cue.sectionIndex];
const stack = (section.variants && section.variants[cue.variant]) || section.layers;
return stack.length;
}
/** Layers changed identity — the compositor needs the new list. */
layersChanged(previous) {
if (!previous || previous.length !== this.activeLayers.length) return true;
@@ -237,11 +331,34 @@ export class ArcDriver {
this.driftPlans.delete(key);
}
}
// A reroll re-plans the section's shots, so the cue list is stale too.
this.cues = this._buildCues();
this._slopeCache = null;
}
invalidateAll() {
this.dispose();
this.driftPlans.clear();
this.cues = this._buildCues();
this._slopeCache = null;
}
/**
* Create and compile every layer the look will ever show.
*
* Layers are otherwise built on first use, which means a shader compile on
* the frame of a cut — a visible hitch, and there are now many more cuts
* than there were sections. Paying for all of them once at load is cheaper
* than paying for one at every transition.
*/
prewarm(onLayer = null) {
for (const cue of this.cues) {
for (let slot = 0; slot < this._stackSize(cue); slot++) {
const layer = this._layerFor(cue.sectionIndex, cue.variant, slot);
if (onLayer) onLayer(layer);
}
}
return this;
}
/** Push a palette change through without rebuilding layers. */
flow-state/src/look/LookGenerator.js
+161 -64
View File
@@ -8,6 +8,8 @@ import { Rng, hashSamples } from '../engine/rng.js';
import { AudioPalette, generateUsablePalette } from './palette.js';
import { scenes, scenesInFamily, FAMILIES } from '../scenes/registry.js';
import { sampleValues, defaultValues } from '../params/schema.js';
import { planShots } from './shots.js';
import { generatePersonality, sceneHonours, describePersonality } from './Personality.js';
/**
* Which families suit which section kind, in preference order.
@@ -49,15 +51,64 @@ function biasFor(section, summary) {
}
/**
* Scenes are chosen per section KIND, not per section.
* Scenes eligible for a section kind, weighted by how well the family fits.
*
* All of a track's drops therefore share a scene, all its breakdowns share
* another, and the video acquires an identity instead of reading as a shuffle.
* Variation between two sections of the same kind comes from their parameter
* sets and from the arc driver's drift, which is enough to keep them distinct
* without losing the through-line.
* `signature` is the track's personality signature, and it is a hard filter
* rather than a weight: a scene with no way to express what the track is built
* on is not a worse choice, it is the shot that was clearly filmed somewhere
* else. See look/Personality.js.
*/
function assignScenesByKind(sections, rng) {
function candidatesForKind(kind, used, signature = []) {
const families = FAMILY_BY_KIND[kind] || Object.keys(FAMILIES);
const candidates = [];
for (const family of families) {
const inFamily = scenesInFamily(family)
.filter((m) => m.role !== 'accent' && sceneHonours(m, signature));
// Weight by family preference order, and push down anything already
// used so a five-section track doesn't show one scene five times.
const weight = families.length - families.indexOf(family);
for (const scene of inFamily) {
candidates.push({ scene, weight: weight * (used.has(scene.name) ? 0.15 : 1) });
}
}
if (!candidates.length) {
// Every family for this kind was emptied by the signature filter. Widen
// to the whole library, still honouring the signature; only if that is
// empty too does the personality lose and the video keep its scenes.
const anywhere = scenes.filter((m) => m.role !== 'accent' && sceneHonours(m, signature));
const pool = anywhere.length ? anywhere : scenes.filter((m) => m.role !== 'accent');
return pool.map((scene) => ({ scene, weight: 1 }));
}
return candidates;
}
/**
* How many stage visuals a kind rotates between. Busy material takes more.
*
* Sized against the library rather than picked out of the air: a kind draws
* from three families, which is seven to nine non-accent scenes, so a roster of
* four still leaves the weighting room to avoid what other kinds already took.
* Variants a section never reaches cost nothing — layers are built per cue, so
* only the ones its shots actually show are ever compiled.
*/
function rosterSizeFor(kind) {
return (KIND_ENERGY[kind] ?? 0.5) > 0.5 ? 4 : 3;
}
/**
* Scenes are chosen per section KIND, not per section — and a kind gets a
* ROSTER of two or three, not one.
*
* All of a track's drops therefore cut between the same small set of visuals,
* all its breakdowns between another, and the video acquires an identity
* instead of reading as a shuffle. The first entry is the anchor: it opens
* every section of that kind and comes back most often, so the rotation reads
* as one idea with variations rather than as three unrelated scenes.
*
* Variation between two sections of the same kind comes from their parameter
* sets, from where their shots fall, and from the arc driver's drift.
*/
function assignRostersByKind(sections, rng, signature = []) {
const byKind = new Map();
const used = new Set();
@@ -68,27 +119,27 @@ function assignScenesByKind(sections, rng) {
kinds.sort((a, b) => priority.indexOf(a) - priority.indexOf(b));
for (const kind of kinds) {
const families = FAMILY_BY_KIND[kind] || Object.keys(FAMILIES);
let candidates = [];
for (const family of families) {
const inFamily = scenesInFamily(family).filter((m) => m.role !== 'accent');
// Weight by family preference order, and push down anything already
// used so a five-section track doesn't show one scene five times.
const weight = families.length - families.indexOf(family);
for (const scene of inFamily) {
candidates.push({ scene, weight: weight * (used.has(scene.name) ? 0.15 : 1) });
}
}
if (!candidates.length) {
candidates = scenes.filter((m) => m.role !== 'accent').map((scene) => ({ scene, weight: 1 }));
const roster = [];
const size = rosterSizeFor(kind);
for (let slot = 0; slot < size; slot++) {
const pool = candidatesForKind(kind, used, signature)
.filter((c) => !roster.includes(c.scene))
.map((c) => ({
scene: c.scene,
// Companions stay in the anchor's family where possible: a
// cut inside a section should change the image, not the
// whole visual language.
weight: c.weight * (roster.length && c.scene.family === roster[0].family ? 3 : 1),
}));
if (!pool.length) break;
const chosen = rng.pickWeighted(pool.map((c) => c.scene), pool.map((c) => c.weight));
roster.push(chosen);
used.add(chosen.name);
}
const chosen = rng.pickWeighted(
candidates.map((c) => c.scene),
candidates.map((c) => c.weight),
);
byKind.set(kind, chosen);
used.add(chosen.name);
byKind.set(kind, roster.length ? roster : [scenes[0]]);
}
return byKind;
}
@@ -127,6 +178,36 @@ function derivePost(summary, rng) {
};
}
/**
* One layer stack: a background scene plus an optional accent over it.
*
* The accent is composited additively at low opacity and drawn from a DIFFERENT
* family, so it reads as depth rather than as a second competing scene. Quiet
* material mostly goes without — an intro is supposed to be sparse.
*/
function buildStack(module, accentRoster, bias, rng) {
const layers = [{
module,
params: sampleValues(module, rng, bias),
seed: rng.int(0, 0x7fffffff),
blend: 'normal',
opacity: 1,
}];
if (accentRoster.length && rng.bool(bias.energy * 0.8)) {
const eligible = accentRoster.filter((m) => m.family !== module.family);
const accent = rng.pick(eligible.length ? eligible : accentRoster);
layers.push({
module: accent,
params: sampleValues(accent, rng.fork('accent'), bias),
seed: rng.int(0, 0x7fffffff),
blend: rng.pickWeighted(['add', 'screen'], [2, 1]),
opacity: rng.range(0.18, 0.5),
});
}
return layers;
}
/**
* @param {FeatureTrack} track
* @param {object} options
@@ -143,43 +224,36 @@ export function generateLook(track, { seed = null, samples = null, overrides = n
const paletteSource = new AudioPalette(summary, rng.fork('palette'));
const palette = generateUsablePalette(paletteSource, 6);
const sceneByKind = assignScenesByKind(track.sections, rng.fork('scenes'));
// The production design, decided before a single scene is cast — casting
// depends on it. See look/Personality.js.
const personality = generatePersonality(summary, rng.fork('personality'), (signature) =>
scenes.filter((m) => m.role !== 'accent' && sceneHonours(m, signature)).length);
const rosterByKind = assignRostersByKind(
track.sections, rng.fork('scenes'), personality.signature);
const { post, feedback } = derivePost(summary, rng.fork('post'));
// Scenes that declare role 'accent' composite over a background rather than
// being one — most of their frame is empty by design. They are never chosen
// as a section's primary scene.
const accentRoster = scenes.filter((m) => m.role === 'accent');
// Accents honour the signature too where they can. If none can, the track
// goes without depth layers rather than putting an off-design element into
// every stack.
const accentRoster = scenes.filter((m) => m.role === 'accent'
&& sceneHonours(m, personality.signature));
const sections = track.sections.map((section) => {
const module = sceneByKind.get(section.kind) || scenes[0];
const sectionRng = rng.fork(`section:${section.index}:${module.name}`);
const roster = rosterByKind.get(section.kind) || [scenes[0]];
const sectionRng = rng.fork(`section:${section.index}:${roster[0].name}`);
const bias = biasFor(section, summary);
const layers = [{
module,
params: sampleValues(module, sectionRng, bias),
seed: sectionRng.int(0, 0x7fffffff),
blend: 'normal',
opacity: 1,
}];
const variants = roster.map((module, v) => buildStack(
module, accentRoster, bias, sectionRng.fork(`variant:${section.index}:${v}`),
));
// Accent layer. Composited additively over the background at low opacity,
// and drawn from a DIFFERENT family so it reads as depth rather than as a
// second competing scene. Quiet sections mostly go without — an intro is
// supposed to be sparse.
const accentChance = bias.energy * 0.8;
if (accentRoster.length && sectionRng.bool(accentChance)) {
const accent = sectionRng.pick(accentRoster.filter((m) => m.family !== module.family) || accentRoster)
|| accentRoster[0];
layers.push({
module: accent,
params: sampleValues(accent, sectionRng.fork(`accent:${section.index}`), bias),
seed: sectionRng.int(0, 0x7fffffff),
blend: sectionRng.pickWeighted(['add', 'screen'], [2, 1]),
opacity: sectionRng.range(0.18, 0.5),
});
}
const shots = planShots(
section, track, bias, variants.length, sectionRng.fork(`shots:${section.index}`),
);
return {
index: section.index,
@@ -190,13 +264,19 @@ export function generateLook(track, { seed = null, samples = null, overrides = n
end: section.end,
locked: false,
bias,
layers,
variants,
shots,
// The anchor stack, aliased. Everything that predates shots — the
// param panel, presets, the checks — edits a section through this,
// and it is the same object the first variant holds.
layers: variants[0],
};
});
const look = {
seed: resolvedSeed,
palette,
personality,
paletteScheme: paletteSource.lastScheme,
post,
feedback,
@@ -213,17 +293,33 @@ export function rerollSection(look, track, sectionIndex, salt = 0) {
if (!section || section.locked) return look;
const rng = new Rng((look.seed ^ (sectionIndex * 0x9e3779b9) ^ (salt * 0x85ebca6b)) >>> 0);
const signature = (look.personality && look.personality.signature) || [];
const families = FAMILY_BY_KIND[section.kind] || Object.keys(FAMILIES);
const candidates = families.flatMap((f) => scenesInFamily(f)).filter((m) => m.role !== 'accent');
const module = candidates.length ? rng.pick(candidates) : scenes[0];
let candidates = families.flatMap((f) => scenesInFamily(f))
.filter((m) => m.role !== 'accent' && sceneHonours(m, signature));
if (!candidates.length) {
candidates = scenes.filter((m) => m.role !== 'accent' && sceneHonours(m, signature));
}
section.layers = [{
module,
params: sampleValues(module, rng, section.bias),
seed: rng.int(0, 0x7fffffff),
blend: 'normal',
opacity: 1,
}];
// Re-roll the whole roster, not just the anchor: the section's shots cut
// between all of them, so replacing one would leave the section half old.
const size = Math.min(rosterSizeFor(section.kind), Math.max(1, candidates.length));
const roster = [];
while (roster.length < size) {
const pool = candidates.filter((m) => !roster.includes(m));
if (!pool.length) break;
roster.push(rng.pick(pool));
}
if (!roster.length) roster.push(scenes[0]);
const accentRoster = scenes.filter((m) => m.role === 'accent');
section.variants = roster.map((module, v) => buildStack(
module, accentRoster, section.bias, rng.fork(`variant:${v}`),
));
section.shots = planShots(
section, track, section.bias, section.variants.length, rng.fork('shots'),
);
section.layers = section.variants[0];
return look;
}
@@ -256,7 +352,8 @@ function applyOverrides(look, overrides) {
/** Compact description, used by the HUD and by check output. */
export function describeLook(look) {
const kinds = look.sections.map((s) => `${s.kind}:${s.layers[0].module.name}`);
return `seed ${look.seed.toString(16)} · ${look.paletteScheme} · ${[...new Set(kinds)].join(', ')}`;
return `seed ${look.seed.toString(16)} · ${look.paletteScheme} · ` +
`${describePersonality(look.personality)} · ${[...new Set(kinds)].join(', ')}`;
}
export { defaultValues };
flow-state/src/look/Personality.js
+205
View File
@@ -0,0 +1,205 @@
// The track's production design.
//
// A music video is not held together by its cuts. It is held together by the
// fact that every shot was filmed in the same location, with the same actors,
// the same camera operator and the same art direction. Cut between two shots of
// that and it reads as one piece even when the framing changes completely.
//
// Nothing in this project had an equivalent. Sections shared a palette and a
// post grade, and past that every scene was a separate short film. This module
// is the missing layer: one procedurally generated PERSONALITY per track, in
// four traits that map onto the four things a production shares.
//
// shape — the actors. A signature form: how many sides, how round, how
// elongated, at what tilt. Scenes that draw discrete elements stamp
// this form instead of whatever primitive they would have used.
// camera — the operator. A drift direction, a sway, a slow spin, a breathing
// zoom. Applied to the coordinate a scene works in, so every scene
// is filmed by the same hand.
// space — the location. A horizon height, a depth falloff, a background
// wash direction. Scenes that have a sense of place share one.
// style — the art direction. Line weight, edge softness, texture, and how
// many times the frame is folded.
//
// A scene declares which traits it can honour. Each track picks a SIGNATURE of
// one or two traits, and a scene that does not honour all of them is
// disqualified from that track — the library shrinks per track, on purpose. A
// scene with no way to express a hexagon should not appear in the hexagon
// video; it would be the shot that was clearly filmed somewhere else.
//
// Everything here is seeded off the look seed, so a track's personality is as
// reproducible as everything else.
export const TRAITS = ['shape', 'camera', 'space', 'style'];
/**
* Traits eligible to be a track's signature, and how often.
*
* `shape` and `space` carry the most identity — they are the ones a viewer can
* actually name on a second watch — so they are the ones a signature is built
* around. `camera` and `style` are near-universally supported and read as
* treatment rather than as subject, so they join a signature but rarely define
* one alone.
*/
const SIGNATURE_WEIGHTS = { shape: 4, space: 3, camera: 2, style: 2 };
/** Minimum scenes that must survive the signature filter for it to be usable. */
export const MIN_ELIGIBLE_SCENES = 6;
/**
* Generate the personality.
*
* Trait VALUES lean on what was measured in the audio — a bright, noisy track
* gets sharper lines and more texture; a slow one gets a lazier camera — but
* the seed dominates, so two tracks with similar statistics still look like
* different productions.
*
* @param {object} summary FeatureTrack summary
* @param {Rng} rng
* @param {(traits: string[]) => number} countEligible
* How many scenes would survive a given signature. Injected rather than
* imported so this module never has to know the registry exists.
*/
export function generatePersonality(summary, rng, countEligible = null) {
const bright = summary.meanCentroid;
const noisy = Math.min(1, summary.meanFlatness * 3);
const fast = Math.min(1, Math.max(0, (summary.bpm - 80) / 80));
const shape = {
// 0 sides means round. Everything else is a polygon the whole track
// shares — the single most recognisable thing here.
sides: rng.pickWeighted([0, 3, 4, 5, 6, 8], [3, 2, 3, 2, 3, 1]),
roundness: rng.range(0.05, 0.5),
elongation: rng.range(0.85, 1.45),
tilt: rng.range(0, Math.PI),
};
const camera = {
driftAngle: rng.range(0, Math.PI * 2),
// A slow track should not be filmed from a moving car.
driftRate: rng.range(0.01, 0.06) * (0.6 + fast * 0.8),
sway: rng.range(0.0, 0.06),
swayRate: rng.range(0.05, 0.22),
spin: rng.range(-0.05, 0.05),
// Breathing is locked to the bar, so it is the one camera move that
// reads as musical rather than as drifting.
breathe: rng.range(0.0, 0.05),
};
const space = {
horizon: rng.range(0.32, 0.62),
depth: rng.range(0.2, 0.9),
washAngle: rng.range(0, Math.PI * 2),
wash: rng.range(0.1, 0.5),
};
const style = {
lineWeight: 0.4 + bright * 0.4 + rng.range(-0.15, 0.25),
softness: 0.25 + (1 - bright) * 0.4 + rng.range(-0.1, 0.2),
texture: noisy * 0.5 + rng.range(0, 0.25),
// Fold counts stay low and are usually off. Symmetry is the fastest way
// to make a library look like one series and also the fastest way to
// make every track look like a screensaver.
symmetry: rng.pickWeighted([1, 1, 2, 3, 4, 6], [6, 4, 2, 2, 2, 1]),
};
const signature = pickSignature(rng, countEligible);
return { signature, shape, camera, space, style };
}
/**
* Choose the one or two traits this track is BUILT on.
*
* Two is the target: one trait alone is not enough to recognise, and three
* disqualifies most of the library. If the pair leaves too few scenes to build
* rosters from, fall back to the stronger of the two rather than shipping a
* track whose every section is forced onto the same two scenes.
*/
function pickSignature(rng, countEligible) {
const primary = rng.pickWeighted(TRAITS, TRAITS.map((t) => SIGNATURE_WEIGHTS[t]));
const rest = TRAITS.filter((t) => t !== primary);
const secondary = rng.pickWeighted(rest, rest.map((t) => SIGNATURE_WEIGHTS[t]));
const pair = [primary, secondary];
if (!countEligible || countEligible(pair) >= MIN_ELIGIBLE_SCENES) return pair;
const single = [primary];
if (countEligible(single) >= MIN_ELIGIBLE_SCENES) return single;
return [];
}
/** Does a scene honour everything this track is built on? */
export function sceneHonours(module, signature) {
const traits = module.traits || [];
return signature.every((t) => traits.includes(t));
}
/**
* Flatten to the uniform values the shader contract expects.
*
* Neutral defaults matter: a layer with no personality attached must render
* exactly what it rendered before this existed, because the range sweeps and
* the library regression checks build layers directly and would otherwise all
* shift at once.
*/
export function signatureUniforms(personality) {
if (!personality) return NEUTRAL_UNIFORMS;
const { shape, camera, space, style } = personality;
return {
u_sigSides: shape.sides,
u_sigRound: shape.roundness,
u_sigElong: shape.elongation,
u_sigTilt: shape.tilt,
u_sigDrift: [Math.cos(camera.driftAngle) * camera.driftRate,
Math.sin(camera.driftAngle) * camera.driftRate],
u_sigSway: camera.sway,
u_sigSwayRate: camera.swayRate,
u_sigSpin: camera.spin,
u_sigBreathe: camera.breathe,
u_sigHorizon: space.horizon,
u_sigDepth: space.depth,
u_sigWash: [Math.cos(space.washAngle) * space.wash,
Math.sin(space.washAngle) * space.wash],
u_sigLine: style.lineWeight,
u_sigSoft: style.softness,
u_sigTexture: style.texture,
u_sigFold: style.symmetry,
};
}
export const NEUTRAL_UNIFORMS = {
u_sigSides: 0,
u_sigRound: 0.25,
u_sigElong: 1,
u_sigTilt: 0,
u_sigDrift: [0, 0],
u_sigSway: 0,
u_sigSwayRate: 0.1,
u_sigSpin: 0,
u_sigBreathe: 0,
u_sigHorizon: 0.5,
u_sigDepth: 0,
u_sigWash: [0, 0],
u_sigLine: 0.5,
u_sigSoft: 0.5,
u_sigTexture: 0,
u_sigFold: 1,
};
const SHAPE_NAMES = { 0: 'round', 3: 'triangular', 4: 'square', 5: 'pentagonal', 6: 'hexagonal', 8: 'octagonal' };
/** One line for the HUD, the look panel and check output. */
export function describePersonality(personality) {
if (!personality) return 'no personality';
const { signature, shape, style } = personality;
const parts = [
`on ${signature.length ? signature.join('+') : 'nothing'}`,
SHAPE_NAMES[shape.sides] || `${shape.sides}-sided`,
];
if (style.symmetry > 1) parts.push(`${style.symmetry}-fold`);
return parts.join(' · ');
}
flow-state/src/look/shots.js
+161
View File
@@ -0,0 +1,161 @@
// Shot planning: the level of hierarchy between a song section and a frame.
//
// A section is a STAGE of the song (intro, build, drop, …) and can easily run
// ninety seconds. One scene held for ninety seconds reads as a still image with
// a wobble on it, no matter how much per-frame reactivity is underneath. So a
// section is cut into SHOTS, each showing one of the section's few "stage
// visuals" — the roster the look generator picked for that kind of section.
//
// Two rules keep this from turning into a shuffle:
//
// * the roster is per section KIND, not per section, so all of a track's drops
// still cut between the same two or three visuals and the video keeps an
// identity;
// * cuts land on phrase lines, so a change of image lands with the music
// instead of across it.
//
// Shot length follows energy: a drop cuts every four to eight bars, an intro
// holds for eight to sixteen, and nothing holds past the ceiling below.
// Everything here is seeded, so a track always cuts in the same places.
/** Never cut faster than this, whatever the tempo or the energy says. */
export const MIN_SHOT_SECONDS = 5;
/**
* And never hold longer than this either. Half a minute of one image is the
* complaint this whole level of hierarchy exists to answer, so it is a hard
* ceiling rather than something the bar maths is trusted to stay under: at a
* slow tempo sixteen bars is already past it.
*/
export const MAX_SHOT_SECONDS = 22;
/** Below this section energy, a shot change is always a dissolve, never a cut. */
export const HARD_CUT_ENERGY = 0.66;
/** Phrase length for one shot, in bars, from the section's energy. */
function shotBarsFor(energy, rng) {
if (energy > 0.72) return rng.pick([4, 8, 8]);
if (energy > 0.45) return rng.pick([8, 8, 16]);
return rng.pick([8, 16, 16]);
}
/** Nearest downbeat to `time`, or null if none is close enough to be the same line. */
function nearestDownbeat(time, downbeats, tolerance) {
let best = null;
let bestDist = Infinity;
for (const d of downbeats) {
const dist = Math.abs(d - time);
if (dist < bestDist) { bestDist = dist; best = d; }
else if (d > time && dist > bestDist) break; // sorted: past the minimum
}
return best !== null && bestDist <= tolerance ? best : null;
}
/**
* Divide a section into shots.
*
* @param {object} section a track section (start/end/startFrame/endFrame)
* @param {object} track FeatureTrack, for fps and the bar grid
* @param {object} bias the section's bias, for energy
* @param {number} variantCount how many stage visuals the section has
* @param {Rng} rng
* @returns {Array<{index,startFrame,endFrame,variant,hardCut}>}
*/
export function planShots(section, track, bias, variantCount, rng) {
const fps = track.fps;
const duration = Math.max(0, section.end - section.start);
const barSeconds = (track.tempo.period * track.tempo.beatsPerBar) / fps;
const bars = shotBarsFor(bias.energy, rng);
const target = Math.min(MAX_SHOT_SECONDS,
Math.max(MIN_SHOT_SECONDS, barSeconds > 0.2 ? bars * barSeconds : 12));
// Round to the nearest whole number of shots — a 40s section at a 12s target
// gets three of 13s, not three of 12 and a stub — then force enough shots to
// stay under the ceiling, and finally refuse any count that would push a
// shot below the floor. The floor wins if they ever disagree.
//
// The ceiling gets headroom because snapping moves a cut by up to the
// tolerance below, and a cut that snaps LATE would otherwise land just past
// the limit the count was chosen to respect.
let count = Math.max(
Math.round(duration / target),
Math.ceil(duration / (MAX_SHOT_SECONDS * 0.88)),
1,
);
count = Math.min(count, Math.max(1, Math.floor(duration / MIN_SHOT_SECONDS)));
if (variantCount < 2) count = 1;
// Cut times: evenly spaced, then pulled onto the nearest downbeat. The
// tolerance is deliberately under half a shot, so a snap can never reorder
// two cuts or collapse one onto another.
const tolerance = Math.min(barSeconds * 1.5, target * 0.35);
const downbeats = track.tempo.downbeats || [];
const cuts = [];
for (let k = 1; k < count; k++) {
const raw = section.start + (duration * k) / count;
const snapped = nearestDownbeat(raw, downbeats, tolerance);
const previous = cuts.length ? cuts[cuts.length - 1] : section.start;
const fits = (t) => t - previous >= MIN_SHOT_SECONDS && section.end - t >= MIN_SHOT_SECONDS;
// Prefer the downbeat, but a snap that pushes the cut inside the floor
// is worse than an unsnapped cut: dropping it would leave the hold this
// whole mechanism exists to break up.
const time = snapped !== null && fits(snapped) ? snapped : raw;
if (!fits(time)) continue;
cuts.push(time);
}
const bounds = [section.start, ...cuts, section.end];
const shots = [];
const lastSeen = new Array(variantCount).fill(-1);
let previousVariant = -1;
for (let i = 0; i < bounds.length - 1; i++) {
const variant = i === 0 ? 0 : pickVariant(variantCount, previousVariant, lastSeen, i, rng);
lastSeen[variant] = i;
previousVariant = variant;
shots.push({
index: i,
startFrame: i === 0 ? section.startFrame : Math.round(bounds[i] * fps),
endFrame: i === bounds.length - 2 ? section.endFrame : Math.round(bounds[i + 1] * fps),
start: bounds[i],
end: bounds[i + 1],
variant,
// A dissolve is the default. A straight cut is what makes a drop feel
// edited, but on anything calmer it reads as a glitch, so cuts are
// gated on real energy rather than sprinkled everywhere: nothing below
// the threshold ever cuts, and only the loudest material cuts often.
hardCut: bias.energy > HARD_CUT_ENERGY
&& rng.bool(Math.min(0.85, (bias.energy - HARD_CUT_ENERGY) * 2.5)),
});
}
return shots;
}
/**
* Next visual in the rotation.
*
* The shape is A B A C A D: the anchor comes back between companions, so the
* section reads as one idea with departures from it rather than as a playlist.
* It is a strong tendency and not a rule — strict alternation is audible as a
* pattern within about three cycles.
*
* When a companion is due, the LEAST RECENTLY SHOWN one wins. With a roster of
* four that is the difference between a section showing B, C, D and a section
* showing B twice and never reaching D.
*/
function pickVariant(variantCount, previous, lastSeen, shotIndex, rng) {
if (previous !== 0 && rng.bool(0.75)) return 0;
const options = [];
const weights = [];
for (let v = 0; v < variantCount; v++) {
if (v === previous) continue;
options.push(v);
// Unseen variants sort first, then by how long ago they were last up.
// The anchor stays in the draw so the rotation cannot become rigid.
weights.push(v === 0 ? 1 : 2 + (lastSeen[v] < 0 ? variantCount : shotIndex - lastSeen[v]));
}
if (!options.length) return 0;
return rng.pickWeighted(options, weights);
}