dejvino/music-video-gen
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Phase 10: variety, twelve scenes, and tooling to write the next one

Browse Source 
Watching several finished tracks side by side turned up the problem neither
Phase 8 (too few cuts) nor Phase 9 (no through-line) addressed: the same
scene cast in two different videos looked like the same footage twice.
Section bias is nearly identical between two tracks' drops, so both sampled
their parameters around the same centre, and the library's own averageness
did the rest.

Three answers, none of them a new scene:

  Temperament — a per-track hand on every parameter dial: intensity, pace,
  detail, and an extremity that decides how far toward the ends of a range
  the track is willing to sample. Bias comes from the section and is shared
  between tracks; temperament comes from the track and is not.

  Overlays — sometimes a second full scene composited over the shot, from a
  different family, in a blend that preserves what is underneath and never
  above 0.6 opacity. Not always: a stack that always doubled up would read
  as permanently cluttered rather than as occasionally layered.

  A wider palette — hue now derives from SPECTRAL TILT, the log ratio of
  treble to body. The centroid is a number most masters sit in the middle
  of, and the plain body/(body+treble) fraction is worse: low frequencies
  carry most of the energy in all music, so it read 0.98-1.00 for
  everything and four different battery tracks came out within 0.02 of
  each other. The ratio is multiplicative, so its logarithm is what
  spreads — the same four measure -9.3, -5.0, -4.1, -3.8. Also both ways
  round the wheel (violet, magenta and pink were unreachable by
  construction), four new schemes, and seeded chroma profile and lightness
  curve. Closest battery pair went from 0.005 to 0.113.

Twelve scenes take the library to 36, six per family: Aurora Veil, Vortex
Drift, Tide Rings, Ink Bleed, Dust Chamber, Salt Flat, Cargo Belt, Gate
Corridor, Circuit Bloom, Truchet Fold, Signal Decay, Storm Rift. Weighted
toward the 'space' and 'shape' traits, which were thinnest and so the
signatures most likely to run a track out of cast — the Phase 9 casting
rule means the pool a track draws from is smaller than the library.

Also fixes a real one in shots.js: heavy LRU weighting was not enough to
make a section reach its whole roster, and a five-shot section still came
out 0,2,0,2,0 about a fifth of the time. An unseen companion now wins
outright; which one is still free, so only the coverage is guaranteed.

Block Mosh declared the camera trait, assigned sigCamera(p) to a p it then
never read, and passed the lint's evidence grep. The Phase 9 render gate
measured its response to the camera at exactly zero.

--- tooling ---

Adding a scene was mostly boilerplate and round-trips, which is expensive
in both senses. The irreducible cost is the shader body; everything around
it is now mechanical:

  npm run new:scene -- "Name" --family=... --traits=...

writes the module, registers it, and leaves a skeleton that already passes
every gate, with name-derived constants so two skeletons are not twins.

The lint grew the rules that previously needed a GPU to catch: the dead
camera above, prev() with no base image, and large loops with no early
break (with a `// lint: fixed-cost` opt-out for a genuinely fixed-cost
sampling loop). checks.html?scene=Name runs the per-scene acceptance
battery for one scene — ten lines and a verdict instead of rendering the
whole library to find out whether one shader is alive. The same procedure
is a repo skill under .claude/skills/build-visualizer/.

--- checks changed, with the measurements ---

P5 determinism compared two WebGL CONTEXTS, which is not what it is for.
Measured: one context is bit-exact over 40 frames with feedback at 0.6;
two contexts disagree by up to 2/255 whether feedback is on or off. It now
asserts generation is byte-identical (hard) and rendering within 2/255,
since feedback compounds single-level variance.

P10's cross-track comparison measures distance RELATIVE to how much image
there is. Most scenes are mostly dark, so two genuinely different renders
— 25 bars against 53 — scored under 0.02 absolute purely because the black
background agrees with itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Dejvino
2026-08-05 23:59:20 +02:00
co-authored by Claude Opus 5
parent 0ee7a6a3b3
commit 2806ef1386
31 changed files with 2275 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files
flow-state/src/look/LookGenerator.js
+59 -9
View File
@@ -179,27 +179,68 @@ function derivePost(summary, rng) {
}
/**
* One layer stack: a background scene plus an optional accent over it.
* One layer stack: a background scene, sometimes a second scene composited over
* it, sometimes an accent on top of that.
*
* The accent is composited additively at low opacity and drawn from a DIFFERENT
* family, so it reads as depth rather than as a second competing scene. Quiet
* material mostly goes without — an intro is supposed to be sparse.
* Three deliberately different jobs:
*
* background — the shot. Always present, always opaque.
* overlay — a SECOND full scene at partial opacity. Not always: this is the
* variation valve, and a stack that always doubled up would read
* as permanently cluttered rather than as occasionally layered.
* Drawn from a different family so the two images argue instead
* of blurring, and kept off scenes that are already busy.
* accent — the depth pass. Mostly-empty by design (role: 'accent'),
* additive, low opacity.
*
* Quiet material mostly goes without either — an intro is supposed to be sparse.
*/
function buildStack(module, accentRoster, bias, rng) {
function buildStack(module, accentRoster, overlayRoster, bias, rng, temperament) {
const layers = [{
module,
params: sampleValues(module, rng, bias),
params: sampleValues(module, rng, bias, temperament),
seed: rng.int(0, 0x7fffffff),
blend: 'normal',
opacity: 1,
}];
// --- overlay --------------------------------------------------------
// Roughly a third of stacks on busy material, rarely on quiet material, and
// never on a background that is itself a full-frame glitch — two competing
// corruption passes is noise, not depth.
const overlayChance = module.family === 'glitch'
? 0.05
: 0.12 + bias.energy * 0.35 + (temperament ? Math.max(0, temperament.detail) * 0.2 : 0);
const overlays = overlayRoster.filter((m) => m.family !== module.family && m.name !== module.name);
if (overlays.length && rng.bool(Math.min(0.6, overlayChance))) {
const overlay = rng.pick(overlays);
// Screen and add keep the background readable underneath; softlight and
// overlay tint it instead. All four preserve the shot; 'normal' would
// simply replace it, which is what the shot cut is for.
const blend = rng.pickWeighted(['screen', 'add', 'softlight', 'overlay'], [3, 2, 2, 1]);
layers.push({
module: overlay,
params: sampleValues(overlay, rng.fork(`overlay:${overlay.name}`), {
// An overlay reads as texture over the shot, so it is sampled
// sparser and calmer than it would be as a background.
...bias,
density: Math.max(0, bias.density - 0.25),
energy: Math.max(0, bias.energy - 0.2),
}, temperament),
seed: rng.int(0, 0x7fffffff),
blend,
opacity: blend === 'add' ? rng.range(0.18, 0.42) : rng.range(0.25, 0.55),
});
}
// --- accent ---------------------------------------------------------
if (accentRoster.length && rng.bool(bias.energy * 0.8)) {
const eligible = accentRoster.filter((m) => m.family !== module.family);
const accent = rng.pick(eligible.length ? eligible : accentRoster);
layers.push({
module: accent,
params: sampleValues(accent, rng.fork('accent'), bias),
params: sampleValues(accent, rng.fork('accent'), bias, temperament),
seed: rng.int(0, 0x7fffffff),
blend: rng.pickWeighted(['add', 'screen'], [2, 1]),
opacity: rng.range(0.18, 0.5),
@@ -242,13 +283,20 @@ export function generateLook(track, { seed = null, samples = null, overrides = n
const accentRoster = scenes.filter((m) => m.role === 'accent'
&& sceneHonours(m, personality.signature));
// Scenes eligible to be composited OVER a background. Same casting rule as
// everything else — an overlay is on screen as much as the shot under it,
// so an off-design one would be just as visible.
const overlayRoster = scenes.filter((m) => m.role !== 'accent'
&& sceneHonours(m, personality.signature));
const sections = track.sections.map((section) => {
const roster = rosterByKind.get(section.kind) || [scenes[0]];
const sectionRng = rng.fork(`section:${section.index}:${roster[0].name}`);
const bias = biasFor(section, summary);
const variants = roster.map((module, v) => buildStack(
module, accentRoster, bias, sectionRng.fork(`variant:${section.index}:${v}`),
module, accentRoster, overlayRoster, bias,
sectionRng.fork(`variant:${section.index}:${v}`), personality.temperament,
));
const shots = planShots(
@@ -313,8 +361,10 @@ export function rerollSection(look, track, sectionIndex, salt = 0) {
if (!roster.length) roster.push(scenes[0]);
const accentRoster = scenes.filter((m) => m.role === 'accent');
const overlayRoster = scenes.filter((m) => m.role !== 'accent' && sceneHonours(m, signature));
section.variants = roster.map((module, v) => buildStack(
module, accentRoster, section.bias, rng.fork(`variant:${v}`),
module, accentRoster, overlayRoster, section.bias, rng.fork(`variant:${v}`),
look.personality && look.personality.temperament,
));
section.shots = planShots(
section, track, section.bias, section.variants.length, rng.fork('shots'),
flow-state/src/look/Personality.js
+30 -1
View File
@@ -21,6 +21,14 @@
// style — the art direction. Line weight, edge softness, texture, and how
// many times the frame is folded.
//
// Plus a fifth thing that is not a trait and is not declared by anyone: the
// TEMPERAMENT. Traits decide what a track looks like; temperament decides how
// hard it commits. It is the track's hand on every scene's parameter dials, and
// it exists because section bias alone is nearly identical between two tracks'
// drops — so one scene cast in two videos sampled around the same centre both
// times and the videos looked like each other. Temperament is per track and
// pushes those samples apart. See params/schema.js sampleValues.
//
// A scene declares which traits it can honour. Each track picks a SIGNATURE of
// one or two traits, and a scene that does not honour all of them is
// disqualified from that track — the library shrinks per track, on purpose. A
@@ -64,6 +72,7 @@ export function generatePersonality(summary, rng, countEligible = null) {
const bright = summary.meanCentroid;
const noisy = Math.min(1, summary.meanFlatness * 3);
const fast = Math.min(1, Math.max(0, (summary.bpm - 80) / 80));
const loud = Math.min(1, (summary.dynamicRange ?? 0.5) + (summary.meanLoudness ?? 0.3));
const shape = {
// 0 sides means round. Everything else is a polygon the whole track
@@ -107,9 +116,25 @@ export function generatePersonality(summary, rng, countEligible = null) {
symmetry: rng.pickWeighted([1, 1, 2, 3, 4, 6], [6, 4, 2, 2, 2, 1]),
};
// How hard this track pushes every scene it casts. Deliberately wide, and
// deliberately not derived from the section: two tracks must be able to
// disagree about what "a drop" means.
const temperament = {
// Up or down on the energy/density dials.
intensity: rng.range(-0.85, 0.85) * (0.5 + loud * 0.9),
// Up or down on anything that moves.
pace: rng.range(-0.8, 0.8) * (0.55 + fast * 0.8),
// Fine and busy, or few and large. Independent of loudness on purpose —
// a quiet track can be intricate and a loud one can be blunt.
detail: rng.range(-0.6, 0.6),
// How far toward the ends of a range this track is willing to sample.
// The single most effective knob against "every video looks average".
extremity: rng.range(0.25, 0.95),
};
const signature = pickSignature(rng, countEligible);
return { signature, shape, camera, space, style };
return { signature, shape, camera, space, style, temperament };
}
/**
@@ -205,5 +230,9 @@ export function describePersonality(personality) {
SHAPE_NAMES[shape.sides] || `${shape.sides}-sided`,
];
if (style.symmetry > 1) parts.push(`${style.symmetry}-fold`);
if (personality.temperament) {
const t = personality.temperament;
parts.push(`${t.intensity >= 0 ? 'hot' : 'cool'} ${t.extremity.toFixed(2)} bold`);
}
return parts.join(' · ');
}
flow-state/src/look/palette.js
+94 -23
View File
@@ -70,10 +70,31 @@ const SCHEMES = {
triad: (h) => [h, h + 2.094, h + 4.189, h + 0.5, h + 2.6, h + 4.7],
split: (h) => [h, h + 2.6, h + 3.7, h + 0.35, h + 2.9, h + 3.4],
duo: (h) => [h, h + 1.9, h + 0.2, h + 2.1, h - 0.25, h + 1.7],
// Four evenly spaced hues plus two repeats: the widest spread available, and
// the reason a track can now come out looking like four colours rather than
// a gradient between two.
tetrad: (h) => [h, h + 1.571, h + 3.142, h + 4.712, h + 0.8, h + 2.4],
// One hue family carrying the frame, with a single far-off pop. Reads as a
// deliberate art-directed choice rather than as a spectrum.
accented: (h) => [h, h + 0.25, h - 0.2, h + 0.45, h + 2.9, h + 3.05],
// One hue, everything else carried by lightness and chroma. Needs the
// widened L range below to stay legible, and gives the library the
// near-monochrome look it could not previously reach at all.
mono: (h) => [h, h + 0.12, h - 0.1, h + 0.18, h - 0.16, h + 0.08],
};
export const SCHEME_NAMES = Object.keys(SCHEMES);
/**
* Stretch a value around a centre so a narrow real-world range fills 0..1.
*
* A logistic rather than a linear rescale, because the tails must stay bounded:
* an unusually bass-heavy track should land at the warm end, not past it.
*/
function expand(x, centre = 0.5, slope = 3.0) {
return 1 / (1 + Math.exp(-slope * (x - centre) * 4));
}
/** The interface a palette source implements. */
export class PaletteSource {
/** @returns {number[][]} array of [r,g,b] in 0..1 */
@@ -107,25 +128,50 @@ export class AudioPalette extends PaletteSource {
const rng = this.rng;
// --- Temperature: the track's timbre signature, not its loudness ---
// Warmth places spectral mass from the body (sub/low/mid) against the
// trebles (high/air). It is folded through the whole band profile rather
// than the centroid alone, because the centroid is a one number that
// most mastered pop sits in the middle of — which was why every track
// flared green/purple. This is the feel of the sound: a voice-and-body
// forward track belongs to the warm end of the wheel, a crisp or airy
// track to the cool end.
const body = (bandBalance.sub ?? 0.4) * 0.6
+ (bandBalance.low ?? 0.4) * 0.9
+ (bandBalance.mid ?? 0.3) * 0.4;
const treble = (bandBalance.high ?? 0.3) * 0.7
+ (bandBalance.air ?? 0.3) * 0.5
+ meanCentroid * 0.5;
const warmth = body / (body + treble + 1e-6); // 0 = cold, 1 = warm
// SPECTRAL TILT — the log ratio of treble energy to body energy — rather
// than either the centroid or a plain body/(body+treble) fraction.
//
// Both of those were tried and both collapse. The centroid is one number
// most mastered music sits in the middle of. The plain fraction is worse:
// low frequencies carry most of the energy in essentially all music, so
// it reads 0.98-1.00 for everything and the four check-battery tracks
// came out within 0.02 of each other. The ratio is MULTIPLICATIVE, so its
// logarithm is what actually spreads: the same four tracks measure -9.3,
// -5.0, -4.1 and -3.8, which is a real axis to hang a palette on.
const bands = {
sub: bandBalance.sub ?? 0.2, low: bandBalance.low ?? 0.2, mid: bandBalance.mid ?? 0.2,
high: bandBalance.high ?? 0.2, air: bandBalance.air ?? 0.2,
};
const body = bands.sub * 1.0 + bands.low * 0.9 + bands.mid * 0.35 + 1e-7;
const treble = bands.high * 0.9 + bands.air * 1.0 + bands.mid * 0.15 + 1e-7;
const tilt = Math.log(treble / body);
// Hue sweeps cold(blue, 240°) -> cyan -> green -> yellow -> warm(red),
// so warm material finally reaches red/yellow rather than pooling in the
// blue/green gap. A little seeded jitter keeps identical tracks apart.
const baseHue = (1 - warmth) * (Math.PI * 4 / 3) + rng.range(-0.45, 0.45);
// -9 (nothing above the low mids) .. -2 (bright, airy) covers the range
// real material occupies; the centroid keeps a minority vote so two
// tracks with the same tilt but different brightness still differ.
const tiltWarmth = Math.max(0, Math.min(1, (-2 - tilt) / 7));
const warmth = tiltWarmth * 0.7 + (1 - meanCentroid) * 0.3;
// Hue sweeps cold -> warm, but which WAY round the wheel is seeded.
// Going down from red through yellow and green to blue is the obvious
// route and the only one that existed; it also means violet, magenta and
// pink were unreachable for every track ever generated, because they sit
// on the arc the sweep skipped. Half of tracks now take the other way
// round, so the same warm/cool reading can land on crimson-through-
// magenta instead of crimson-through-amber.
//
// Both routes span the same arc. A short return leg would mean tracks
// that took it barely differ in hue however different they sound.
const clockwise = rng.bool(0.5);
const span = (clockwise ? 1 : -1) * Math.PI * 4 / 3;
// Tempo and dynamics nudge the hue too. Timbre is the main axis, but two
// tracks can be timbrally alike and still feel different — a slow
// spacious one and a fast compressed one should not be handed the same
// colour just because they occupy the same part of the spectrum.
const feel = ((bpm - 120) / 200 + (dynamicRange - 0.5) * 0.5) * 0.6;
const baseHue = (1 - warmth) * span + feel + rng.range(-0.45, 0.45);
const schemeName = rng.pick(SCHEME_NAMES);
const hues = SCHEMES[schemeName](baseHue, rng);
@@ -135,25 +181,50 @@ export class AudioPalette extends PaletteSource {
// stays muted. This is the "does it pop" axis, orthogonal to timbre.
const fast = Math.min(1, Math.max(0, (bpm - 80) / 150));
const energy = Math.min(1, fast * 0.4 + (1 - Math.min(1, meanFlatness)) * 0.4 + dynamicRange * 0.3);
const chromaBase = 0.10 + energy * 0.16;
// Vividness is the track's, but how far it commits is seeded — the old
// fixed mapping meant two tracks with similar statistics got not just
// similar hues but the same saturation, which is most of why they read
// as the same palette.
const vividness = rng.range(0.55, 1.45);
const chromaBase = (0.09 + energy * 0.19) * vividness;
// Chroma profile: does the palette saturate in the middle (the old fixed
// behaviour), at the bright end, or barely at all? A near-neutral set
// with one vivid accent is a look the generator could not previously
// produce.
const profile = rng.pickWeighted(['arch', 'rising', 'flat', 'accent'], [3, 2, 2, 2]);
const chromaAt = (t) => {
switch (profile) {
case 'rising': return 0.35 + t * 1.1;
case 'flat': return 0.9;
case 'accent': return t > 0.72 ? 1.5 : 0.28;
default: return 0.55 + Math.sin(t * Math.PI) * 0.75;
}
};
// A dynamic mercury gets a wider light-to-dark range; warmth keeps warm
// tones from sinking into brown, since dark + orange is mud.
const spread = 0.30 + Math.min(1, dynamicRange) * 0.30;
const anchor = 0.40 - warmth * 0.06 + rng.range(-0.05, 0.10);
const spread = (0.30 + Math.min(1, dynamicRange) * 0.30) * rng.range(0.85, 1.5);
const anchor = 0.40 - warmth * 0.06 + rng.range(-0.14, 0.16);
// How the lightness steps are distributed: 1.7 keeps most entries dark
// with a couple of bright accents (the old fixed curve), below 1 spreads
// them evenly, above 2 makes the set almost entirely dark with one
// highlight. Another axis two similar tracks can differ on.
const curve = rng.range(0.75, 2.4);
const colors = [];
for (let i = 0; i < count; i++) {
const t = count > 1 ? i / (count - 1) : 0;
// Deliberately non-linear: most entries mid-dark, one or two bright.
// Scenes use pal(0) as a base and higher indices as accents.
const L = Math.max(0.06, Math.min(0.95, anchor + Math.pow(t, 1.7) * spread));
const C = chromaBase * (0.55 + Math.sin(t * Math.PI) * 0.75) + rng.range(-0.012, 0.012);
const L = Math.max(0.05, Math.min(0.97, anchor + Math.pow(t, curve) * spread));
const C = chromaBase * chromaAt(t) + rng.range(-0.012, 0.012);
const h = hues[i % hues.length] + rng.range(-0.08, 0.08);
colors.push(oklchToRgb(L, Math.max(0, C), h));
}
this.lastScheme = schemeName;
this.lastProfile = profile;
return colors;
}
}
flow-state/src/look/shots.js
+15 -3
View File
@@ -147,14 +147,26 @@ export function planShots(section, track, bias, variantCount, rng) {
function pickVariant(variantCount, previous, lastSeen, shotIndex, rng) {
if (previous !== 0 && rng.bool(0.75)) return 0;
// A companion this section has not shown yet wins outright. Weighting it
// heavily was not enough — measured, a five-shot section still came out
// 0,2,0,2,0 about a fifth of the time, so the roster existed and the shots
// never reached it. Which unseen one is still a free choice, so the order
// varies between sections; only the coverage is guaranteed.
const unseen = [];
for (let v = 1; v < variantCount; v++) {
if (v !== previous && lastSeen[v] < 0) unseen.push(v);
}
if (unseen.length) return rng.pick(unseen);
const options = [];
const weights = [];
for (let v = 0; v < variantCount; v++) {
if (v === previous) continue;
options.push(v);
// Unseen variants sort first, then by how long ago they were last up.
// The anchor stays in the draw so the rotation cannot become rigid.
weights.push(v === 0 ? 1 : 2 + (lastSeen[v] < 0 ? variantCount : shotIndex - lastSeen[v]));
// Everything has been shown at least once: fall back to least recently
// seen, with the anchor kept in the draw so the rotation cannot become
// a rigid cycle.
weights.push(v === 0 ? 1 : 2 + (shotIndex - lastSeen[v]));
}
if (!options.length) return 0;
return rng.pickWeighted(options, weights);