dejvino/music-video-gen
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

One scene, wide: songs against seeds

Browse Source 
Six thumbnails is enough to rank sixty-eight scenes and far too few to study
one. Worse, six songs at one seed each confounds the two inputs: a scene whose
frames are interchangeable might be ignoring the identity or ignoring the music,
and the library gallery cannot say which.

So a scene's name in the gallery now opens it on a grid — songs down, seeds
across, up to every song in the bank by ten draws. Buttons pick the size and
walk onto a fresh block of seeds, which is a much better answer to "is it flat
or was that ten unlucky rolls" than staring at the same ten. The state lives in
the URL, so every button is also a back button.

Three scores rather than one, and the two new ones are the diagnosis: ACROSS
SEEDS is the same song with a different draw, so low means the scene ignores the
identity; ACROSS SONGS is the same draw against different music, so low means it
ignores the song. They point at different fixes. The closest pair is outlined,
because at 170 cells no eye is finding it.

The first version hashed the song name into the seed so a column would not be
one roll repeated down the grid. It reads better and measures nothing: with the
seed varying on both axes the two scores are the same comparison, and they came
back within 0.001 of each other for every scene tried. A column holds its seed
fixed now, which is what makes it attributable to the music — and the scores
separate, with the seed moving every scene tried more than the song does.

170 cells in 9s. The songs are the slow part and are analysed one at a time so
the count moves, then cached for the session.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Dejvino
2026-08-19 17:16:15 +02:00
co-authored by Claude Opus 5
parent 89277705c4
commit 0a89676cc8
3 changed files with 511 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
flow-state/src/checks/scene-sweep.js
+241
View File
@@ -0,0 +1,241 @@
// One scene, across a wide grid of songs and seeds.
//
// The gallery answers "which scenes repeat themselves" by showing every scene
// six times. This answers the question you have immediately afterwards, which it
// cannot: WHY does this one repeat, and what actually moves it?
//
// Six thumbnails is enough to rank sixty-eight scenes and far too few to study
// one. Six songs at one seed each confounds the two inputs — a scene that looks
// the same six times might be ignoring the song, or ignoring the seed, and the
// gallery cannot tell you which. So this separates them onto the two axes of a
// grid:
//
// rows the SONG — different audio, so different features, sections,
// biases and sampled parameter centres.
// columns the SEED — same audio, so the same everything except the roll:
// a different identity, palette and parameter draw.
//
// A row that is uniform means the seed does nothing for this scene. A column
// that is uniform means the song does nothing. A grid that is uniform means the
// scene is a constant, and the diagonal being varied while the rows are flat is
// the shape you get when the palette is doing all the work.
//
// Everything is a real generator output — the same generateLook the exporter
// runs — so the grid shows what would actually be produced, with only the scene
// held fixed instead of chosen.
import { Rng, hashString } from '../engine/rng.js';
import { featureProviderFor } from '../audio/FeatureTrack.js';
import { song, SONGS } from '../audio/songbank.js';
import { generateLook } from '../look/LookGenerator.js';
import { describeIdentity } from '../look/Identity.js';
import { sampleValues } from '../params/schema.js';
import { frameDescriptor, motionDescriptor } from './variety/descriptors.js';
import { descriptorDistance, STRUCTURAL } from './variety/signature.js';
import { THUMB } from './gallery.js';
/** How many songs the bank can offer. The grid cannot ask for more. */
export const MAX_SONGS = SONGS.length;
/** Grid sizes the buttons offer, widest last. */
export const SWEEP_SIZES = [
{ label: '4 × 4', songs: 4, seeds: 4 },
{ label: '6 × 6', songs: 6, seeds: 6 },
{ label: '8 × 8', songs: 8, seeds: 8 },
{ label: '12 × 8', songs: 12, seeds: 8 },
{ label: 'every song × 10', songs: MAX_SONGS, seeds: 10 },
];
// Analysing a song is the slow part of building a grid — seconds each, against
// milliseconds for a look and a render. Re-rendering with a different seed count
// or a different scene must not pay for it twice, so tracks are held for the
// life of the page.
const trackCache = new Map();
/** Which bank entries a grid of this width uses — the spread songBank would pick. */
function specsFor(count) {
const n = Math.min(count, SONGS.length);
if (n >= SONGS.length) return SONGS;
return Array.from({ length: n }, (_, i) => SONGS[Math.round((i * (SONGS.length - 1)) / (n - 1))]);
}
/**
* Analyse the songs a grid needs, one at a time, reporting as each lands.
*
* Analysis is seconds per song and everything after it is milliseconds, so the
* widest grid spends most of its wall time here. Done in one call it froze the
* page for the better part of a minute with a status line nobody could see
* update; one song per turn of the event loop costs nothing and keeps the count
* moving. Cached for the life of the page, so only the first grid pays.
*/
export async function prepareSongs(count, onProgress) {
const specs = specsFor(count);
const tracks = [];
for (let i = 0; i < specs.length; i++) {
const name = specs[i].name;
if (!trackCache.has(name)) trackCache.set(name, song(name));
tracks.push(trackCache.get(name));
if (onProgress) onProgress(i + 1, specs.length, name);
await new Promise((r) => setTimeout(r, 0));
}
return tracks;
}
function tracksFor(count) {
return specsFor(count).map((s) => {
if (!trackCache.has(s.name)) trackCache.set(s.name, song(s.name));
return trackCache.get(s.name);
});
}
/**
* The grid's cells, in row-major order: song by song, seed by seed.
*
* `seedOffset` walks the grid onto a fresh set of seeds without changing its
* shape, which is how you check whether a flat-looking scene is flat or merely
* unlucky — ten more draws is a much better answer to that than staring harder
* at the same ten.
*
* @returns {{cells: object[], songNames: string[], seedLabels: string[]}}
*/
export function sweepGrid({ songs = 6, seeds = 6, seedOffset = 0 } = {}) {
const bank = tracksFor(Math.min(songs, MAX_SONGS));
const cells = [];
const seedLabels = [];
for (let s = 0; s < seeds; s++) seedLabels.push(`#${seedOffset + s}`);
for (const entry of bank) {
for (let s = 0; s < seeds; s++) {
// The SAME seed all the way down a column, deliberately.
//
// The first version hashed the song name in as well, so that column
// zero would not be one roll repeated down the grid. That reads
// better and measures nothing: with the seed varying on both axes,
// "across songs" and "across seeds" are the same comparison, and
// measured, they came back within 0.001 of each other for every
// scene tried — two numbers that could never disagree. Holding the
// seed fixed down a column is what makes that column attributable
// to the music.
//
// The identity still differs down a column, because it is derived
// from the track's summary as well as the seed. That is the point:
// the difference that survives a fixed roll is the song's doing.
const seed = hashString(`sweep:${seedOffset + s}`);
const look = generateLook(entry.track, { seed });
// The busiest section: where the scene is asked for the most, and
// where two draws are likeliest to converge on the same picture.
const section = look.sections.reduce(
(best, x) => (x.bias.energy > best.bias.energy ? x : best), look.sections[0]);
cells.push({
name: `${entry.name}${seedLabels[s]}`,
song: entry.name,
seedIndex: seedOffset + s,
seed,
track: entry.track,
palette: look.palette,
personality: look.personality,
bias: section.bias,
frame: section.startFrame + Math.floor((section.endFrame - section.startFrame) * 0.5),
identity: describeIdentity(look.personality.identity),
});
}
}
return { cells, songNames: bank.map((e) => e.name), seedLabels };
}
/**
* Render one scene over every cell, reporting each as it lands.
*
* Progressive on purpose: a wide grid is a hundred and seventy renders plus the
* audio analysis in front of it, and a page that shows nothing until the end of
* that reads as broken.
*/
export async function sweepScene({ engine, module, cells, onCell }) {
const thumbs = [];
const descriptors = [];
for (let i = 0; i < cells.length; i++) {
const ctx = cells[i];
engine.timeline.setDuration(ctx.track.duration);
engine.setFeatureProvider(featureProviderFor(ctx.track));
// Params come from the cell's own draw, exactly as the generator would
// sample them: the song's section bias and the seed's temperament.
const rng = new Rng(ctx.seed ^ hashString(module.name));
const params = sampleValues(module, rng, ctx.bias, ctx.personality.temperament);
engine.setLayerSpecs([{
module, params, seed: rng.int(0, 0x7fffffff),
opacity: 1, blend: 'normal',
palette: ctx.palette, personality: ctx.personality,
}]);
engine.compositor.reset();
for (let f = ctx.frame - 6; f < ctx.frame; f++) engine.renderFrame(f);
const pixels = Uint8Array.from(engine.readPixels(engine.renderFrame(ctx.frame)));
const moved = Uint8Array.from(engine.readPixels(engine.renderFrame(ctx.frame + 5)));
thumbs.push(pixels);
const still = frameDescriptor(pixels, THUMB.width, THUMB.height);
const motion = motionDescriptor(pixels, moved, THUMB.width, THUMB.height);
descriptors.push({ ...still, motion: motion.scale.concat(motion.layout) });
if (onCell) await onCell(i, cells.length, pixels);
}
return { thumbs, ...scoreGrid(descriptors, cells) };
}
/**
* Score the grid three ways, because one number cannot say what is wrong.
*
* overall every cell against every other — comparable to the gallery's
* variety score, on the same structural descriptor.
* bySeed cells of the SAME song, different seeds. Low means the seed does
* nothing here: the scene ignores the identity.
* bySong cells at the same seed index, different songs. Low means the song
* does nothing: the scene ignores the music.
*
* Those two are the diagnosis the gallery's single number cannot give, and they
* point at different fixes — a flat seed axis is a scene not reading the cast,
* a flat song axis is a scene not reading its features or its bias.
*/
function scoreGrid(descriptors, cells) {
const byBlock = {};
let total = 0, pairs = 0;
let seedSum = 0, seedPairs = 0;
let songSum = 0, songPairs = 0;
let closest = null;
for (let i = 0; i < descriptors.length; i++) {
for (let j = i + 1; j < descriptors.length; j++) {
const d = descriptorDistance(descriptors[i], descriptors[j]);
const structural = STRUCTURAL.reduce((a, b) => a + (d[b] || 0), 0) / STRUCTURAL.length;
for (const b of [...STRUCTURAL, 'colour']) byBlock[b] = (byBlock[b] || 0) + (d[b] || 0);
total += structural;
pairs++;
if (cells[i].song === cells[j].song) { seedSum += structural; seedPairs++; }
if (cells[i].seedIndex === cells[j].seedIndex) { songSum += structural; songPairs++; }
// The two cells that are most alike. On a wide grid this is the only
// practical way to find the pair worth looking at — a hundred and
// seventy thumbnails is past what an eye will compare.
if (!closest || structural < closest.distance) {
closest = { distance: structural, a: i, b: j };
}
}
}
for (const b of Object.keys(byBlock)) byBlock[b] /= pairs || 1;
return {
variety: pairs ? total / pairs : 0,
bySeed: seedPairs ? seedSum / seedPairs : 0,
bySong: songPairs ? songSum / songPairs : 0,
byBlock,
closest,
};
}