Generate music from a caption

Welcome to the music chapter. In this first lesson, we're going to call ACE-Step and write a .wav. The text chapters took prompts and returned tokens; this one takes a description of a sound and returns audio.

ACE-Step is a four-stage bundle under a single modelId. A caption first runs through a text encoder (Qwen3 Embedding 0.6B) to become a sequence of tokens. The ACE-Step 5Hz LM then schedules those tokens across time, deciding what happens at each second of the clip. The V15 Turbo DiT takes that schedule and denoises it into audio latents, which takes several seconds for a short clip. A VAE finally decodes the latents back into raw PCM. The audiogen modelType is the one key the SDK uses to load all four together:

const modelId = await loadModel({
  modelType: "audiogen",
  modelConfig: {
    textEncModelSrc: AUDIOGEN_QWEN3_EMBEDDING_0_6B_Q8_0,
    lmModelSrc: AUDIOGEN_ACESTEP_5HZ_LM_0_6B_Q8_0,
    ditModelSrc: AUDIOGEN_ACESTEP_V15_TURBO_Q4_K_M,
    vaeModelSrc: AUDIOGEN_VAE_BF16,
    useGPU: true,
    inferenceSteps: 8,
  },
});

The slow stage is the V15 Turbo DiT. It does eight denoising passes through a transformer every time you call audioGen. useGPU: true runs those passes on the GPU; without it the SDK runs them on the CPU instead.

The turbo default is eight inference steps: lower is faster and noisier, higher is slower and cleaner. With the four checkpoints loaded under one modelId, audioGen hands the caption to ACE-Step and gets back a run with progress, audio, and stats:

const run = audioGen({
  modelId,
  caption: "Lo-fi hip hop with mellow piano, soft drums, and a warm bass line",
  lyrics: "[Instrumental]",
  seed: 42,
  duration: 10,
});

audioGen takes four kinds of input:

• caption - a natural-language description of the music
• lyrics: "[Instrumental]" - a string the language model reads; pass real lyrics for a sung vocal track
• seed - an integer pinning the noise so the same caption + seed reproduces the clip
• duration - an integer of seconds the SDK caps at a few tens before quality drops

Between loadModel resolving and the WAV writing to disk, the reader sees nothing. Drain the per-stage ticks while audio and stats resolve in parallel so the terminal moves instead of hanging, for example:

for await (const progress of run.progressStream) {
  console.log(`▸ ${progress.stage}: ${progress.step}/${progress.total}`);
}

const [audio, stats] = await Promise.all([run.audio, run.stats]);

The progress loop walks through stage: step/total lines naming which sub-model is currently working: text encoding, language-model token prediction, DiT diffusion, VAE decoding. Promise.all overlaps the audio and stats awaits so neither one waits on the other; both finish around the same time the diffusion does.

Last step on the way out: run.audio resolves to raw PCM with no RIFF/WAVE header around it, so the file won't play until you add one yourself. You can add it like this:

const wav = createWav(audio.pcm, audio.sampleRate, audio.channels, audio.bitsPerSample);
writeFileSync(outputPath, wav);

const samplesPerChannel = audio.pcm.byteLength / (audio.bitsPerSample / 8) / audio.channels;
console.log(
  `▸ Generated ${samplesPerChannel} samples per channel at ` +
    `${audio.sampleRate} Hz (${audio.channels} channels)`,
);
if (stats) console.log(`▸ Stats: ${JSON.stringify(stats)}`);
console.log(`▸ Saved ${outputPath}`);

await unloadModel({ modelId });

createWav is prefilled in starting.ts; it builds the 44-byte header from the exact audio.sampleRate / audio.channels / audio.bitsPerSample fields the run returns. After that it's a normal writeFileSync and a final unloadModel({ modelId }).