P2P Academy

When a single audio file has more than one speaker, we want a transcript that says who said what. Sortformer diarizes the audio into speaker-attributed segments, then TDT transcribes each segment into text.

The process runs in two passes:

Sortformer returns a text block where each line is Speaker N: hh:mm:ss.mmm - hh:mm:ss.mmm.
TDT transcribes each segment and we attach the text to the speaker metadata.

Each Speaker N: hh:mm:ss.mmm - hh:mm:ss.mmm line becomes a { speaker, start, end } object, with both stamps converted to seconds for the slicing below. The Sortformer-to-TDT chain starts with the diarization pass. The first pass would look like the following:

const sfModelId = await loadModel({
  modelSrc: PARAKEET_SORTFORMER_4SPK_V2_1_Q8_0,
  modelType: "parakeet-transcription",
});

const diarization = await transcribe({
  modelId: sfModelId,
  audioChunk: audioFilePath,
});
await unloadModel({ modelId: sfModelId });

function toSeconds(stamp: string): number {
  return stamp
    .replace(/s$/, "")
    .split(":")
    .map(Number)
    .reduce((total, part) => total * 60 + part, 0);
}

const segments = diarization
  .split("\n")
  .map((line) => line.match(/Speaker (\d+): ([\d:.]+s?) - ([\d:.]+s?)/))
  .filter((m): m is RegExpMatchArray => m !== null)
  .map((m) => ({ speaker: +m[1]!, start: toSeconds(m[2]!), end: toSeconds(m[3]!) }))
  .sort((a, b) => a.start - b.start);

Second pass: TDT takes over. The slicing reads the source WAV once and writes a new WAV per start/end range; transcribe consumes each slice:

const tdtModelId = await loadModel({
  modelSrc: PARAKEET_TDT_0_6B_V3_Q8_0,
  modelType: "parakeet-transcription",
});

const pcm = readPcm(audioFilePath);
const sliceDir = join(tmpdir(), `qvac-diarize-${Date.now()}`);
mkdirSync(sliceDir, { recursive: true });

const results: { speaker: number; start: number; end: number; text: string }[] = [];
for (let i = 0; i < segments.length; i++) {
  const seg = segments[i]!;
  const slicePath = join(sliceDir, `seg-${i}.wav`);

  if (!writeWavSlice(pcm, seg.start, seg.end, slicePath)) {
    results.push({ ...seg, text: "[No speech detected]" });
    continue;
  }

  const text = await transcribe({
    modelId: tdtModelId,
    audioChunk: slicePath,
  });
  results.push({ ...seg, text: text.trim() || "[No speech detected]" });
}
await unloadModel({ modelId: tdtModelId });

One log line per diarized result, with the speaker label and time range:

for (const r of results) {
  console.log(`Speaker ${r.speaker} (${r.start.toFixed(2)}s - ${r.end.toFixed(2)}s): ${r.text}`);
}

readPcm and writeWavSlice (prefilled in the scaffold) handle the WAV byte-level work: skipping the 44-byte header and rebuilding a 16 kHz / 16-bit / mono header in front of each PCM range. The slicing runs once per diarized segment, so the inference cost is N × TDT_inference rather than one shot on the full file.

Note: Sortformer v2.1 supports up to 4 speakers. For more speakers, split the audio or run the diarization in overlapping windows.

Diarize speakers with Parakeet Sortformer

Questions