P2P Academy

Now that we've seen speech-to-text, text generation, and text-to-speech separately, we're going to put them together.

A voice assistant is a loop: listen, transcribe, answer, speak. Each piece lives in a chapter we've already done. This lesson wires the three pieces into a streaming conversation.

Three models to load: ASR with Whisper + Silero VAD, LLM with Llama 3.2 1B, TTS with Supertonic English. You would load them in this sequence:

const asrModelId = await loadModel({
  modelSrc: WHISPER_TINY,
  modelConfig: {
    vadModelSrc: VAD_SILERO_5_1_2,
    audio_format: "f32le",
    language: "en",
  },
});
const llmModelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  modelConfig: { ctx_size: 4096 },
});
const ttsModelId = await loadModel({
  modelSrc: TTS_EN_SUPERTONIC_Q8_0,
  modelConfig: {
    ttsEngine: "supertonic",
    language: "en",
    voice: "F1",
    ttsSpeed: 1.05,
    ttsNumInferenceSteps: 5,
  },
});

The system prompt is what tells the model how to behave: short answers (TTS takes ~1s per sentence), no markdown (asterisks sound bad read aloud), no lists (hard to follow mid-task). The system prompt might look as follows:

const history: Array<{
  role: "system" | "user" | "assistant";
  content: string;
}> = [{ role: "system", content: SYSTEM_PROMPT }];

transcribeStream({ modelId }) returns a Promise<TranscribeStreamSession>. await it to get the session, then iterate with for await (const rawText of session). The session emits plain text strings.

Audio capture is a child ffmpeg process. The startMicrophone() helper spawns the system ffmpeg with the right -i args for the current platform (avfoundation on macOS, pulse on Linux, dshow on Windows) and pipes 16 kHz mono f32le PCM to stdout. Each chunk on stdout is a frame we feed into the session:

const ffmpeg = startMicrophone({ sampleRate: 16000, format: "f32le" });
const session = await transcribeStream({ modelId: asrModelId });

ffmpeg.stdout.on("data", (chunk: Buffer) => {
  session.write(chunk);
});

A startup check on ffmpeg -version and ffplay -version makes the failure mode obvious if either is missing. The for (const tool of ['ffmpeg', 'ffplay']) loop keeps it to a few lines.

The main loop is one async for await over the session. Each iteration produces a user turn, runs the LLM, then speaks the answer:

for await (const rawText of session) {
  const userText = rawText.trim();
  if (userText.length === 0) continue;

  history.push({ role: "user", content: userText });

  const llmResult = completion({
    modelId: llmModelId,
    history,
    stream: true,
  });
  let assistantText = "";
  for await (const token of llmResult.tokenStream) {
    process.stdout.write(token);
    assistantText += token;
  }
  history.push({ role: "assistant", content: assistantText });

  const ttsResult = textToSpeech({
    modelId: ttsModelId,
    text: assistantText.trim(),
    inputType: "text",
    stream: false,
  });
  const samples = await ttsResult.buffer;
  if (samples.length > 0) {
    const wavBuffer = Buffer.concat([
      createWavHeader(samples.length * 2, TTS_SAMPLE_RATE),
      int16ArrayToBuffer(samples),
    ]);
    playAudio(wavBuffer);
  }
}

textToSpeech() returns the audio as raw 16-bit signed PCM samples at 44.1 kHz mono. To play it we wrap the samples in a minimal WAV header and pipe the buffer into ffplay, which ships with ffmpeg.

A cleanup() handler kills the ffmpeg child, ends the session, and unloads all three models. Wire it to SIGINT and SIGTERM so Ctrl+C exits cleanly:

async function cleanup() {
  if (shuttingDown) return;
  shuttingDown = true;
  ffmpeg.kill();
  try { session.end(); } catch {}
  await unloadModel({ modelId: ttsModelId }).catch(() => {});
  await unloadModel({ modelId: llmModelId }).catch(() => {});
  await unloadModel({ modelId: asrModelId }).catch(() => {});
  process.exit(0);
}

process.on("SIGINT", () => void cleanup());
process.on("SIGTERM", () => void cleanup());

The SDK installs its own SIGINT / SIGTERM handler that aborts in-flight RPC streams on shutdown. The abort rejects any pending session.write() or unloadModel() call with a WorkerShutdownError. The bare-rpc socket also emits an RPCError with code: 'CHANNEL_CLOSED' on the same teardown path. Both surface as unhandled stream errors. The process.on("uncaughtException", ...) filter ignores the shutdown noise and re-throws anything else. The lesson scopes its Sets to avoid shadowing the runner preamble's globals of the same name:

const LESSON_TEARDOWN_NAMES = new Set([
  "WorkerShutdownError", "WorkerCrashedError", "BareRuntimeBinaryNotFoundError",
  "InferenceCancelledError", "TranscriptionFailedError", "TranslationFailedError",
  "TextToSpeechFailedError", "TextToSpeechStreamFailedError", "AbortError",
]);
const LESSON_TEARDOWN_CODES = new Set([
  "ABORT_ERR", "CHANNEL_CLOSED", "MODEL_NOT_LOADED", "MODEL_WAS_UNLOADED",
  "WORKER_SHUTDOWN", "RPC_CONNECTION_FAILED",
]);
function isTeardown(err) {
  if (!err) return true;
  if (LESSON_TEARDOWN_NAMES.has((err.name || "").toString())) return true;
  const code = (err.code || "").toString();
  if (LESSON_TEARDOWN_CODES.has(code)) return true;
  const msg = (err.message || String(err) || "").toString();
  if (/\bis shutting down\b/i.test(msg)) return true;
  if (/\bin-flight rpc\b/i.test(msg)) return true;
  return false;
}
process.on("uncaughtException", (err) => {
  if (isTeardown(err)) return;
  throw err;
});

Each teardown-aware RPC also sits in the body of the loop, so guard each call against the shuttingDown flag and catch per-turn errors, otherwise a flush that fires after unloadModel raises Transcription failed: Model X unloaded for every drained segment:

for await (const rawText of session) {
  if (shuttingDown) break;
  // ...
  try {
    for await (const token of llmResult.tokenStream) {
      if (shuttingDown) break;
      process.stdout.write(token);
      assistantText += token;
    }
  } catch (err) {
    if (!isTeardown(err)) console.error("✖ LLM:", err.message);
    break;
  }
}

Note: the system prompt bans markdown and lists because the output is spoken aloud. A ### heading or a 1. list reads as a stuttery mess through TTS.

Build a real-time voice assistant loop

Questions