Stop the voice assistant from hearing itself

The voice assistant from the previous lesson works for a turn or two, then the feedback loop takes over: the TTS output is picked up by the mic, Whisper transcribes it as a new user turn, and the LLM answers it. Each turn triggers the next with no real user input.

Four things drive that loop:

• The default VAD commits segments too eagerly for long-running use.
• The mic keeps recording through TTS, so the TTS output comes back as a new user turn.
• VAD occasionally commits phantom transcripts from near-silence, single tokens like "you" or "Thanks." with no real speech behind them.
• The TTS audio rings through the speaker into the mic for a moment after the model finishes, getting transcribed as the tail of the user's next turn.

The defaults are tuned for one-shot dictation, not a loop, so the first fix is to override vad_params in the ASR modelConfig:

const vad_params = {
  threshold: 0.6,
  min_speech_duration_ms: 300,
  min_silence_duration_ms: 700,
  max_speech_duration_s: 15.0,
  speech_pad_ms: 200,
};

min_silence_duration_ms: 700 is the value that matters. VAD uses it to decide when the user stopped talking, so a longer quiet window keeps the TTS tail ringing through the speaker from getting folded into the user's turn.

VAD handles the first issue. The other three happen in the loop body, so the fix is three helpers defined right before the main loop:

• isSpeaking flag: the loop checks it at the top of each iteration to skip frames while TTS plays
• isMeaningfulTranscript: filter that drops the empty and phantom transcripts before they reach the LLM
• sleep(ms) helper: the main loop calls it to wait for the post-playback cooldown.

Here's how that looks in code:

const POST_PLAYBACK_COOLDOWN_MS = 300;
const MIN_UTTERANCE_CHARS = 3;

function isMeaningfulTranscript(text: string): boolean {
  const trimmed = text.trim();
  if (trimmed.length === 0) return false;
  if (trimmed.includes("[No speech detected]")) return false;
  if (/^\[[^\]]+\]$/.test(trimmed)) return false;
  const letters = trimmed.replace(/[^\p{L}\p{N}]/gu, "");
  return letters.length >= MIN_UTTERANCE_CHARS;
}

function sleep(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

let isSpeaking = false;

The mic side gets a new gate. The previous lesson's data handler pipes every frame into the session; this one drops frames while the assistant is talking so the speaker output never reaches Whisper:

ffmpeg.stdout.on("data", (chunk: Buffer) => {
  if (isSpeaking) return;
  session.write(chunk);
});

The session is the async iterable; transcribeStream({ modelId }) returns a Promise<session>, so you await it before the for await. The loop checks isSpeaking and isMeaningfulTranscript at the top of each iteration to skip frames, and the try/finally around the LLM+TTS block flips isSpeaking back to false even if the LLM throws, so a crashed turn doesn't leave the mic muted:

const session = await transcribeStream({ modelId: asrModelId });
for await (const rawText of session) {
  if (isSpeaking) continue;
  if (!isMeaningfulTranscript(rawText)) continue;
  const userText = rawText.trim();

  history.push({ role: "user", content: userText });

  isSpeaking = true;
  try {
    const llmResult = completion({ modelId: llmModelId, history, stream: true });
    let assistantText = "";
    for await (const token of llmResult.tokenStream) {
      process.stdout.write(token);
      assistantText += token;
    }
    history.push({ role: "assistant", content: assistantText });

    const spoken = assistantText.trim();
    if (spoken.length > 0) {
      const ttsResult = textToSpeech({
        modelId: ttsModelId,
        text: spoken,
        inputType: "text",
        stream: false,
      });
      const samples = await ttsResult.buffer;
      if (samples.length > 0) {
        const wavBuffer = Buffer.concat([
          createWavHeader(samples.length * 2, TTS_SAMPLE_RATE),
          int16ArrayToBuffer(samples),
        ]);
        playAudio(wavBuffer);
      }
      await sleep(POST_PLAYBACK_COOLDOWN_MS);
    }
  } finally {
    isSpeaking = false;
  }
}

Two tuning knobs to revisit if the loop still misbehaves. If VAD commits segments while the user is still talking, raise min_silence_duration_ms. If VAD commits segments out of near-silence, raise threshold to 0.7.

Note: the isSpeaking flag drops the transcripts after Whisper processes them, but the mic is still recording the whole time. Pausing the ffmpeg pipe would let the buffer pile up, so we keep the pipe draining and drop the transcripts in software. The trade-off is a little extra VAD work on audio we'll throw away, in exchange for never stalling on a full buffer.

The SDK installs its own SIGINT / SIGTERM handler that aborts in-flight RPC streams on shutdown. The abort rejects any pending session.write() or unloadModel() call with a WorkerShutdownError. The bare-rpc socket also emits an RPCError with code: 'CHANNEL_CLOSED' on the same teardown path. Both surface as unhandled stream errors. The process.on("uncaughtException", ...) filter ignores the shutdown noise and re-throws anything else. The lesson scopes its Sets to avoid shadowing the runner preamble's globals of the same name:

const LESSON_TEARDOWN_NAMES = new Set([
  "WorkerShutdownError", "WorkerCrashedError", "BareRuntimeBinaryNotFoundError",
  "InferenceCancelledError", "TranscriptionFailedError", "TranslationFailedError",
  "TextToSpeechFailedError", "TextToSpeechStreamFailedError", "AbortError",
]);
const LESSON_TEARDOWN_CODES = new Set([
  "ABORT_ERR", "CHANNEL_CLOSED", "MODEL_NOT_LOADED", "MODEL_WAS_UNLOADED",
  "WORKER_SHUTDOWN", "RPC_CONNECTION_FAILED",
]);
function isTeardown(err) {
  if (!err) return true;
  if (LESSON_TEARDOWN_NAMES.has((err.name || "").toString())) return true;
  const code = (err.code || "").toString();
  if (LESSON_TEARDOWN_CODES.has(code)) return true;
  const msg = (err.message || String(err) || "").toString();
  if (/\bis shutting down\b/i.test(msg)) return true;
  if (/\bin-flight rpc\b/i.test(msg)) return true;
  return false;
}
process.on("uncaughtException", (err) => {
  if (isTeardown(err)) return;
  throw err;
});

Inside the turn body, the same teardown error families fire whenever an RPC drains after unloadModel. So wrap each per-turn SDK call in a try { ... } catch (err) { if (!isTeardown(err)) ... } and skip the rest of the turn, otherwise a single Stop spawns dozens of Transcription failed: Model X unloaded lines.