Run a GR00T action inference across embodiments

SmolVLA and π₀.₅ both take pixel images and stick to one robot configuration each. This lesson closes out the chapter with the third model, GR00T, which breaks both of those assumptions. Its camera frames arrive pre-patchified, and one GGUF can hold several robot configurations at once.

loadModel still takes modelType: "ggml-vla", but GR00T's GGUF can carry more than one embodiment, a specific robot configuration with its own camera count and action space. Pick the starting one with modelConfig.embodiment:

const modelId = await loadModel({
  modelSrc: GROOT_MULTI_Q8_VF16,
  modelType: "ggml-vla",
  modelConfig: { backend: "cpu", embodiment: "libero_sim" },
});

vlaHparams works the same way it did in the previous two lessons. GR00T also reports selectedEmbodimentTag and the active numCameras, two fields SmolVLA and π₀.₅ don't have.

const { hparams, backendName } = await vlaHparams({ modelId });
console.log(`▸ Backend: ${backendName ?? "(unknown)"}`);
console.log(`▸ Embodiment: ${hparams.selectedEmbodimentTag} (${hparams.numCameras} cameras)`);

GR00T reports hparams.imageInputMode as "patches" instead of "pixels". Each images[] entry is a pre-patchified buffer of hparams.imagePatchElems floats instead of a 3·w·h plane, so vlaPreprocessImage() doesn't apply here.

The state is still continuous, padded like SmolVLA's state was. GR00T's flow-matching step also can't generate its own starting noise the way SmolVLA and π₀.₅ can when noise is left out, so a noise buffer is required rather than optional:

const numCameras = hparams.numCameras ?? 2;
const images = Array.from({ length: numCameras }, () =>
  new Float32Array(hparams.imagePatchElems!).fill(0.02),
);
const state = vlaPadState([0, 0, 0, 0, 0, 0], hparams.maxStateDim);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);

const promptLength = numCameras * (MERGED_TOKENS_PER_IMAGE + 1) + PROMPT_TEXT_TAIL;
const tokens = new Int32Array(promptLength);
let w = 0;
for (let cam = 0; cam < numCameras; cam++) {
  for (let k = 0; k < MERGED_TOKENS_PER_IMAGE; k++) tokens[w++] = IMAGE_TOKEN_ID;
  tokens[w++] = TEXT_TOKEN_ID + cam;
}
for (; w < tokens.length; w++) tokens[w] = TEXT_TOKEN_ID + w;
const mask = new Uint8Array(promptLength).fill(1);

This matches the layout the Qwen3-VL tokenizer produces around each image. A real consumer gets that layout from the tokenizer; this exercise builds it directly instead. With the buffers built, vla() runs exactly as it did in the previous two lessons:

const { actions, actionDim, chunkSize, stats } = await vla({
  modelId,
  images,
  imgWidth: hparams.visionImageSize,
  imgHeight: hparams.visionImageSize,
  state,
  tokens,
  mask,
  noise,
});

console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(Array.from(actions.subarray(0, actionDim)));
if (stats) {
  console.log(
    `▸ Timing: vision=${stats.vision_ms?.toFixed(0)}ms ` +
      `ode=${stats.ode_ms?.toFixed(0)}ms ` +
      `total=${stats.total_ms?.toFixed(0)}ms`,
  );
}

vlaSetEmbodiment is the other feature unique to GR00T. It swaps the active robot configuration by reading roughly 20 MB of the new embodiment's rows out of the loaded GGUF, instead of reloading the whole ~4 GB file, and returns the refreshed hparams for whatever configuration you switched to:

const { hparams: refreshed } = await vlaSetEmbodiment({ modelId, embodiment: "real_r1_pro_sharpa" });
console.log(`▸ Embodiment: ${refreshed.selectedEmbodimentTag} (${refreshed.numCameras} cameras)`);

real_r1_pro_sharpa has six camera views versus libero_sim's two, so numCameras changes after the switch. Any inference run against the new embodiment needs buffers rebuilt from refreshed, since it reflects the active embodiment now.

Note: despite the brand name, GR00T doesn't need NVIDIA hardware specifically. @qvac/vla-ggml accelerates on Vulkan (Linux, Windows, Android), Metal (Apple), and OpenCL (Qualcomm Adreno), falling back to CPU everywhere else. Setting modelConfig.backend to "cpu" pins this example to a predictable baseline; a GPU on any of those backends speeds this up too.

Note: vlaSetEmbodiment is rejected on a single-embodiment GGUF like GROOT_Q8_VF16, and while a previous vla() response is still pending. Only the GROOT_MULTI_* constants support switching at all.