Run a SmolVLA action inference

VLA (vision-language-action) models are the model class for robot control. They take a camera frame and a task description and produce an action chunk the robot should take.

The training data covers real-world manipulation tasks: grasping, picking, placing. The SDK includes two families: SmolVLA (a small, fast model from Hugging Face) and π₀.₅ (a larger model from Physical Intelligence). Both share the same API surface but the input sizes differ.

SmolVLA is the smaller of the two VLAs in the SDK. Load it into memory like this:

const modelId = await loadModel({
  modelSrc: SMOLVLA_LIBERO_VISION_Q8,
  modelType: "ggml-vla",
  modelConfig: { backend: "cpu" },
});

Each VLA model has different input sizes, so read hparams before building any buffers:

const { hparams } = await vlaHparams({ modelId });

The hparams tell you the input sizes. For SmolVLA, you need two camera frames of visionImageSize × visionImageSize × 3, a state vector padded to maxStateDim, tokens of length tokenizerMaxLength, and a noise buffer of length chunkSize × maxActionDim.

SmolVLA wants state as a separate field, unlike π₀.₅ which tokenises it. The synthetic inputs sized to hparams would look like:

const size = hparams.visionImageSize;
const dummyPixels = new Uint8Array(size * size * 3).fill(128);
const front = vlaPreprocessImage(dummyPixels, size, size, { size });
const wrist = vlaPreprocessImage(dummyPixels, size, size, { size });

const tokens = new Int32Array(hparams.tokenizerMaxLength);
const mask = new Uint8Array(hparams.tokenizerMaxLength);
tokens[0] = 1;
mask[0] = 1;

const state = vlaPadState([0, 0, 0, 0, 0, 0], hparams.maxStateDim);
const noise = new Float32Array(hparams.chunkSize * hparams.maxActionDim);

vla() takes the model id and the prebuilt inputs, and returns an action chunk and per-stage timings:

const { actions, actionDim, chunkSize, stats } = await vla({
  modelId,
  images: [front, wrist],
  imgWidth: size,
  imgHeight: size,
  state,
  tokens,
  mask,
  noise,
});

console.log(`▸ Got ${chunkSize} action steps of dim ${actionDim}.`);
console.log(`▸ Timing: vision=${stats.vision_ms}ms prefill=${stats.prefill_total_ms}ms ode=${stats.ode_ms}ms total=${stats.total_ms}ms`);

The stats object has per-stage timings: vision encoder, language model, ODE solver, and the wall-clock total. They're useful for spotting which stage is the bottleneck on a given machine.

Note: vlaPadState is the helper that pads a short state vector to the model's maxStateDim. For SmolVLA, the state is six floats (end-effector pose). For π₀.₅ the state is tokenised into the prompt instead, the state buffer is ignored.

Note: vlaSetEmbodiment switches a loaded GR00T multi-embodiment model between robot configurations, such as vlaSetEmbodiment({ modelId, embodiment: "real_r1_pro_sharpa" }). SmolVLA and π₀.₅ reject it. See the GR00T example when you need to change embodiments.