Walk a world from a single image

In this first lesson, we're going to build a walkable world from a single image. Unlike the diffusion models in earlier chapters, ABot-World doesn't render a fixed clip. Instead, it starts from a photo, and each worldStep call extends the walk in whatever direction the keys point.

The model load needs three side files alongside the main DiT checkpoint: t5XxlModelSrc encodes the prompt, taehvModelSrc decodes every step's frames, and vaeModelSrc converts the first frame into latents once, at scene creation. The two VAEs are easy to swap by name. A swapped pair still loads without error, and only fails once the walk starts.

const modelId = await loadModel({
  modelSrc: ABOT_WORLD_0_5B_Q8_0,
  modelType: "sdcpp-generation",
  modelConfig: {
    mode: "world",
    taehvModelSrc: ABOT_WORLD_0_5B_LF_VAE,
    t5XxlModelSrc: UMT5_XXL_ENC_Q8_0,
    vaeModelSrc: ABOT_WORLD_0_5B_LF_VAE_F16,
    world: { seed: 42, kvCache: true },
  },
});

worldCreateScene turns a still image and a prompt into a walkable scene. The scene stays live on the session, so awaiting stats is enough to know it's ready:

const creation = worldCreateScene({
  modelId,
  prompt: "A realistic outdoor world scene with a navigable path.",
  image: firstFrame,
  width: 832,
  height: 480,
});
await creation.stats;
console.log("Scene created");

worldStep({ modelId, keys }) walks the scene one block at a time. Pass keys like ['W', 'L'] to move forward while turning left, or [] to idle. frameStream delivers that block's decoded frames as they're produced:

const tape: string[][] = [["W"], ["W", "L"], ["W"], []];
for (const keys of tape) {
  const { frameStream } = worldStep({ modelId, keys });
  for await (const frame of frameStream) {
    fs.writeFileSync(`output/abot-world/frame-${String(frameNumber++).padStart(4, "0")}.jpg`, frame);
  }
}
console.log(`Walked ${tape.length} blocks`);

That's why the loop above runs sequentially. The SDK rejects a second worldStep request made before the previous block's frameStream finishes.

Note: worldStep itself can also be cancelled mid-block with cancel({ requestId }). Cancellation is block-granular, since the engine finishes the current block internally regardless. The cancel either arrives in time or loses the race to a block that already completed, and both outcomes are correct.