Build a video from a still image

The previous lesson built a video from text. This one builds one from a still image.

txt2vid builds the clip from text. img2vid starts from a still and animates it forward. The first frame becomes the input; the rest of the clip is generated from the prompt.

The still comes in via init_image. strength controls how much motion happens, and at 0 we get a frozen frame, and at 1 we lose all resemblance to the source. Most clips sit somewhere around 0.6 to 0.85.

For img2vid, the model needs a vision encoder alongside the diffusion model. WAN2_1_I2V_14B_Q4_K_M is the I2V model, with CLIP_VISION_H as the vision encoder:

The I2V model needs a vision tower as a side file (CLIP-style encoder), in the same slot T2V uses for T5-XXL. You would load it like so:

const videoId = await loadModel({
  modelSrc: WAN2_1_I2V_14B_Q4_K_M,
  modelType: "sdcpp-generation",
  modelConfig: {
    mode: "video",
    t5XxlModelSrc: UMT5_XXL_FP16,
    vaeModelSrc: WAN_2_1_COMFYUI_REPACKAGED_VAE,
    clipVisionModelSrc: CLIP_VISION_H,
  },
});

init_image is the first frame the diffusion model starts denoising from; the vision tower encodes it, the diffusion model adds motion, and the VAE decodes:

const initImage = new Uint8Array(fs.readFileSync("./examples/qvac/video-generation/input/portrait.png"));

video({ mode: "img2vid", ... }) kicks off the animation; awaiting outputs resolves to the encoded video bytes. Note that init_image is the first frame and strength is how much it can change. The still's pixel dimensions must match width and height exactly: the I2V model rejects the call with init_image dimensions WxH do not match video dimensions WxH otherwise, so size the source image to your target resolution before passing it. Here's an example that reads a portrait PNG, animates it forward, and writes the first clip to disk:

const result = video({
  modelId: videoId,
  mode: "img2vid",
  prompt: "the subject slowly turns and smiles, soft natural lighting, cinematic",
  init_image: initImage,
  strength: 0.85,
  width: 480,
  height: 832,
  video_frames: 17,
  fps: 16,
});
const outputs = await result.outputs;
const firstClip = outputs[0];
if (!firstClip) throw new Error("No video returned from video()");
fs.writeFileSync("output/video-generation/portrait.avi", firstClip);
console.log(`Generated ${outputs.length} video`);

Note: clipVisionModelSrc is the vision encoder that comes alongside the I2V model. Pairing it with the wrong model (or omitting it) means the first frame isn't in the input and the output has nothing to anchor on, so the result drifts frame-to-frame.