Generate image with img2img

Now that we can do txt2img, we're going to learn img2img.

With txt2img, the model by default starts from noise. With img2img, the model starts from an image we supply and iteratively refines it until the result matches our prompt.

init_image is a Uint8Array of PNG or JPEG bytes. We read it off disk with fs.readFileSync. The strength number tells the model how much room it has. At 0 it keeps the source image untouched, and at 1 it ignores it and behaves like txt2img.

Common uses include rough sketch to colored illustration, screenshot to wireframe turned into a design mock, or an existing photo restyled.

Reading the source image into a Uint8Array hands the bytes to diffusion(). It uses the same single-file loader as Stable Diffusion, since SD supports init_image directly:

const modelId = await loadModel({
  modelSrc: SD_V2_1_1B_Q8_0,
  modelType: "sdcpp-generation",
  modelConfig: { prediction: "v" },
});
const initImage = fs.readFileSync("./examples/qvac/image-generation/input/sketch.png");

Calling diffusion() with init_image set to the source bytes is the img2img path. The four-knob call we'd run looks like:

const result = diffusion({
  modelId,
  prompt: "an oil painting of a fox in a snowy forest",
  init_image: initImage,
  strength: 0.6,
  width: 512,
  height: 512,
  steps: 25,
});
const outputs = await result.outputs;

Once the awaited PNG is in hand, write outputs[0] and log the count:

const first = outputs[0];
if (!first) throw new Error("No image returned from diffusion");
fs.writeFileSync("output/image-generation/fox-painting.png", first);
console.log(`Generated ${outputs.length} image`);

We tweak strength until the balance between "preserves the source" and "rewrites everything" feels right.

Note: init_image dimensions are the lower bound. The output resolution is governed by width and height, not the source size.