Pause, resume, and cancel a fine-tune

Now that we've started a fine-tune, we're going to see how to control it.

A fine-tune takes minutes to hours. We don't want to wait around to find out the loss is exploding. The QVAC SDK exposes a small operation surface on finetune() itself:

• pause saves the current state, stops the trainer, and resolves a promise we can await
• resume picks up from the latest checkpoint, the same call signature as the original finetune(), just with operation: "resume"
• cancel is the hard kill: it frees GPU memory immediately and resolves with { status: "CANCELLED" }.

The worker can only run one fine-tune at a time, so each control call has to reach the worker while the prior run is alive (the pause) and can only fire after it has fully ended (the resume). finetune() returns a handle with a progressStream we iterate, and a result promise that resolves when the run ends. We need both.

The setup mirrors the SDK's llamacpp-finetune example. The finetuneParams wrapper keeps the model + options together, so the resume spreads it and adds operation: "resume":

const finetuneParams = { modelId, options: baseOptions };
const handle = finetune(finetuneParams);

The progress stream runs in an IIFE so the resume after the awaits sees a worker slot that's free. We fire the pause from a callback inside the loop, then wait for the run and the stream to drain before resuming:

let pauseRequested = false;
let pauseResultPromise;
const progressTask = (async () => {
  for await (const tick of handle.progressStream) {
    // 1: pause from a callback
  }
})();

const initialResult = await handle.result;
await progressTask;

Now the three control calls. Pause, fire from a callback once training is rolling so the trainer sees it before the run ends:

if (!pauseRequested && tick.global_steps >= 4) {
  pauseRequested = true;
  pauseResultPromise = finetune({ operation: "pause", modelId });
}

Resume, same params + operation: "resume", after await handle.result and await progressTask confirm the worker slot is free:

if (initialResult.status === "PAUSED") {
  const resumed = finetune({ ...finetuneParams, operation: "resume" });
  await resumed.result;
  console.log("▸ Resumed status: COMPLETED");
}

Cancel, same as pause but synchronous, returns the final status:

const cancelResult = await finetune({ operation: "cancel", modelId });
console.log("▸ Cancelled status:", cancelResult.status);

Pause writes a checkpoint under checkpointSaveDir and resume consumes it, so the pair costs us one checkpoint on disk for as long as the run stays paused. Cancel clears it too.

There is also checkpointSaveSteps: N, which writes a checkpoint every N steps as a safety net for a long run. We leave it off here. Each periodic checkpoint holds optimizer state on top of the adapter weights, every run writes to the same directory, and neither the SDK nor the app ever deletes one. Turn it on for real training and budget the disk for it.

Note: pause and cancel are both fire-and-forget at the SDK level. We await them to confirm the operation completed, but the returned promise resolves as soon as the trainer acknowledges the request.