Ingest documents into a workspace

Now that we've built the smallest useful search engine in chapter 3, we're going to scale it up to a workspace that lives on disk.

Real search needs persistence: we build the index once, query it many times. The chapter 3 search kept everything in memory; ragIngest writes the vectors to a folder under the SDK's data directory and reads them back on the next call.

ragIngest({ modelId, documents, workspace, chunk: false }) runs the whole sequence in one call: chunk (skip, in this lesson), embed, save.

Without chunking, each document becomes one entry. This is especially useful for short documents that don't need splitting. You would call it like so:

const result = await ragIngest({
  modelId,
  workspace,
  documents: samples,
  chunk: false,
});
console.log(`Ingested ${result.processed.length} documents`);
console.log("First entry:", result.processed[0]);

chunk: false tells the SDK to skip splitting. The next lesson uses chunking for longer text.

Note: re-running ragIngest against the same workspace doesn't double-ingest. The SDK identifies already-embedded documents and skips them. The first run takes the longest; the second is faster.

Note: the built-in store defaults to HyperDB. Set ragTurbovec: true in qvac.config.json to put new workspaces on a TurboVec index instead; ragIngest and ragSearch don't change. ragIngest fixes the adapter once, when it creates the workspace: a workspace already on HyperDB stays there even after the flag turns on. TurboVec needs an embedding dimension divisible by 8 and no larger than 1024, which GTE_LARGE_FP16 meets exactly.