Chunk documents for RAG

Now that we know how to ingest and search, let's handle longer documents.

A 1024-token embedding model can't take a 5000-word article as a single input. chunk: true tells ragIngest to split documents before embedding. The search results we saw before were whole documents. Chunked search returns the passage that matched, which is what we want for question-answering.

The default strategy is paragraph: the SDK splits on blank lines, then groups paragraphs to roughly hit chunkSize tokens. chunkOverlap keeps a few tokens of shared context at every boundary so a sentence that straddles two chunks isn't lost.

chunk: true splits each document before embedding. The default paragraph strategy splits on blank lines, then groups paragraphs to roughly hit chunkSize tokens:

const result = await ragIngest({
  modelId,
  workspace: "tech",
  documents: samples,
  chunk: true,
  chunkOpts: {
    chunkSize: 200,
    chunkOverlap: 20,
  },
});
console.log(`Created ${result.processed.length} chunks from ${samples.length} documents`);

processed is now a list of chunks, not documents. Search against this workspace returns the specific passage that matched, not the whole document.

Note: chunkSize is in tokens, not characters. A typical English word is roughly 1.3 tokens, so chunkSize: 200 gives you chunks of about 150 words.