Production RAG Pipelines at Scale

Retrieval-Augmented Generation is simple in a demo and brutal in production. Chunk boundaries destroy context, embedding quality degrades on domain-specific content, and single-vector retrieval misses nuance. This guide covers the hard-won patterns for building RAG pipelines that actually work — from chunking strategies through hybrid search to automated evaluation — all with ElsiumAI’s @elsium-ai/rag package.

Beyond Basic RAG

The canonical RAG demo looks like this: split a document into chunks, embed them, store them in a vector database, retrieve the top-K nearest neighbors, and stuff them into a prompt. It works on a README. It falls apart on a 10,000-page regulatory corpus.

Here’s why naive RAG fails at scale:

• Chunk boundary loss — A fixed-size splitter can slice a sentence in half. The embedding for “The dosage should not exceed” is meaningless without “200mg per day.” Context bleeds across boundaries, and your retriever has no way to recover it.
• Embedding quality degradation — General-purpose embedding models (like text-embedding-3-small) perform well on common English but struggle with domain jargon, acronyms, and highly technical content. A medical corpus full of ICD-10 codes behaves nothing like Wikipedia.
• Single-vector retrieval misses nuance — Dense vector similarity captures semantic meaning but is blind to exact keyword matches. If a user asks for “OSHA regulation 1910.134,” vector search may return semantically similar content about respiratory safety — but miss the exact regulation number. You need both dense and sparse retrieval.
• No evaluation loop — Without measuring retrieval precision and recall, you’re flying blind. A pipeline that “feels right” in testing can silently degrade as your corpus grows or shifts in distribution.

The rest of this article addresses each of these failure modes with concrete code and configuration using ElsiumAI.

Chunking Strategies

Chunking is where most RAG pipelines go wrong first. ElsiumAI ships three chunkers, each suited to different content types. The right choice depends on your source material and how much structural information it carries.

Fixed-Size Chunker

The simplest approach. Splits text into chunks of N tokens with a configurable overlap. Use this when your source material has no meaningful structure — raw logs, transcripts, or flat text dumps.

import { fixedSizeChunker } from '@elsium-ai/rag'

const chunks = fixedSizeChunker(document, {
  chunkSize: 512,    // tokens per chunk
  overlap: 64,       // token overlap between consecutive chunks
  tokenizer: 'cl100k_base',  // match your embedding model's tokenizer
})

The overlap parameter is critical. Without it, information at chunk boundaries is lost. A good rule of thumb: set overlap to 10–15% of your chunk size. Too much overlap wastes storage and embedding compute; too little and you get boundary artifacts.

Sentence Chunker

Splits on sentence boundaries, then groups sentences until the token limit is reached. This preserves complete thoughts and works well for prose-heavy content — articles, legal text, documentation.

import { sentenceChunker } from '@elsium-ai/rag'

const chunks = sentenceChunker(document, {
  maxChunkSize: 512,
  minChunkSize: 100,   // avoid tiny trailing chunks
  overlap: 2,          // overlap by N sentences
  language: 'en',
})

Sentence-level overlap is more semantically meaningful than token-level overlap. Overlapping by 2 sentences ensures that cross-sentence references (“This regulation”, “The above criteria”) are captured in at least one chunk.

Recursive Chunker

The most sophisticated option. It recursively splits by structural markers — first by headings, then by paragraphs, then by sentences — and only falls back to token-level splitting as a last resort. Use this for structured documents like markdown, HTML, or anything with headings.

import { recursiveChunker } from '@elsium-ai/rag'

const chunks = recursiveChunker(document, {
  maxChunkSize: 1024,
  separators: ['\n## ', '\n### ', '\n\n', '\n', '. '],
  preserveMetadata: true,  // attach source heading to each chunk
})

When preserveMetadata is enabled, each chunk carries a metadata.heading field with the nearest parent heading. This is invaluable for retrieval — you can display the section title alongside retrieved content, and it improves reranking quality significantly.

When to Use Which

Embedding Pipeline

Once your documents are chunked, you need to convert them into vectors. The key challenge at scale is throughput: embedding a million chunks one-at-a-time is painfully slow and expensive. ElsiumAI provides batch embedding and incremental ingestion out of the box.

Basic Setup

import { createOpenAIEmbeddings } from '@elsium-ai/rag'

const embeddings = createOpenAIEmbeddings({
  model: 'text-embedding-3-large',
  dimensions: 1536,       // or 3072 for higher fidelity
  apiKey: process.env.OPENAI_API_KEY,
})

Batch Embedding for Millions of Documents

For large corpora, use createBatch to process chunks in parallel batches with automatic retries and rate-limit backoff. This is the difference between a 3-hour ingestion job and a 15-minute one.

import { createBatch } from '@elsium-ai/rag'

const batchEmbedder = createBatch(embeddings, {
  batchSize: 2048,         // chunks per API call
  concurrency: 8,          // parallel API calls
  retries: 3,              // retry on 429/5xx
  backoff: 'exponential',  // backoff strategy
  onProgress: (processed, total) => {
    console.log(`Embedded ${processed}/${total} chunks`)
  },
})

// Embed all chunks — returns EmbeddedChunk[]
const embeddedChunks = await batchEmbedder.embedAll(chunks)

Incremental Ingestion

In production, you rarely re-embed your entire corpus. New documents arrive daily, and re-embedding unchanged content wastes money. Use content hashing to skip already-embedded chunks:

import { createIncrementalIngester } from '@elsium-ai/rag'

const ingester = createIncrementalIngester({
  embeddings: batchEmbedder,
  store: vectorStore,
  hashAlgorithm: 'xxhash64',  // fast content hashing
})

// Only embeds and stores chunks whose content hash is new
const result = await ingester.ingest(newChunks)
console.log(`${result.added} new, ${result.skipped} unchanged, ${result.updated} modified`)

Vector Store Tuning

Your vector store is the backbone of retrieval performance. ElsiumAI’s createPgVectorStore wraps pgvector with connection pooling, index management, and query optimization built in.

Connection and Pooling

import { createPgVectorStore } from '@elsium-ai/rag'

const vectorStore = createPgVectorStore({
  connectionString: process.env.DATABASE_URL,
  tableName: 'document_embeddings',
  dimensions: 1536,
  pool: {
    min: 5,
    max: 20,
    idleTimeout: 30_000,   // ms before idle connections are released
  },
})

Indexing Strategies: IVFFlat vs HNSW

The choice of index type determines your latency/recall tradeoff. This is one of the most consequential decisions in a RAG pipeline.

// IVFFlat — faster to build, lower recall at low nprobe
await vectorStore.createIndex({
  type: 'ivfflat',
  lists: 1000,        // sqrt(N) is a good starting point
  probes: 10,         // higher = better recall, slower queries
})

// HNSW — slower to build, better recall, higher memory usage
await vectorStore.createIndex({
  type: 'hnsw',
  m: 16,              // connections per node (higher = better recall, more memory)
  efConstruction: 200, // build quality (higher = slower build, better graph)
  efSearch: 100,       // query quality (higher = better recall, slower search)
})

For most production deployments with more than a million vectors, HNSW is the right choice. The memory overhead is worth the recall improvement — a missed relevant chunk is far more costly than a few extra gigabytes of RAM.

Hybrid Search

Dense vector similarity alone is insufficient for production RAG. It excels at semantic matching but fails on exact keywords, entity names, regulation numbers, and other precise identifiers. Hybrid search combines dense retrieval with sparse (keyword) retrieval to cover both modes.

Combining Dense and Sparse Retrieval

import { createHybridRetriever, bm25Retriever } from '@elsium-ai/rag'

const hybrid = createHybridRetriever({
  dense: vectorStore.asRetriever({ topK: 20 }),
  sparse: bm25Retriever({
    index: bm25Index,
    topK: 20,
  }),
  weights: {
    dense: 0.7,   // semantic similarity weight
    sparse: 0.3,  // keyword match weight
  },
  topK: 10,        // final results after fusion
  fusion: 'reciprocal-rank',  // also: 'linear', 'relative-score'
})

The weights parameter is your primary tuning lever. For general Q&A over prose, bias toward dense (0.7/0.3). For technical documentation with specific identifiers, shift toward sparse (0.5/0.5 or even 0.4/0.6). Always measure with your actual query distribution.

MMR Reranking for Diversity

Maximal Marginal Relevance (MMR) reranking solves a subtle problem: when your top-K results are all paraphrases of the same paragraph. MMR balances relevance against diversity, ensuring your context window contains distinct information.

import { mmrRerank } from '@elsium-ai/rag'

const results = await hybrid.retrieve('OSHA respiratory protection requirements')

const reranked = mmrRerank(results, {
  lambda: 0.7,   // 1.0 = pure relevance, 0.0 = pure diversity
  topK: 5,        // final context window size
})

The lambda parameter controls the relevance–diversity tradeoff. Start at 0.7 and decrease if you notice redundant chunks in your retrieved context. For broad research questions, values around 0.5 work well. For highly specific factual queries, keep lambda at 0.8 or higher.

Multi-Source Ingestion

Real-world RAG pipelines ingest from many sources simultaneously — HTML documentation, CSV exports, JSON API responses, and plain text files. ElsiumAI provides loaders for each format and a workflow system to orchestrate them in parallel.

Source Loaders

import {
  htmlLoader,
  textLoader,
  csvLoader,
  jsonLoader,
} from '@elsium-ai/rag'

const htmlDocs = await htmlLoader('./docs/**/*.html', {
  selectors: ['article', 'main'],   // extract only content areas
  stripTags: true,
  preserveLinks: true,
})

const csvDocs = await csvLoader('./data/products.csv', {
  contentColumns: ['description', 'specifications'],
  metadataColumns: ['sku', 'category'],
})

const jsonDocs = await jsonLoader('./api-dump/*.json', {
  contentPath: '$.items[*].body',
  metadataPath: '$.items[*].metadata',
})

const textDocs = await textLoader('./legal/**/*.txt')

Parallel Ingestion Workflow

Use defineParallelWorkflow to run all ingestion pipelines concurrently, then merge results into a single chunking and embedding step:

import { defineParallelWorkflow, defineStep } from '@elsium-ai/workflows'
import { recursiveChunker, createBatch } from '@elsium-ai/rag'

const ingestWorkflow = defineParallelWorkflow({
  name: 'multi-source-ingest',
  branches: {
    html: defineStep('load-html', async () => {
      return htmlLoader('./docs/**/*.html', { selectors: ['article'] })
    }),
    csv: defineStep('load-csv', async () => {
      return csvLoader('./data/products.csv', {
        contentColumns: ['description'],
      })
    }),
    json: defineStep('load-json', async () => {
      return jsonLoader('./api-dump/*.json', {
        contentPath: '$.items[*].body',
      })
    }),
    text: defineStep('load-text', async () => {
      return textLoader('./legal/**/*.txt')
    }),
  },
  merge: defineStep('chunk-and-embed', async (results) => {
    const allDocs = [
      ...results.html,
      ...results.csv,
      ...results.json,
      ...results.text,
    ]

    const chunks = recursiveChunker(allDocs, {
      maxChunkSize: 1024,
      separators: ['\n## ', '\n### ', '\n\n', '\n', '. '],
      preserveMetadata: true,
    })

    const batchEmbedder = createBatch(embeddings, {
      batchSize: 2048,
      concurrency: 8,
    })

    const embedded = await batchEmbedder.embedAll(chunks)
    await vectorStore.upsert(embedded)

    return { total: embedded.length }
  }),
})

const result = await ingestWorkflow.run()
console.log(`Ingested ${result.total} chunks from all sources`)

RAG Evaluation

You cannot improve what you do not measure. ElsiumAI’s @elsium-ai/testing package includes a full evaluation suite for RAG pipelines, measuring retrieval quality and generation faithfulness.

Setting Up an Evaluation Suite

import {
  runEvalSuite,
  precisionAtK,
  recallAtK,
  relevanceScore,
  faithfulness,
  createEvalDataset,
} from '@elsium-ai/testing'

// Define your evaluation dataset — ground-truth query/answer pairs
const evalData = createEvalDataset([
  {
    query: 'What is the maximum dosage for ibuprofen?',
    expectedChunks: ['chunk-id-42', 'chunk-id-43'],
    expectedAnswer: 'The maximum daily dosage for ibuprofen is 1200mg for OTC use.',
  },
  {
    query: 'OSHA regulation 1910.134 requirements',
    expectedChunks: ['chunk-id-891'],
    expectedAnswer: 'OSHA 1910.134 mandates employer-provided respiratory protection...',
  },
  // ... hundreds more for statistical significance
])

Running Evaluations

const results = await runEvalSuite({
  retriever: hybrid,
  generator: llm,
  dataset: evalData,
  metrics: [
    precisionAtK(5),     // of top-5 retrieved, how many are relevant?
    recallAtK(10),       // of all relevant chunks, how many are in top-10?
    relevanceScore(),    // LLM-judged relevance of retrieved context
    faithfulness(),      // is the generated answer grounded in retrieved context?
  ],
  concurrency: 10,       // parallel evaluation runs
})

console.log('Retrieval Metrics:')
console.log(`  Precision@5:  ${results.precisionAt5.toFixed(3)}`)
console.log(`  Recall@10:    ${results.recallAt10.toFixed(3)}`)
console.log(`  Relevance:    ${results.relevance.toFixed(3)}`)
console.log(`  Faithfulness: ${results.faithfulness.toFixed(3)}`)

Run this suite on every pipeline change — new chunking strategy, different embedding model, index parameter tuning. Track metrics over time. A 2% drop in recall@10 after a chunker change tells you more than any amount of manual spot-checking.

Production Architecture

A production RAG pipeline is more than retrieval. Here’s the full architecture, from document ingestion through to generation:

# Production RAG Pipeline Architecture

Source Files (HTML, CSV, JSON, TXT)
        |
        v
  +-----------+
  |  Loaders  |  htmlLoader, csvLoader, jsonLoader, textLoader
  +-----------+
        |
        v
  +-----------+
  |  Chunker  |  recursiveChunker (with metadata preservation)
  +-----------+
        |
        v
  +------------+
  | Embeddings |  createBatch → OpenAI text-embedding-3-large
  +------------+
        |
        v
  +--------------+      +----------------+
  | Vector Store | <--- | HNSW Index     |
  | (pgvector)   |      | (efSearch=100) |
  +--------------+      +----------------+
        |
        v
  +------------------+
  | Hybrid Retriever |  dense (0.7) + BM25 sparse (0.3)
  +------------------+
        |
        v
  +-------------+
  | MMR Rerank  |  lambda=0.7, topK=5
  +-------------+
        |
        v
  +-----------+      +-----------------+
  |  Context  | <--- | Response Cache  |
  |  Assembly |      | (Redis, TTL=1h) |
  +-----------+      +-----------------+
        |
        v
  +------------+
  | Generation |  LLM with system prompt + retrieved context
  +------------+

Caching layer: Cache at two levels. First, cache embedding results by content hash (so re-ingesting unchanged documents costs nothing). Second, cache final retrieval results by query hash with a short TTL — identical queries within a window skip the entire retrieval pipeline.

Cost controls: Embedding is the most expensive step. Use text-embedding-3-small for initial development and switch to text-embedding-3-large only when evaluation metrics justify it. Batch embedding with high concurrency minimizes wall-clock time, and incremental ingestion ensures you never pay to re-embed unchanged content.

Performance Checklist

Before shipping your RAG pipeline to production, validate each layer against this checklist:

What’s Next

This article covered the core mechanics of production RAG. From here, you can extend your pipeline with:

• Agentic RAG — use @elsium-ai/agents to build agents that decide when to retrieve, what query to use, and how many retrieval rounds to run before answering
• Multi-index routing — route queries to different vector stores based on intent classification (e.g., product catalog vs. support docs vs. legal corpus)
• Fine-tuned embeddings — train domain-specific embedding models on your corpus using contrastive learning, then swap them into createOpenAIEmbeddings via the model parameter
• Streaming generation — pipe retrieved context into a streaming LLM call so users see tokens as they arrive, rather than waiting for the full retrieval + generation cycle
• Continuous evaluation — run runEvalSuite in CI on every pull request that touches the RAG pipeline, and fail the build if recall drops below a threshold

Check out the @elsium-ai/rag docs for the full API reference, or explore 30 more products you can build with ElsiumAI.