@elsium-ai/rag

RAG pipeline, document processing, embeddings, and vector stores for ElsiumAI.

Install

npm install @elsium-ai/rag @elsium-ai/core

What’s Inside

Types

Document

Represents a loaded document ready for chunking.

interface Document {
  id: string
  content: string
  metadata: DocumentMetadata
}

DocumentMetadata

Metadata attached to a document. Includes required fields plus arbitrary extra properties.

interface DocumentMetadata {
  source: string
  type: string
  title?: string
  language?: string
  createdAt?: string
  [key: string]: unknown
}

Chunk

A segment of a document produced by a chunker.

interface Chunk {
  id: string
  content: string
  documentId: string
  index: number
  metadata: ChunkMetadata
}

ChunkMetadata

Positional and token metadata for a chunk. Supports arbitrary extra properties.

interface ChunkMetadata {
  startChar: number
  endChar: number
  tokenEstimate: number
  [key: string]: unknown
}

EmbeddingVector

A numeric embedding vector with its dimensionality.

interface EmbeddingVector {
  values: number[]
  dimensions: number
}

EmbeddedChunk

A chunk that has been embedded. Extends Chunk with an embedding field.

interface EmbeddedChunk extends Chunk {
  embedding: EmbeddingVector
}

RetrievalResult

A single result returned from a vector store query, pairing a chunk with its relevance score.

interface RetrievalResult {
  chunk: Chunk
  score: number
  distance: number
}

QueryOptions

Options passed to a vector store query or pipeline query.

interface QueryOptions {
  topK?: number
  minScore?: number
  filter?: Record<string, unknown>
}

LoaderType

Union of supported document loader types.

type LoaderType = 'text' | 'markdown' | 'html' | 'json' | 'csv' | 'pdf'

ChunkingStrategy

Union of supported chunking strategies.

type ChunkingStrategy = 'fixed-size' | 'recursive' | 'sentence'

ChunkingConfig

Configuration object for creating a chunker via getChunker.

interface ChunkingConfig {
  strategy: ChunkingStrategy
  maxChunkSize?: number
  overlap?: number
  separator?: string
}

EmbeddingConfig

Configuration object for creating an embedding provider.

interface EmbeddingConfig {
  provider: string
  model?: string
  apiKey?: string
  baseUrl?: string
  dimensions?: number
  batchSize?: number
}

VectorStoreConfig

Configuration object for specifying a vector store backend.

interface VectorStoreConfig {
  provider: string
  connectionString?: string
  tableName?: string
  dimensions?: number
}

RetrievalConfig

Configuration for retrieval behavior within a RAG pipeline.

interface RetrievalConfig {
  topK?: number
  minScore?: number
  strategy?: 'similarity' | 'mmr'
  mmrLambda?: number
}

Loaders

DocumentLoader

Interface implemented by all document loaders.

interface DocumentLoader {
  load(source: string, content: string): Document
}

BinaryDocumentLoader Interface

pdfLoader reads binary data asynchronously rather than a string, so it implements a separate interface.

interface BinaryDocumentLoader {
  load(source: string, data: Buffer | Uint8Array): Promise<Document>
}

textLoader

Creates a loader for plain text content.

function textLoader(): DocumentLoader

Returns: A DocumentLoader that produces documents with type: 'text'.

import { textLoader } from '@elsium-ai/rag'

const loader = textLoader()
const doc = loader.load('notes.txt', 'Hello, world!')

console.log(doc.metadata.type) // "text"

markdownLoader

Creates a loader for Markdown content. Automatically extracts the first # heading as the document title.

function markdownLoader(): DocumentLoader

Returns: A DocumentLoader that produces documents with type: 'markdown' and an optional title in metadata.

import { markdownLoader } from '@elsium-ai/rag'

const loader = markdownLoader()
const doc = loader.load('readme.md', '# My Project\n\nSome content here.')

console.log(doc.metadata.title) // "My Project"

htmlLoader

Creates a loader for HTML content. Strips tags, scripts, and styles, then extracts the <title> element as the document title.

function htmlLoader(): DocumentLoader

Returns: A DocumentLoader that produces documents with type: 'html', the HTML body converted to plain text, and an optional title in metadata.

import { htmlLoader } from '@elsium-ai/rag'

const loader = htmlLoader()
const doc = loader.load('page.html', '<html><title>Hello</title><body><p>World</p></body></html>')

console.log(doc.metadata.title) // "Hello"
console.log(doc.content)        // "World"

jsonLoader

Creates a loader for JSON content. Extracts text from a configurable content field and can pull additional metadata fields.

function jsonLoader(options?: {
  contentField?: string
  metadataFields?: string[]
}): DocumentLoader

Returns: A DocumentLoader that produces documents with type: 'json'. If the parsed JSON is an array, each item’s content field is joined with double newlines.

import { jsonLoader } from '@elsium-ai/rag'

const loader = jsonLoader({ contentField: 'text', metadataFields: ['author'] })
const doc = loader.load('data.json', JSON.stringify({ text: 'Hello', author: 'Alice' }))

console.log(doc.content)          // "Hello"
console.log(doc.metadata.author)  // "Alice"

csvLoader

Creates a loader for CSV content. Parses headers and rows, optionally selecting specific columns.

function csvLoader(options?: {
  separator?: string
  contentColumns?: string[]
}): DocumentLoader

Returns: A DocumentLoader that produces documents with type: 'csv'. Metadata includes rowCount and columns.

import { csvLoader } from '@elsium-ai/rag'

const loader = csvLoader({ contentColumns: ['name', 'bio'] })
const doc = loader.load('people.csv', 'name,age,bio\nAlice,30,Engineer\nBob,25,Designer')

console.log(doc.metadata.rowCount) // 2
console.log(doc.metadata.columns)  // ["name", "age", "bio"]

getLoader

Factory function that returns a DocumentLoader for the given LoaderType.

function getLoader(type: LoaderType): DocumentLoader

Returns: The corresponding DocumentLoader with default options.

import { getLoader } from '@elsium-ai/rag'

const loader = getLoader('markdown')
const doc = loader.load('file.md', '# Title\n\nBody text.')

Chunkers

Chunker

Interface implemented by all chunking strategies.

interface Chunker {
  chunk(document: Document): Chunk[]
}

fixedSizeChunker

Creates a chunker that splits documents into fixed-size character windows with optional overlap.

function fixedSizeChunker(options?: {
  maxChunkSize?: number
  overlap?: number
}): Chunker

Returns: A Chunker that produces fixed-size chunks.

import { fixedSizeChunker } from '@elsium-ai/rag'

const chunker = fixedSizeChunker({ maxChunkSize: 256, overlap: 32 })
// Assuming `doc` is a Document:
const chunks = chunker.chunk(doc)

recursiveChunker

Creates a chunker that recursively splits text using a hierarchy of separators (paragraph breaks, line breaks, sentence endings, spaces), falling back to fixed-size splitting when no separator fits.

function recursiveChunker(options?: {
  maxChunkSize?: number
  overlap?: number
  separators?: string[]
}): Chunker

Returns: A Chunker that recursively splits documents using natural boundaries.

import { recursiveChunker } from '@elsium-ai/rag'

const chunker = recursiveChunker({ maxChunkSize: 512, overlap: 50 })
const chunks = chunker.chunk(doc)

sentenceChunker

Creates a chunker that splits text on sentence boundaries (., !, ? followed by whitespace) and groups sentences into chunks up to maxChunkSize characters with optional sentence-level overlap.

function sentenceChunker(options?: {
  maxChunkSize?: number
  overlap?: number
}): Chunker

Returns: A Chunker that groups sentences into chunks. Each chunk’s metadata includes sentenceCount.

import { sentenceChunker } from '@elsium-ai/rag'

const chunker = sentenceChunker({ maxChunkSize: 300, overlap: 2 })
const chunks = chunker.chunk(doc)

console.log(chunks[0].metadata.sentenceCount) // number of sentences in the first chunk

getChunker

Factory function that returns a Chunker for the given ChunkingConfig.

function getChunker(config: ChunkingConfig): Chunker

Returns: The corresponding Chunker instance.

import { getChunker } from '@elsium-ai/rag'

const chunker = getChunker({ strategy: 'recursive', maxChunkSize: 512, overlap: 50 })
const chunks = chunker.chunk(doc)

Embeddings

EmbeddingProvider

Interface implemented by all embedding providers.

interface EmbeddingProvider {
  readonly name: string
  readonly dimensions: number

  embed(text: string): Promise<EmbeddingVector>
  embedBatch(texts: string[]): Promise<EmbeddingVector[]>
}

createOpenAIEmbeddings

Creates an embedding provider backed by the OpenAI embeddings API.

function createOpenAIEmbeddings(config: EmbeddingConfig): EmbeddingProvider

Returns: An EmbeddingProvider with name: 'openai'.

import { createOpenAIEmbeddings } from '@elsium-ai/rag'

const embeddings = createOpenAIEmbeddings({
  provider: 'openai',
  apiKey: process.env.OPENAI_API_KEY,
  model: 'text-embedding-3-small',
  dimensions: 1536,
})

const vector = await embeddings.embed('Hello, world!')
console.log(vector.dimensions) // 1536

createMockEmbeddings

Creates a deterministic mock embedding provider for testing. Produces normalized vectors derived from character codes.

function createMockEmbeddings(dims?: number): EmbeddingProvider

Returns: An EmbeddingProvider with name: 'mock'.

import { createMockEmbeddings } from '@elsium-ai/rag'

const embeddings = createMockEmbeddings(64)
const vector = await embeddings.embed('test input')

console.log(vector.dimensions) // 64

getEmbeddingProvider

Factory function that returns an EmbeddingProvider for the given EmbeddingConfig. Supports 'openai' and 'mock' providers.

function getEmbeddingProvider(config: EmbeddingConfig): EmbeddingProvider

Returns: The corresponding EmbeddingProvider. Throws an error for unknown providers.

import { getEmbeddingProvider } from '@elsium-ai/rag'

const embeddings = getEmbeddingProvider({ provider: 'mock', dimensions: 256 })
const vector = await embeddings.embed('sample text')

GoogleEmbeddingsConfig

Configuration for createGoogleEmbeddings — Google Generative AI embeddings (default text-embedding-004, 768 dimensions).

interface GoogleEmbeddingsConfig {
  apiKey: string         // required
  model?: string         // default: 'text-embedding-004'
  dimensions?: number    // default: 768 (sent as outputDimensionality)
}

CohereEmbeddingsConfig

Configuration for createCohereEmbeddings — Cohere embeddings (default embed-v4.0, 1024 dimensions).

interface CohereEmbeddingsConfig {
  apiKey: string         // required
  model?: string         // default: 'embed-v4.0'
  inputType?: string     // default: 'search_document'
}

Vector Store

VectorStore

Interface implemented by all vector store backends.

interface VectorStore {
  readonly name: string

  upsert(chunks: EmbeddedChunk[]): Promise<void>
  query(embedding: EmbeddingVector, options?: QueryOptions): Promise<RetrievalResult[]>
  delete(ids: string[]): Promise<void>
  clear(): Promise<void>
  count(): Promise<number>
}

createInMemoryStore

Creates an in-memory vector store backed by a Map. Supports cosine-similarity search and automatic eviction when exceeding maxChunks.

function createInMemoryStore(options?: {
  maxChunks?: number
}): VectorStore

Returns: A VectorStore with name: 'in-memory'.

import { createInMemoryStore, createMockEmbeddings } from '@elsium-ai/rag'

const store = createInMemoryStore({ maxChunks: 10_000 })
const embeddings = createMockEmbeddings()

// Upsert embedded chunks
await store.upsert(embeddedChunks)

// Query
const queryVector = await embeddings.embed('search query')
const results = await store.query(queryVector, { topK: 5, minScore: 0.5 })

console.log(await store.count()) // number of stored chunks

cosineSimilarity

Computes the cosine similarity between two numeric vectors. Returns 0 if the vectors have different lengths or either has zero magnitude.

function cosineSimilarity(a: number[], b: number[]): number

Returns: A number between -1 and 1 representing cosine similarity.

import { cosineSimilarity } from '@elsium-ai/rag'

const similarity = cosineSimilarity([1, 0, 0], [1, 0, 0])
console.log(similarity) // 1

const orthogonal = cosineSimilarity([1, 0], [0, 1])
console.log(orthogonal) // 0

mmrRerank

Reranks retrieval results using Maximal Marginal Relevance (MMR) to balance relevance and diversity. Requires that each result includes its embedding field.

function mmrRerank(
  queryEmbedding: EmbeddingVector,
  results: Array<RetrievalResult & { embedding: EmbeddingVector }>,
  options?: { topK?: number; lambda?: number },
): RetrievalResult[]

Returns: An array of RetrievalResult objects reranked by MMR.

import { mmrRerank } from '@elsium-ai/rag'

const reranked = mmrRerank(queryEmbedding, candidateResults, {
  topK: 3,
  lambda: 0.5,
})

PgVectorStoreConfig

Configuration for createPgVectorStore — PostgreSQL + pgvector store (requires the pg peer dependency).

interface PgVectorStoreConfig {
  connectionString: string   // required
  tableName?: string         // default: 'vector_chunks' (validated identifier)
  dimensions?: number        // default: 1536
}

QdrantStoreConfig

Configuration for createQdrantStore — Qdrant store over its HTTP API.

interface QdrantStoreConfig {
  url: string                // required
  apiKey?: string
  collectionName: string     // required
  dimensions: number         // required
}

Pipeline

RAGPipelineConfig

Configuration object for creating a RAG pipeline via rag().

interface RAGPipelineConfig {
  loader?: LoaderType
  chunking?: ChunkingConfig
  embeddings: EmbeddingConfig
  store?: VectorStoreConfig
  retrieval?: RetrievalConfig
}

RAGPipeline

Interface representing a fully configured RAG pipeline with ingest and query capabilities.

interface RAGPipeline {
  ingest(source: string, content: string): Promise<IngestResult>
  ingestDocument(document: Document): Promise<IngestResult>
  query(text: string, options?: QueryOptions): Promise<RetrievalResult[]>
  clear(): Promise<void>
  count(): Promise<number>
  readonly embeddingProvider: EmbeddingProvider
  readonly vectorStore: VectorStore
}

IngestResult

Summary returned after ingesting a document into the pipeline.

interface IngestResult {
  documentId: string
  chunkCount: number
  totalTokens: number
}

rag

Creates a complete RAG pipeline that handles loading, chunking, embedding, storing, and querying documents in a single unified API.

function rag(config: RAGPipelineConfig): RAGPipeline

Returns: A RAGPipeline instance.

import { rag } from '@elsium-ai/rag'

const pipeline = rag({
  loader: 'markdown',
  chunking: { strategy: 'recursive', maxChunkSize: 512, overlap: 50 },
  embeddings: { provider: 'openai', apiKey: process.env.OPENAI_API_KEY },
  retrieval: { topK: 5, minScore: 0.5 },
})

// Ingest a document
const result = await pipeline.ingest('docs/guide.md', markdownContent)
console.log(result.chunkCount)  // number of chunks created
console.log(result.totalTokens) // estimated total tokens

// Query the pipeline
const hits = await pipeline.query('How do I configure the pipeline?')
for (const hit of hits) {
  console.log(hit.score, hit.chunk.content)
}

// Pipeline also exposes its internals
console.log(pipeline.embeddingProvider.name) // "openai"
console.log(await pipeline.count())          // total chunks stored

Keyword Search (BM25)

In-memory lexical index using Okapi BM25 scoring, for keyword-based retrieval over chunks.

BM25Index Interface

interface BM25Index {
  index(chunks: Chunk[]): void
  search(query: string, topK?: number): RetrievalResult[]  // topK default: 5
}

Options

createBM25Index({
  k1?: number,  // Term-frequency saturation (default: 1.2)
  b?: number,   // Length normalization (default: 0.75)
})

Example

import { createBM25Index } from '@elsium-ai/rag'

const bm25 = createBM25Index()
bm25.index(chunks)
const results = bm25.search('vector database', 5)
// => [{ chunk, score, distance: 0 }, ...]

Hybrid Search

Combine dense vector retrieval with BM25 keyword retrieval using Reciprocal Rank Fusion (RRF).

HybridSearch Interface

interface HybridSearch {
  search(
    query: string,
    queryEmbedding: EmbeddingVector,
    topK?: number,
  ): Promise<RetrievalResult[]>
}

HybridSearchConfig

interface HybridSearchConfig {
  k?: number             // RRF rank constant (default: 60)
  vectorWeight?: number  // Weight for vector results (default: 1)
  bm25Weight?: number    // Weight for BM25 results (default: 1)
  topK?: number          // Default result count (default: 10)
}

Example

import { createHybridSearch, createBM25Index, createInMemoryStore } from '@elsium-ai/rag'

const hybrid = createHybridSearch(vectorStore, bm25Index, {
  vectorWeight: 1,
  bm25Weight: 1,
})

const results = await hybrid.search('how to configure providers', queryEmbedding, 5)

Registries

Global registries let custom embedding providers and vector stores be selected by name (the built-in google, cohere, and qdrant integrations auto-register themselves).

interface Registry<T> {
  register(name: string, factory: T): void
  get(name: string): T | undefined
  list(): string[]
}

type EmbeddingProviderFactory = (config: EmbeddingConfig) => EmbeddingProvider
type VectorStoreFactory = (config: Record<string, unknown>) => VectorStore

Example

import { embeddingProviderRegistry, vectorStoreRegistry } from '@elsium-ai/rag'

embeddingProviderRegistry.register('my-provider', (config) => createMyEmbeddings(config))
embeddingProviderRegistry.list() // => ['google', 'cohere', 'my-provider']

Part of ElsiumAI

This package is the RAG pipeline layer of the ElsiumAI framework. See the Getting Started guide or the Fundamentals for comprehensive coverage of every feature.

License

MIT