Building an AI Gateway with ElsiumAI

Every company using LLMs ends up needing the same thing: a gateway that sits between your applications and LLM providers. It routes requests, caches responses, enforces budgets, and gives you a single pane of glass for observability. Here’s how to build one in under 50 lines of TypeScript.

The Problem

You start with one OpenAI call. Then someone wants Anthropic for creative writing. Google for long context. Suddenly you have API keys scattered across services, no unified cost tracking, and no way to failover when a provider goes down.

An AI gateway solves this by centralizing all LLM traffic through a single, instrumented proxy. Think of it as an API gateway — but purpose-built for LLMs.

What We’ll Build

By the end of this tutorial, you’ll have a gateway that:

• Routes across providers — Anthropic, OpenAI, and Google with automatic failover
• Caches responses — identical prompts hit cache instead of the API
• Tracks costs — real-time cost tracking per request and per tenant
• Rate limits — protect providers from burst traffic
• Observes everything — traces, latency, token usage via OpenTelemetry

Setup

Install the packages you need:

npm install @elsium-ai/gateway @elsium-ai/observe @elsium-ai/app

Step 1: Create the Provider Mesh

The provider mesh is the core of your gateway. It manages connections to multiple LLM providers and decides where to route each request.

import { createProviderMesh } from '@elsium-ai/gateway'

const mesh = createProviderMesh({
  providers: [
    { name: 'anthropic', config: { apiKey: env('ANTHROPIC_API_KEY') }, priority: 1 },
    { name: 'openai', config: { apiKey: env('OPENAI_API_KEY') }, priority: 2 },
    { name: 'google', config: { apiKey: env('GOOGLE_API_KEY') }, priority: 3 },
  ],
  strategy: 'cost-optimized', // also: 'latency-optimized', 'round-robin', 'priority'
})

The strategy option controls routing behavior. cost-optimized picks the cheapest provider that supports the requested model. If that provider is down, it automatically fails over to the next one.

Step 2: Add Middleware

ElsiumAI’s gateway uses a middleware stack — each middleware wraps the request/response lifecycle. Order matters: middleware executes top-to-bottom on requests and bottom-to-top on responses.

import {
  cacheMiddleware,
  costTrackingMiddleware,
  rateLimitMiddleware,
  authMiddleware,
} from '@elsium-ai/gateway'

const gw = gateway({
  mesh,
  middleware: [
    authMiddleware({ apiKeys: true }),       // Validate API keys first
    rateLimitMiddleware({ rpm: 1000 }),       // Then enforce rate limits
    cacheMiddleware({ ttl: '1h' }),           // Cache before hitting providers
    costTrackingMiddleware(),                 // Track costs on the way out
  ],
})

Step 3: Add Observability

Every request through the gateway should be traced. ElsiumAI’s observe package plugs directly into the gateway middleware stack and exports spans to any OpenTelemetry-compatible backend.

import { createSpan, createMetrics, createOTLPExporter } from '@elsium-ai/observe'
import { createCostEngine } from '@elsium-ai/gateway'

const metrics = createMetrics()
const exporter = createOTLPExporter({ endpoint: 'https://otel.example.com' })

const cost = createCostEngine()
cost.on('threshold', (event) => {
  console.warn(`Cost spike detected: $${event.total}`)
})

Step 4: Serve It

Wrap the gateway in an HTTP server using @elsium-ai/app. This gives you health checks, graceful shutdown, and CORS out of the box.

import { createApp } from '@elsium-ai/app'

const app = createApp({
  gateway: gw,
  port: 3000,
  cors: true,
  healthCheck: '/health',
})

app.listen()

That’s it. Your gateway is running. Any application can now point to http://localhost:3000 and use it as a drop-in replacement for direct provider calls.

Step 5: Test It

Use the built-in testing utilities to verify your gateway works without making real API calls:

import { createTestGateway, mockProvider } from '@elsium-ai/testing'

const testGw = createTestGateway({
  providers: [
    mockProvider('anthropic', {
      responses: [{ content: 'Hello from mock!' }],
    }),
  ],
})

const response = await testGw.chat({
  model: 'claude-sonnet-4-6',
  messages: [{ role: 'user', content: 'Hi' }],
})

assert(response.content === 'Hello from mock!')

Production Checklist

Before shipping your gateway to production, make sure you’ve got these covered:

What’s Next

From here you can extend your gateway with:

• Multi-tenant isolation — add tenantMiddleware for per-customer routing and billing
• Governance policies — enforce content filtering, PII redaction, or compliance rules at the gateway level
• Streaming — the gateway supports SSE streaming out of the box — just use stream: true
• Model registry — use registerModelTier to control which models each tier can access

Check out the @elsium-ai/gateway docs for the full API reference, or explore 30 more products you can build with ElsiumAI.