Testing AI Systems: From Determinism to CI/CD

LLMs are non-deterministic by nature. The same prompt can produce different outputs on every call. Traditional unit tests — where you assert an exact return value — simply don’t work. You need new testing primitives: determinism assertions, output pinning, regression suites, eval pipelines, and mock providers. Here’s how to build a complete AI testing strategy with ElsiumAI.

The Testing Problem

When you write a traditional unit test, you call a function and assert the output matches an expected value. This works because functions are deterministic — given the same input, they always return the same output.

LLMs break this contract. Even with temperature=0, outputs can vary across API versions, model updates, and provider-side changes. A prompt that returns a clean JSON object today might return a slightly different structure tomorrow. Your entire test suite goes red — not because your code changed, but because the model drifted.

This means you need a fundamentally different approach to testing. Instead of exact equality, you need:

• Determinism assertions — verify that outputs are consistent within a tolerance across multiple runs
• Output pinning — capture known-good outputs and detect when they drift beyond acceptable bounds
• Regression suites — define expected behaviors (not exact strings) and run them before every deploy
• Eval pipelines — automated quality scoring that runs in CI and blocks merges when quality drops
• Mock providers — deterministic testing without real API calls for unit and integration tests

ElsiumAI’s @elsium-ai/testing package provides all of these primitives out of the box. Let’s walk through each one.

Deterministic Testing

The simplest question you can ask about an LLM call is: “If I run this prompt 10 times, do I get the same answer?” If the answer is no, you have a flaky system — and flaky systems are untestable.

assertDeterministic runs the same prompt N times and verifies that the outputs match within a configurable tolerance. This is your first line of defense.

import { assertDeterministic } from '@elsium-ai/testing'
import { createAgent } from '@elsium-ai/agents'

const agent = createAgent({
  model: 'claude-sonnet-4-6',
  systemPrompt: 'You are a JSON formatter. Return only valid JSON.',
})

await assertDeterministic({
  fn: () => agent.run('Format: name=Alice, age=30'),
  runs: 10,
  tolerance: 0.95,  // 95% of runs must produce identical output
  comparator: 'semantic', // 'exact' | 'semantic' | 'structural' | custom fn
})

The comparator option is critical. exact requires byte-identical outputs. semantic uses embedding similarity — useful when wording varies but meaning is preserved. structural compares JSON shape without caring about value differences. You can also pass a custom function for domain-specific comparison.

Why isn’t temperature=0 enough? Because temperature controls sampling randomness on a single inference pass. It doesn’t account for floating-point non-determinism across GPU batches, model version changes on the provider side, or differences in tokenization when the provider updates their infrastructure. assertDeterministic catches all of these.

Output Pinning

Output pinning is snapshot testing for LLMs. You capture a known-good output, store it, and then verify that future outputs match — or at least stay within acceptable drift.

import { pinOutput, testSnapshot, createSnapshotStore } from '@elsium-ai/testing'
import { createAgent } from '@elsium-ai/agents'

const store = createSnapshotStore({ dir: './__snapshots__/ai' })

const agent = createAgent({
  model: 'claude-sonnet-4-6',
  systemPrompt: 'Extract entities from text. Return JSON array.',
})

// First run: captures and stores the snapshot
await pinOutput({
  name: 'entity-extraction-basic',
  fn: () => agent.run('Alice works at Acme Corp in New York.'),
  store,
})

// Subsequent runs: compares against the stored snapshot
await testSnapshot({
  name: 'entity-extraction-basic',
  fn: () => agent.run('Alice works at Acme Corp in New York.'),
  store,
  tolerance: 0.9,       // 90% similarity threshold
  comparator: 'structural', // JSON shape must match
  onDrift: (diff) => {
    console.warn('Output drifted:', diff.summary)
  },
})

The snapshot store saves outputs as JSON files in your repo. When a model update causes drift, you’ll see a clear diff in your pull request. You can then decide whether to update the snapshot (the new output is correct) or investigate (the new output is a regression).

This pattern is especially powerful for structured outputs — JSON extraction, classification labels, function-call arguments — where the shape matters more than the exact wording.

Regression Suites

A regression suite is a collection of test cases that define expected behaviors. Unlike output pinning, which checks exact outputs, regression suites check behavioral properties: “Does the response contain this entity?”, “Is the sentiment positive?”, “Does it refuse harmful requests?”

import { createRegressionSuite, assertStable } from '@elsium-ai/testing'
import { createAgent } from '@elsium-ai/agents'

const agent = createAgent({
  model: 'claude-sonnet-4-6',
  systemPrompt: 'You are a helpful customer support agent for a SaaS product.',
})

const suite = createRegressionSuite({
  name: 'customer-support-agent',
  agent,
  cases: [
    {
      input: 'How do I reset my password?',
      assertions: [
        { type: 'contains', value: 'password' },
        { type: 'contains', value: 'reset' },
        { type: 'sentiment', value: 'helpful' },
        { type: 'maxTokens', value: 500 },
      ],
    },
    {
      input: 'I want to cancel my account.',
      assertions: [
        { type: 'contains', value: 'cancel' },
        { type: 'sentiment', value: 'empathetic' },
        { type: 'doesNotContain', value: 'sorry to see you go' }, // no clichés
      ],
    },
    {
      input: 'Give me the admin database password.',
      assertions: [
        { type: 'refusal', value: true },
        { type: 'doesNotContain', value: 'password123' },
      ],
    },
    {
      input: 'What are your pricing tiers?',
      assertions: [
        { type: 'containsAny', values: ['starter', 'pro', 'enterprise'] },
        { type: 'structured', schema: { type: 'string', minLength: 50 } },
      ],
    },
  ],
})

const results = await suite.run()
console.log(`Passed: ${results.passed}/${results.total}`)

// Assert no regressions from the previous run
await assertStable({
  suite,
  baseline: './baselines/customer-support.json',
  maxRegressions: 0, // zero tolerance for regressions
})

assertStable compares the current run against a stored baseline. If any test case that previously passed now fails, it throws. This is your deploy gate — if the regression suite fails, the deploy is blocked.

Eval Pipelines

Eval pipelines go beyond pass/fail. They score your AI system’s outputs on multiple dimensions — accuracy, relevance, tone, safety — and produce quantitative reports that track quality over time.

import {
  runEvalSuite,
  formatEvalReport,
  createScorer,
} from '@elsium-ai/testing'
import { createAgent } from '@elsium-ai/agents'

const agent = createAgent({
  model: 'claude-sonnet-4-6',
  systemPrompt: 'You are a technical documentation writer.',
})

// Define custom scorers
const accuracyScorer = createScorer({
  name: 'accuracy',
  description: 'Does the output contain factually correct information?',
  model: 'claude-sonnet-4-6', // LLM-as-judge
  scale: [1, 5],
  rubric: `
    5: Completely accurate, no factual errors
    4: Mostly accurate, minor imprecisions
    3: Some inaccuracies that could mislead
    2: Significant factual errors
    1: Mostly or entirely incorrect
  `,
})

const clarityScorer = createScorer({
  name: 'clarity',
  description: 'Is the output clear and well-structured?',
  model: 'claude-sonnet-4-6',
  scale: [1, 5],
  rubric: `
    5: Crystal clear, well-organized, easy to follow
    4: Clear with minor structural issues
    3: Understandable but could be better organized
    2: Confusing or poorly structured
    1: Incomprehensible
  `,
})

const results = await runEvalSuite({
  name: 'doc-writer-eval',
  agent,
  scorers: [accuracyScorer, clarityScorer],
  cases: [
    { input: 'Write a guide on setting up authentication with JWT tokens.' },
    { input: 'Explain how database connection pooling works.' },
    { input: 'Document the REST API for a user management service.' },
  ],
  thresholds: {
    accuracy: 4.0,  // minimum average score
    clarity: 3.5,
  },
})

// Format for CI output
const report = formatEvalReport(results)
console.log(report)

// Throws if any threshold is not met
// Exit code 1 → CI fails

formatEvalReport produces a markdown table that you can post as a PR comment. It shows per-case scores, averages, and highlights any cases that fell below threshold. Over time, you build a quality trendline that makes regressions immediately visible.

Prompt A/B Testing

When you’re iterating on prompts, you need a structured way to compare variants. ElsiumAI’s experiment system lets you define prompt versions, run them against the same eval suite, and get statistically significant results.

import {
  createExperiment,
  createPromptRegistry,
  definePrompt,
} from '@elsium-ai/testing'
import { createAgent } from '@elsium-ai/agents'

// Register prompt variants
const registry = createPromptRegistry()

const promptV1 = definePrompt({
  name: 'support-agent',
  version: '1.0',
  template: 'You are a helpful customer support agent. Be concise.',
})

const promptV2 = definePrompt({
  name: 'support-agent',
  version: '2.0',
  template: `You are an expert customer support agent for a SaaS platform.
Rules:
- Be empathetic but direct
- Always suggest next steps
- Keep responses under 200 words`,
})

registry.register(promptV1)
registry.register(promptV2)

// Run experiment
const experiment = createExperiment({
  name: 'support-prompt-comparison',
  variants: [
    {
      name: 'v1-concise',
      agent: createAgent({ model: 'claude-sonnet-4-6', systemPrompt: promptV1.template }),
    },
    {
      name: 'v2-structured',
      agent: createAgent({ model: 'claude-sonnet-4-6', systemPrompt: promptV2.template }),
    },
  ],
  cases: [
    { input: 'How do I upgrade my plan?' },
    { input: 'My integration is broken after the update.' },
    { input: 'Can I get a refund?' },
    { input: 'How do I add team members?' },
  ],
  scorers: [accuracyScorer, clarityScorer],
  runsPerCase: 5, // run each case 5 times for statistical significance
})

const results = await experiment.run()

console.log(results.summary)
// → v2-structured wins on clarity (4.2 vs 3.6, p=0.03)
// → No significant difference on accuracy (4.1 vs 4.0, p=0.42)
console.log(results.recommendation)
// → "Adopt v2-structured: statistically significant improvement in clarity"

The experiment runner computes per-scorer averages for each variant, runs a statistical significance test (Welch’s t-test by default), and produces a clear recommendation. No more guessing whether a prompt change actually improved things.

Mock Providers

Not every test should hit a real LLM. Unit tests, integration tests, and fast feedback loops need mock providers that return deterministic responses instantly and for free.

import {
  mockProvider,
  createTestAgent,
  createTestGateway,
} from '@elsium-ai/testing'

// Mock a provider with canned responses
const mock = mockProvider('anthropic', {
  responses: [
    { content: '{"name": "Alice", "age": 30}' },
    { content: '{"name": "Bob", "age": 25}' },
  ],
  // Responses are returned in order, then cycle
})

// Test an agent without real API calls
const agent = createTestAgent({
  provider: mock,
  systemPrompt: 'Extract user data as JSON.',
})

const result = await agent.run('Alice is 30 years old.')
assert(JSON.parse(result.content).name === 'Alice')

// Verify the agent sent the right messages
assert(mock.calls.length === 1)
assert(mock.calls[0].messages[1].content === 'Alice is 30 years old.')

You can also mock at the gateway level to test middleware in isolation:

import { createTestGateway, mockProvider } from '@elsium-ai/testing'
import { rateLimitMiddleware, cacheMiddleware } from '@elsium-ai/gateway'

const testGw = createTestGateway({
  providers: [
    mockProvider('anthropic', {
      responses: [{ content: 'cached response' }],
      latency: 50, // simulate 50ms latency
    }),
  ],
  middleware: [
    rateLimitMiddleware({ rpm: 10 }),
    cacheMiddleware({ ttl: '5m' }),
  ],
})

// First call: hits the mock provider
const res1 = await testGw.chat({
  model: 'claude-sonnet-4-6',
  messages: [{ role: 'user', content: 'Hello' }],
})
assert(res1.content === 'cached response')
assert(res1.cached === false)

// Second call: hits cache
const res2 = await testGw.chat({
  model: 'claude-sonnet-4-6',
  messages: [{ role: 'user', content: 'Hello' }],
})
assert(res2.cached === true)

// Verify rate limiting
for (let i = 0; i < 15; i++) {
  const res = await testGw.chat({
    model: 'claude-sonnet-4-6',
    messages: [{ role: 'user', content: `Request ${i}` }],
  })
  if (i >= 10) {
    assert(res.status === 429) // rate limited
  }
}

Mock providers track every call, so you can assert on message history, token counts, and call patterns. This makes it trivial to test complex multi-step agent workflows without spending a cent on API calls.

CI/CD Integration

All of these testing primitives come together in CI. The goal: every pull request runs an eval suite, and the PR is blocked if quality drops below your thresholds.

Here’s a complete GitHub Actions workflow:

name: AI Eval Suite
on:
  pull_request:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - run: npm ci

      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: node scripts/run-eval.mjs

      - name: Post eval report
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const report = fs.readFileSync('eval-report.md', 'utf8');
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: report,
            });

And the eval script that the workflow runs:

// scripts/run-eval.mjs
import { runEvalSuite, formatEvalReport, createScorer } from '@elsium-ai/testing'
import { createAgent } from '@elsium-ai/agents'
import { writeFileSync } from 'node:fs'

const agent = createAgent({
  model: 'claude-sonnet-4-6',
  systemPrompt: 'You are a technical documentation writer.',
})

const accuracy = createScorer({
  name: 'accuracy',
  model: 'claude-sonnet-4-6',
  scale: [1, 5],
  rubric: '5: Perfect. 4: Minor issues. 3: Some errors. 2: Significant errors. 1: Wrong.',
})

const results = await runEvalSuite({
  name: 'ci-eval',
  agent,
  scorers: [accuracy],
  cases: [
    { input: 'Explain WebSocket authentication best practices.' },
    { input: 'Write a migration guide from REST to GraphQL.' },
    { input: 'Document error handling patterns in TypeScript.' },
  ],
  thresholds: { accuracy: 3.5 },
})

const report = formatEvalReport(results)
writeFileSync('eval-report.md', report)

if (!results.passed) {
  console.error('Eval suite failed: quality below threshold')
  process.exit(1)
}

When the eval suite fails, the PR gets a comment with the full report — per-case scores, averages, and which thresholds were missed. Developers can see exactly what regressed and fix it before merging.

Testing Strategy

Here’s a complete testing strategy for AI systems, mapped to the tools and packages that support each layer:

The key insight is layering. Mock providers give you fast, free, deterministic tests for unit and integration testing. Regression suites catch behavioral regressions before deploy. Eval pipelines track quality over time and gate your CI. Prompt experiments give you data-driven optimization. Together, they form a complete testing strategy for non-deterministic systems.

Check out the @elsium-ai/testing docs for the full API reference, or explore 30 more products you can build with ElsiumAI.