Portable regression tests

Turn a verified memory incident into executable retrieval, context, and answer assertions.

A verified incident exports an engram.memory-regression artifact. Version 2 declares semantic lifecycle and answer assertions plus a controlled perturbation matrix, without claiming access to hidden model reasoning.

Run with the Studio executor

Your project owns the executor boundary. The recommended LangGraph setup puts that module in engram.config.json, so Studio and CI import the same code:

{
  "version": 1,
  "framework": "langgraph",
  "executor": "engram.executor.mjs",
  "regressions": ["regressions"]
}

For custom integrations, an executor can also be a legacy function:

export default async function run({ source, variant }) {
  const result = await runAgentVersionUnderTest({
    memories: source.memoryState.before,
    userMessage: source.input
  });

  return {
    variantId: variant.id,
    memories: result.memories,
    answer: result.answer,
    selectedMemoryIds: result.selectedMemoryIds,
    loadedMemoryIds: result.loadedMemoryIds,
    forcedMemoryIds: result.forcedMemoryIds ?? []
  };
}

Execute the artifact:

npx --yes @engramviz/cli test incident.engram-test.json \
  --executor ./engram-regression.mjs \
  --format pretty

With project configuration, omit both paths and run the entire suite:

npx engram test

The command invokes the executor once per matrix variant and exits nonzero when any required selection, exclusion, context load, or answer assertion fails. Failed findings identify the variant and show expected and observed values.

Check a recorded observation

When another system already produced the matrix results, compare the observation array directly:

npx --yes @engramviz/cli test incident.engram-test.json \
  --observation ./latest-observation.json

Run in CI

- run: npm ci
- run: >-
    npx engram test
    --format github
    --output engram-regression-report.json
- if: always()
  uses: actions/upload-artifact@v7
  with:
    name: engram-memory-regression
    path: engram-regression-report.json

The repository includes a complete copyable workflow at examples/github-actions/memory-regression.yml.

Version 1 artifacts and their ID-based executor contract remain supported. A version 2 artifact proves that every supplied matrix variant passed observable semantic assertions. Production equivalence exists only when the executor calls the production-equivalent retrieval and generation stack.