Vis Series
VisCompile
Behavioral diff tool for AI agents. Deterministic canonical snapshots of transcripts compared by stable case ID - the CI-friendly 'did behavior change?' gate.
Quick start
Start with the primary command, then continue with the full workflow below.
git clone https://github.com/LatticeAG/viscompile.gitProblem
Prompt edits, model swaps, and tool updates silently change agent behavior - and teams usually find out in production. An agent that answered correctly yesterday answers differently today after someone tweaked a system prompt, bumped the model version, or changed a tool's argument schema, and nobody knows which change did it. Manual eyeballing does not scale: a ten-case session produces thousands of tool-call events, and two runs of the same prompt are never byte-identical even when behavior is unchanged. What teams need is a deterministic answer to did behavior change before deploy - a way to pin today's behavior to a canonical, reproducible form and diff it against tomorrow's, with a classification that separates real regressions from benign noise. Existing eval harnesses score answers against expected outputs, but they do not compare whole behavioral traces structurally.
Solution
VisCompile turns recorded agent transcripts into deterministic canonical snapshots and compares them by stable case ID. It is an offline Rust CLI (installed as lattice) that never executes an agent and never sends a network request. Input is a versioned transcript document with named cases; the CLI validates it against a bundled JSON Schema, normalizes key order and line endings, and produces a byte-stable canonical snapshot plus a SHA-256 digest you can check into git. The diff command compares baseline and target - raw transcripts or snapshots - and classifies each case as added, removed, changed, unchanged, regression, or improvement, with an optional evidence comparison using JSON Pointers, tool-call alignment, and closed why-codes. --fail-on-regression exits with status 2 when any regression is found, which makes it a CI gate: the pipeline fails when agent behavior changes in a way the team did not intend.
How it works
Build the CLI from this repository: cargo install --path . (or cargo build --release) - the installed executable is lattice; MSRV is Rust 1.85. Note: crates.io publishing is pending, so cargo install viscompile is not yet available.
Prepare a versioned transcript JSON document: the document carries a schema version and one or more named cases, each with an id, an input, and an ordered event list ending in exactly one final or error event - events include tool_call, tool_result, and final output.
The compiler validates the document against the bundled JSON Schema and semantic invariants: it rejects empty case lists, duplicate or empty IDs, unknown event types, malformed terminal sequences, and invalid JSON-encoded tool arguments. Binary tool payloads are preserved as JSON representations such as base64 strings.
Run lattice compile --input run.json --out baseline.snapshot.json (or lattice snapshot, a compatibility alias). The compiler sorts cases and object keys deterministically, normalizes CRLF line endings, retains array and tool-call order, and adds no generated metadata - producing a byte-stable canonical snapshot.
With --digest-out baseline.sha256, the compiler writes a SHA-256 digest of the canonical snapshot bytes; check the snapshot and digest into git to pin today's agent behavior to a verifiable fingerprint.
After any change - prompt edit, model swap, tool update - run lattice diff --baseline baseline.snapshot.json --target after.transcript.json. Case IDs are the diff identity: a target-only case is added coverage, a baseline-only case is a regression, and changed final output, error, or ordered tool-call behavior is conservatively classified as a regression. A baseline error resolved to a final output is an improvement.
In CI, run the diff with --fail-on-regression and optionally --require-baseline-digest sha256:<hex> to verify the snapshot was not tampered with. A clean diff exits 0; any regression exits 2 and fails the pipeline.
For deeper analysis, --report comparison writes a latticeag.viscompile.comparison document with JSON Pointers, tool-call alignment, and closed why-codes naming the structural delta, and lattice graph exports a deterministic Graphviz DOT graph. Reports render as human-readable text, machine-readable JSON on stdout, or a self-contained HTML document; all commands support stdin/stdout with -.
Technical architecture
Each handoff carries structured context through the product's execution path. Hover a node to inspect its role.
01
Transcript Ingest
02
Schema Validation
03
Canonical Compile
04
Digest Pinning
05
Baseline Diff
06
Case Classification
07
Regression Gate
08
Report & Graph Export
When to use
- Before deploying prompt changes to production agents - run a baseline diff and gate on regressions.
- When swapping models or upgrading a tool SDK - verify behavior is preserved against the pinned snapshot.
- CI/CD pipelines that need a deterministic did-behavior-change gate with a non-zero exit on regression.
- Auditing agent behavior drift across versions - classify each case as regression, improvement, or unchanged.
- Shell pipelines that record agent transcripts and stream them straight into a canonical snapshot.
Not for
- Real-time monitoring - VisCompile is a pre-deploy diff tool, not a runtime monitor.
- Agents with no defined expected behavior (creative or exploratory tasks) - there is no baseline to diff against.
- Recovering hidden reasoning - comparison names structural deltas in the recording; it does not recover chain-of-thought the transcript never recorded.
Features
Deterministic canonical snapshots - byte-stable across runs, key-order and line-ending normalized, no generated metadata
SHA-256 digest pinning - check the digest into git and verify it with --require-baseline-digest
Stable case-ID diff identity - added, removed, changed, unchanged, regression, and improvement classification
CI-friendly regression gate - --fail-on-regression exits 2 when any regression is found
Versioned transcript format - one or more named cases, each ending in exactly one final or error event
JSON Schema validation - rejects malformed documents, duplicate IDs, unknown event types, and bad terminal sequences
Evidence comparison mode - JSON Pointers, tool-call alignment, and closed why-codes for the structural delta
Graphviz DOT export - deterministic graph of compared cases for visual inspection
Text, JSON, and HTML reports - colorized terminal output, machine-readable stdout, standalone CI artifacts
Shell-pipeline friendly - stdin/stdout with -, --verbose/--quiet flags, 64 MiB per-document cap
Offline by design - never executes an agent and never sends a network request
MIT licensed - buildable from source as the lattice binary (crates.io publish pending)
Install and usage
# Build from source (binary name: lattice) - crates.io publish pending
git clone https://github.com/LatticeAG/viscompile.git
cd viscompile
cargo install --path .
lattice --help
# Or build a release binary
cargo build --release
./target/release/lattice --help
# Compile a recording into a canonical snapshot + digest
lattice compile --input run.json --out baseline.snapshot.json --digest-out baseline.sha256
# After a prompt edit or model swap, diff against the baseline
lattice diff \
--baseline baseline.snapshot.json \
--target after.transcript.json \
--format json \
--out behavior-diff.json \
--fail-on-regression
# Stream a recording straight into a snapshot via stdin
agent-recorder --json | lattice compile --quiet --input - --out baseline.snapshot.json
# Export a Graphviz graph of the comparison
lattice graph --baseline before.snapshot.json --target after.transcript.json --quiet > changes.dotArchitecture explorer
Problem
Prompt edits, model swaps, and tool updates silently change agent behavior - and teams usually find out in production. An agent that answered correctly yesterday answers differently today after someone tweaked a system prompt, bumped the model version, or changed a tool's argument schema, and nobody knows which change did it. Manual eyeballing does not scale: a ten-case session produces thousands of tool-call events, and two runs of the same prompt are never byte-identical even when behavior is unchanged. What teams need is a deterministic answer to did behavior change before deploy - a way to pin today's behavior to a canonical, reproducible form and diff it against tomorrow's, with a classification that separates real regressions from benign noise. Existing eval harnesses score answers against expected outputs, but they do not compare whole behavioral traces structurally.
Solution
VisCompile turns recorded agent transcripts into deterministic canonical snapshots and compares them by stable case ID. It is an offline Rust CLI (installed as lattice) that never executes an agent and never sends a network request. Input is a versioned transcript document with named cases; the CLI validates it against a bundled JSON Schema, normalizes key order and line endings, and produces a byte-stable canonical snapshot plus a SHA-256 digest you can check into git. The diff command compares baseline and target - raw transcripts or snapshots - and classifies each case as added, removed, changed, unchanged, regression, or improvement, with an optional evidence comparison using JSON Pointers, tool-call alignment, and closed why-codes. --fail-on-regression exits with status 2 when any regression is found, which makes it a CI gate: the pipeline fails when agent behavior changes in a way the team did not intend.