Forge Series

ForgeDistill

Open Source

Correctness-by-construction distillation harness for agentic tool-calling models. Deterministic tool-call chains, teacher writes prose only, grounding gates. (MIT)

Quick start

Start with the primary command, then continue with the full workflow below.

git clone https://github.com/LatticeAG/ForgeDistill.git

Problem

Synthetic agentic training data is almost always structurally broken. Most distillation frameworks ask a teacher model to demonstrate correct agentic behavior end to end - chain the tool calls, then write the reasoning - and most models cannot do that reliably. The result is the industry-wide pattern: shallow one-call trajectories, fabricated tool arguments (invented emails, hallucinated IDs), malformed JSON, and dependencies that were never actually enforced. Even the strongest frameworks verify after generation: distilabel leaves correctness to your own pipeline steps, APIGen and xLAM verify post-hoc and throw away the garbage, Glaive is mostly single-turn with no dependency enforcement, and AgentInstruct has no tool-dependency correctness. Every approach spends frontier-expensive teacher tokens to generate data that then fails its own checks. The teacher is doing the structural work it is worst at, and the harness is doing the verification work it should not need to do.

Solution

ForgeDistill is a correctness-by-construction distillation harness for agentic tool-calling models. It inverts the pipeline: Phase 1 builds the tool-call chain deterministically - agentic_plans.py picks from 47 plan templates across 15 skill tags (multi_hop, branch, recovery, join, fanout, reorder, digest, schema, idempotent, disambiguate, stop, and more), fills variables, resolves $S.result references against prior step results, and executes the trajectory against a deterministic mock executor. Sequential dependencies are unskippable by construction: send_email(to="$0.result.email") forces the teacher to use the opaque address only obtainable from a prior get_user call. Phase 2 spends teacher tokens only on prose: the teacher receives the full execution record and writes N thought blocks and a FINAL_ANSWER grounded in the real tool results - it never emits tool calls, so malformed JSON is impossible. Format and semantic grounding gates reject fabricated values before a trace is exported as reversed-v2 training JSONL. The result: structurally-perfect, prose-grounded training traces from any OpenAI-compatible teacher endpoint, verified at 100% gate pass rates across a 500-trace audit with zero nudge leaks and zero malformed tool calls.

How it works

Clone the repo, create a venv, and install the package (pip install -e ".[dev]"). The harness exposes console scripts - distill, eval_card, export_sft - plus direct modules like agentic_plans.py and prose_writer.py.

Configure a multi-provider OpenAI-compatible roster (cp configs/roster.example.yaml configs/roster.yaml) with a base_url and key_env per provider. Keys live in environment variables only - the roster references them by name, and a per-provider health state machine (healthy / backoff / quarantined) manages the fleet.

Phase 1 runs with zero teacher tokens: agentic_plans.py picks a plan template, fills variable pools, and executes the chain against a deterministic mock executor (11 shared tools). $S.result.field references create real dependencies - a later call's argument must come from an earlier call's result, so guessing fails with a 400.

Error-recovery plans deliberately fail a first step (bad id, bad city) before a corrected retry, teaching the observe-error-and-retry loop. Plans are tiered (easy 9, medium 12, hard 21, expert 5) and mixed via --curriculum {off,uniform,linear}; a --holdout-frac 0.15 split writes holdout_plan_ids.json for later evaluation.

Phase 2 spends the only teacher tokens: the teacher receives the full execution record and writes ONLY thought blocks (reasoning before each call, including recovery thoughts) and a FINAL_ANSWER grounded in the real tool results. Because the teacher never emits tool calls, malformed JSON is impossible by construction.

Format and grounding gates validate each assembled trace: validate_prose_trace checks the prose contract, and validate_answer_grounding rejects final answers that fabricate values not present in the real tool results. A trajectory-hash dedup keeps prompts unique, and the trace is tagged distill_version: reversed-v2 with an eval block recording gate results, n_rounds, n_tool_calls, and skills.

Audit output on disk, not stdout: traces land in data/raw/traces_<provider>.jsonl, and eval_card.py --require-gates produces a structural audit card. The harness refuses to overwrite existing data and never deletes - archive_data.py moves runs to data/archive/<timestamp>_<label>/ - and distill_tools.py refuses to run over existing traces without an explicit --wipe.

export_sft.py renders SFT training JSONL with template-driven assistant-only loss masks (nanbeige, chatml templates, --check-mask validates spans), and dpo_pairs.py builds offline preference pairs from assembled traces. Publishing pushes the dataset and eval card to Hugging Face (LatticeAG/ForgeDistill-agentic).

Technical architecture

Each handoff carries structured context through the product's execution path. Hover a node to inspect its role.

Roster Config

Step 1: Roster Config receives structured context from the prior stage and prepares it for the next handoff.

Phase 1 Chain Build

Step 2: Phase 1 Chain Build receives structured context from the prior stage and prepares it for the next handoff.

Deterministic Execution

Step 3: Deterministic Execution receives structured context from the prior stage and prepares it for the next handoff.

Phase 2 Teacher Prose

Step 4: Phase 2 Teacher Prose receives structured context from the prior stage and prepares it for the next handoff.

Format & Grounding Gates

Step 5: Format & Grounding Gates receives structured context from the prior stage and prepares it for the next handoff.

Trace Assembly

Step 6: Trace Assembly receives structured context from the prior stage and prepares it for the next handoff.

Structural Audit

Step 7: Structural Audit receives structured context from the prior stage and prepares it for the next handoff.

SFT / DPO Export

Step 8: SFT / DPO Export receives structured context from the prior stage and prepares it for the next handoff.

When to use

Generating training data for agentic tool-calling models that need real multi-turn, sequentially dependent tool calls - not shallow one-call demos.
Teams with weak or cost-sensitive teachers that still need structurally-correct traces - prose writing is the only teacher spend.
Building fine-tuning datasets where fabricated values, malformed JSON, or unenforced dependencies have poisoned previous runs.
Scaling synthetic data generation across a multi-provider model fleet with per-provider health, quarantine, and resume semantics.
Creating SFT or DPO training sets for open-weight agentic models, with assistant-only loss masking and holdout splits for evaluation.

Not for

General-purpose synthetic text generation - ForgeDistill is scoped to agentic tool-calling trajectories with deterministic structure.
Replacing distilabel or general pipeline frameworks where you need arbitrary custom generation steps.
Evaluating already-trained models - evaluation hooks exist but the harness's core job is data generation.

Features

Correctness by construction - tool-call chains are structurally guaranteed before a single teacher token is spent

47 plan templates across 15 skill tags - multi_hop, branch, recovery, join, fanout, reorder, digest, schema, idempotent, disambiguate, stop, and more

Unskippable dependencies - $S.result.field refs force learned values like opaque emails and exact plan strings through prior tool results

Error-recovery plans - deliberate first-step failure followed by correction teaches the observe-error-and-retry loop

Teacher writes prose only - never emits tool calls, so malformed JSON is impossible

Semantic grounding gates - reject final answers that fabricate values not present in real tool results

Teacher-agnostic fleet - any OpenAI-compatible endpoint, weak or strong; the harness chunks the structural work

Per-provider health state machine - healthy / backoff / quarantined, 429 quarantine, Retry-After honoring, exponential backoff with jitter

Fleet management - per-provider semaphores, concurrency scaling, weighted model sampling, dead-route re-probing

Trajectory-hash dedup and per-provider checkpoint/resume with per-worker RNG and token accounting

Curriculum sampling - uniform or linear tier mixes across easy/medium/hard/expert tiers

Safety-first data handling - never overwrites existing data, archive-before-run, refuses --wipe unless explicit

Structural eval card - eval_card.py --require-gates audits every gate pass rate from on-disk traces

SFT export with assistant-only loss masks - nanbeige and chatml templates with --check-mask span validation

Offline DPO pair builder - dpo_pairs.py creates preference pairs from assembled traces

500-trace verified audit - prose, grounding, chain, and dependency fidelity gates all at 1.0, zero nudge leaks, zero malformed tool calls

Install and usage

usage.sh

# 1. Clone and set up
git clone https://github.com/LatticeAG/ForgeDistill.git
cd ForgeDistill
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# 2. Configure your multi-provider OpenAI-compatible roster
cp configs/roster.example.yaml configs/roster.yaml
#    - set base_url / key_env per provider
#    - export your keys, e.g. export MY_PROVIDER_KEY=sk-...

# 3. Sanity check the modules
python -c "import distill_tools, agentic_plans, prose_writer; print('OK')"

# 4. Pilot run (10 traces), then scale
ulimit -n 65536
distill --count 10 --pilot

# 5. Audit the output on disk (don't trust stdout)
cat data/raw/traces_*.jsonl | python -m json.tool --json-lines | head -20

# 6. Structural audit card
python src/eval_card.py --input data/raw --out data/raw/eval_card.json --require-gates

# 7. Export SFT jsonl with assistant-only loss masks
python src/export_sft.py \
  --input data/raw \
  --out data/export/nanbeige.jsonl \
  --template configs/templates/nanbeige.json \
  --format messages \
  --check-mask

Architecture explorer

Problem

Solution

View on GitHub