wai
Technical · August 2026
A pass or fail line turned out to be the least useful part of debugging. The hypotheses, the one-variable experiments that killed them, the diffs between runs, had no record at all. wai keeps those as files in the repo.
And I'll plug stuff into this as I go. Now I kinda want this to be true for all model types. Not just llms but anything plug and play: data in, black box, delta out. wai sticks to any of these.
investigations as files
Technical
An investigation is one JSON file in the repo:
| field | what it holds |
|---|---|
observed / expected | the behavior gap being investigated |
hypotheses[] | every explanation considered, its status, and the evidence for and against it |
experiments[] | one changed variable each, tied to a baseline run and a result run |
conclusion | what held, with confidence as a word: hunch, likely, or confident |
Runs are files too. wai run --set retrieval=oracle -- python eval.py executes a command and records its output, exit code, and declared variables. wai diff compares two runs on config, checks, exit codes, and output. An experiment's result lands on the hypothesis it tested, rejected explanations keep their evidence, and wai search reads it all back.