PRACTICE

Model-swap evaluation matrix

Before swapping the model behind Argus's vision auto-tagging, I ran a live 5-model matrix against a golden set: tag recall, LLM-judge score, red-team pass rate, cost per run, latency. Total spend: $0.56.

The challenger beat production on every axis at once: +10.7 points tag recall — above the run-to-run noise floor of roughly 7–8 points — 40% cheaper, four times faster. The quality-ceiling model scored highest but truncated its output at the production max_tokens cap of 1024 tokens, so it needed a contract change before it could ship. That's the kind of failure a spot-check misses and a fixed-threshold eval catches automatically.

Four-gate build with an independent QC catch

A four-gate build ported a web app's surfaces to native macOS, one gate per surface, chained so no gate started until the one before it was approved. Test coverage grew 26 → 29 → 58 → 60 → 66 across the four gates. Every gate passed CI. One didn't pass review.

On gate three, an independent QC pass — run from a clean checkout against the live production data, not against the ship comment — caught a data-parity defect invisible to both the tests and the visual render: the native app was grouping issues from a row-capped list and showing 132 issues where the source system showed 270. The fix was a scoped server-side fetch; the corrected count matched exactly at 269. The gate stayed blocked until an independent re-verification confirmed the fix, then a named human approved it.

Weekly literature scan with a re-verify step

A weekly research routine scans for new evaluation papers and tooling — arXiv, plus tools like Inspect, promptfoo, Braintrust — and posts a digest. Two rules keep it honest: a strict quiet-week ceiling (5–8 bullets is a maximum, not a quota — a thin week gets a one-line "nothing significant" instead of padding), and every claimed source gets independently re-verified before it's published, because a first pass through an external API silently returned empty results and would have shipped a false "not found" if I'd trusted it.

The merge gate rubric

Every pull request clears the same four-part test before an agent can merge it: CI is green, the work traces to a Founder-approved ticket within its approved scope, no unresolved review comments remain, and the change doesn't fall into a Founder-gated category. Auth and secrets, spend, public content, and data or schema changes are hard stops — no agent clears those, ever, regardless of how clean the diff looks. Everything else routes to the agent that owns the relevant judgment — a design call to the person who owns taste, an architecture call to the person who owns architecture — and that approval only counts once it's recorded as a decision on the ticket, not asserted in a comment.

One thing in flight, one model per task

Two standing constraints on how the work itself gets done, not just reviewed. First: one active build at a time per agent — a second task that comes in gets a one-line "queued behind [ticket]" and waits, because four concurrent sessions on one codebase contend for the same working tree and stall each other out rather than shipping faster. Second: a model-dispatch rule — the senior model is reserved for judgment calls (architecture, security, debugging), and mechanical work (file sweeps, formatting, running scripts) gets dispatched to a cheaper model explicitly. Ten or more consecutive mechanical tool calls on the senior model is a tripwire to stop and delegate.