Field notes / AI systems · Level 5 of 5

Evals and Cost Control, or How to Know It Is Still Right

Your system runs every morning without you. Congratulations, and now the harder question: how do you know this morning's judgment was as good as last month's? In software this is a solved problem. Here it is in operator terms.

A thing nobody tells you when you automate judgment: the vendor changes the model underneath you. Not on your schedule, usually without you noticing, sometimes making your results better, occasionally making them worse in ways that look fine sentence by sentence. Add your own prompt edits, each fixing the case in front of you while quietly bending three cases you did not retest, and you get drift: a system that still runs perfectly and has slowly stopped thinking like you.

At Level 4 drift is invisible, because logs prove the system ran, not that it was right. Level 5 is the rung where you can prove it was right, and where the bill gets cut in half as a side effect. It is the least glamorous post in this series and the one that separates an automation from an asset.

The eval, in operator terms

An eval is a set of cases where you already know the right answer, kept fixed, re-run against the system whenever anything changes. That is the whole concept. If the system used to get 27 of 30 right and today it gets 22, something moved, and you found out from your test set on Tuesday instead of from a capital partner in three weeks.

You already run evals on humans. A new acquisitions VA does not touch live offers on day one; you give them deals you have already judged and compare answers. The only difference here is that the machine takes thirty seconds per exam and never resents being tested.

Build your golden set in an afternoon

If you kept the Triage Log from Level 3, most of the work is done: it is a record of real inputs and the decisions you actually stood behind.

1. Pull 20-30 real cases: emails your triage handled, deals your buy box screened. Real ones, with the mess left in.
2. For each, write the verdict you defend today: the priority it deserved, the PURSUE / DIG / PASS it should have gotten. That judgment is the gold.
3. Make the mix adversarial, not flattering. A third obvious calls, a third judgment calls, and a third the ugly ones: the seller who buried the real number in paragraph six, the wholesaler deal that looks juicy and fails a dealbreaker, the P3-looking email that was actually a lender deadline.
4. Store as files: input, expected verdict, one line on why. A folder of text files or one spreadsheet, nothing fancier.
5. Add every future miss. Each one is a free test case with a documented wrong answer; the set gets meaner as your system gets better, which is exactly the arms race you want.

When to run it, what to do when it dips

Run the set through your system and count matches. At Level 5 scale this is a small script (the same claude -p pattern from Level 4 handles it, and tools like promptfoo exist when you outgrow that); truthfully, you can start by pasting ten cases into a project once a month and eyeballing the verdicts. A manual eval beats no eval by roughly the same margin that a written buy box beats vibes.

Run it: before any prompt or document change ships, when the vendor announces a model update, and monthly as a floor because they do not always announce. Log the score with the date, prompt version, and model version.

When the score dips, the eval tells you where: which cases flipped, in which direction. Wrong on new edge cases means your documents have a gap to patch. Wrong on cases it used to nail after a model update means you test alternatives and switch deliberately instead of absorbing the regression. Either way you are choosing, not drifting; that is the entire dignity of this level.

Model routing: the $40 bill that was $900

Model pricing spans two orders of magnitude, and most automations pay flagship prices for commodity work. Sorting four hundred emails into P1 through IGNORE is commodity work; a small fast model (Haiku-class, and every vendor has one) does it at pennies per thousand. Deciding whether an eight-word seller reply means the deal is alive is judgment; that is what the expensive model is for.

So you split the pipeline: the cheap model reads everything and extracts and sorts, and only the items that clear the bar reach the expensive model for verdicts, maybe five percent of volume. Same or better quality, roughly a tenth to a twentieth of the spend. At real volumes that is a $900 monthly bill becoming $40-90, and here is the part that makes the whole post cohere: your eval is what makes routing safe. Run the golden set through the cheap model for the sorting stage; if it scores the same as the expensive one there, the savings are free money. Without the eval, cheaping out is a guess. With it, it is an engineering decision.

Versioned prompts and the audit trail

Three habits finish the level, each cheap, each the difference between an incident and a shrug:

1. Version everything the model reads. Buy box, triage rules, prompts: dated versions, old ones kept (git if you are comfortable, dated file copies if not). When Tuesday goes weird, "what changed" takes one minute, and rollback is a file swap.
2. Log every decision with its inputs. Not just what ran: what it read, what it decided, which prompt and model versions decided it. When a partner asks why the system passed on the fourplex in March, you reconstruct it instead of shrugging. Anything touching offers, wires, or partner reporting gets this without exception.
3. Scope who sees what. Your buyers list, your seller PII, your margins are now in documents that systems and staff can read. The dispo VA needs buyer criteria, not your spreads. Takes an hour to set up; reputationally priceless.

The asset test

Walk the ladder back for a second. Level 1 was you and a browser tab. Now: criteria written and versioned, judgment tested monthly against gold, work running on schedule under business-owned accounts, every decision reconstructable, costs known per run. Hand a new hire the documents and they run it in a week. Sell the company and it transfers with the deal.

That was the definition of an asset from the first post, and the vacation test finally passes: two weeks dark, the work happens, failures page somebody who is not you, and the eval catches what failure alerts cannot. One caveat stands: even Level 5 needs an owner. Somebody reviews the eval scores, patches the documents, decides when to switch models. The system survives you leaving for two weeks. It does not survive nobody caring for two years.

The whole series stands on one idea: the ladder is not a maturity contest, it is a costed decision at every rung. Most operators should climb to excellent Level 2 or 3 and stop, proudly. The ones with real deal flow, a team, and capital partners should know that Levels 4 and 5 are not magic, just infrastructure, and now you have seen every piece of it.

The series, complete

1. 00The Five Levels of AI in a Real Estate BusinessThe map
2. 01Getting real value out of a chat windowPublished
3. 02Building a buy box that the model actually followsPublished
4. 03Wiring an assistant into your inbox, sheets, and CRMPublished
5. 04Getting it off your laptop: hosting, logging, and alertsPublished
6. 05Evals and cost control, or how to know it is still rightYou are here