Qualitative findings are the easy part. The output that actually matters — and the one I trust an LLM with least — is a number. A figure someone will read, believe, and make a real decision on.
A wrong number delivered with confidence is the worst thing this system could produce. Worse than "I don't know," worse than an obvious error. So I built the number-making to be the least AI-driven part of the whole system.
The division of labor I settled on:
- The model classifies and sizes. It's good at "this is a large effort of this type, at this severity." That's judgment over evidence — the thing it's actually for.
- The math is deterministic. The model never emits the number. A transparent, rule-based model turns the classification into a figure. Same inputs, same assumptions, same result — every time. No creative arithmetic.
- Every assumption is named and overridable. The rate, the effort tiers, the parameters — all explicit, all stored, all editable. Change an assumption and the numbers recompute. Nothing is buried in a prompt.
The test I care most about: same inputs plus same assumptions must produce byte- identical numbers, run after run. If an estimate can't be reproduced, it can't be defended — and if it can't be defended, it's worthless in the kind of room this ends up in.
The payoff is that every figure has a trail: number → the assumptions behind it → the finding it rests on → the exact source evidence. A skeptic can walk the whole chain. That auditability is the product. The confident-sounding guess is the anti-product.
You don't earn trust with a better model. You earn it by making the output impossible to fake and easy to check.
More as it ships. I'm logging the whole build in my Lab.