Your model is not your engineering strategy
The expensive part of AI-generated code is often the work it creates for somebody else.
The implementation arrives quickly. Then a senior engineer has to reconstruct its assumptions. A reviewer discovers that the tests agree with the implementation because both misunderstood the requirement. An operator eventually learns what "done" left out.
Generate twice as much code under those conditions and you may just double the review queue. Calling that productivity does not make it cheaper to ship software.
I want the speed AI gives engineers. I use it myself. But I am increasingly impatient with an approach to adoption that amounts to buying licenses, choosing a model, and asking people to move faster. That leaves engineers responsible for the result without doing much to improve their chances of getting it right.
As an engineering leader, I own more than the choice of model. I own whether the work has a clear purpose, whether the agent can test it in a real environment, whether decisions survive a session boundary, and whether we have credible evidence that the result works. Those are management and engineering responsibilities. They do not disappear into a prompt.
That was the argument behind my KCDC talk, Don't Outsource Your Judgment. AI can make bad engineering decisions cheaper, faster, and harder to notice. Our job is to make correct decisions easier to carry through to working software.
I use five principles to do that: Define, Equip, Remember, Verify, and Learn. Here is what they looked like in practice.
We used a game because the rules fit in a sentence
Tile Rush: tap an open tile, claim it, earn a point. Highest score wins the round. Then the board resets.
The review prototype. A deliberately small game, with ordinary stateful-service problems.
Two people can claim the same tile at once. A server can save a claim and lose the response. A waiting request can outlive the round it belongs to. The product is simple; the promises still have to hold.
We built two paths with the same requirements, tools, and GPT-5.5 model at medium reasoning effort. Standard let the agent choose its working process. Guided prescribed the five principles. Construction ran in isolated environments with fresh sessions; files and database state survived between sessions. We recorded the work and evaluated the submitted artifacts independently.
During the talk, a gateway assigned each browser to one backend and kept it there. Each backend had its own database, and a separate observer checked ownership and scores. People played deployed software. There was no model generating code behind their clicks.
Before the talk: build and check the artifacts locally. During the talk: run those artifacts on Fly.io, retain request and correctness evidence, and feed aggregate metrics to the slides.
I will get to the numbers. First, the engineering decisions that made those numbers worth collecting.
1. Define: be precise about what must be true
A long spec can still leave the important question unanswered: what is this system allowed to get wrong?
For Tile Rush, the answer includes ownership and scoring. Two successful-looking responses cannot both entitle different players to the same tile. A retry cannot award another point because the client missed the first response.
A useful working spec could be this short:
Mission: Give the room a quick competition with trustworthy scores. Boundaries: One owner per tile per round; one point per owned tile. Recovery: A retry returns the original result and never awards twice. Scope: Implement claiming now; defer the leaderboard to the next slice. Risk: Two requests may both observe that the same tile is available. First evidence: Competing claims produce one durable owner and one point.
That leaves the agent room to choose an implementation. It does not leave room to redefine success when the implementation becomes inconvenient. Detailed API and operating contracts still matter; this is the short reference that keeps their purpose available during the work.
We learned to check when that reference appeared. In an early exploratory build, the durable decision record arrived near the end of the session. It documented work it had never been available to guide. Across the five development runs that followed, the initial record appeared before the first application edit. We had improved the sequence, even though we had not eliminated implementation mistakes.
This is where I want an engineer to intervene. If satisfying a recovery requirement seems to require breaking an invariant, resolve that conflict explicitly. More implementation effort will not resolve a disagreement about what the system owes its users.
Measure: Corrections caused by violating an already-stated boundary, and whether that boundary was available before the consequential edit. Keep new requirements separate from forgotten ones. Count spec preparation and clarification as work.
2. Equip: let the agent discover it is wrong cheaply
Telling an agent to test its work is not enough when its environment cannot run the test.
Our builds encountered missing dependencies, database permissions that did not allow an attempted change, and connection limits the implementation exceeded. Those failures needed different responses. A more emphatic instruction to "be careful" would have helped with none of them.
The useful sequence was dependency installation, compilation, migration with the supplied database role, service startup, one claim, then a bounded contention check. Each result gave the next step a meaningful starting point.
In one resumed build, the agent read its handoff, installed dependencies, edited the application, and built successfully. Other sessions read their notes and still tried to build before installing. Successful compilation did not establish application correctness, but the command history let us distinguish a working setup sequence from another avoidable failure.
Give that sequence a repeatable entrypoint in the existing workflow. Report which prerequisite failed. Do not bury the answer in enough output to require another model call just to find it.
There is a limit here: a static review may not need a full runtime environment. Equipping the agent should make the intended work easier, not authorize it to install everything it can find.
Measure: Checks that fail before they can exercise the application, time to the first meaningful integration result, and total setup effort. Keep a readiness check only if the work it prevents is worth the work it adds.
3. Remember: save the decision, including why you made it
An agent can start a fresh session against a database that very much remembers the last one.
For Tile Rush, the next session needed to understand operation identity. A missing response does not mean a claim failed. The same tile coordinate in a new round is not automatically the same operation. Losing either distinction makes a seemingly reasonable retry change dangerous.
A useful handoff would preserve the reasoning and the next check:
Decision: Retain operation identity across retries, including round changes. Reason: A missing response does not establish that the claim failed. Evidence: Link the retry test and the artifact it exercised. Current state: Record what is running and what cleanup removed. Next action: Restore prerequisites, then rerun the relevant check.
An exploratory build did retain the rule that an exact retry returns the stored operation row. The fresh session then read it. That demonstrated continuity: the decision survived and returned to working context. It did not prove that the next implementation respected the decision. We still needed the test.
This is why I care less about whether a team has a memory service than whether its next session can act on what the previous session learned. A local HANDOFF.md may be enough. Someone still has to define when it is updated, how the next session loads it, and how stale statements are checked against the workspace. Distributed memory adds ownership, versioning, and conflict problems; it does not excuse them.
There is nothing advanced about paying a model to rediscover a rule your team already knows.
Measure: Repeated discovery of known prerequisites, recurrence of previously diagnosed mistakes, and time from resumption to useful work. A handoff file being written and read is evidence of the mechanism, not evidence of the benefit.
4. Verify: make the test disagree when the code is wrong
The most reassuring test suite can be the one that shares all the implementation's mistakes.
For Tile Rush, a second player being rejected is often correct. So is a new claim arriving after the round closes. Counting those outcomes as undifferentiated "errors" would reward the wrong behavior.
The important question is whether the promise held. In a controlled test shown in the talk, a backend committed a claim and the test deliberately dropped the response. It retried immediately, after a process restart, and after the round closed. Each retry returned the same claim ID. Independently read storage showed one owner for the tested tile, and the scoring checks passed.
That was a hand-authored evaluator fixture, not a generated game having an outage. It demonstrated the distinction the evaluator needed to make: recovering an existing answer versus creating another claim.
We took the expected result from the reviewed contract, independently of the implementation. We tied the evidence to the running artifact. A test against yesterday's process does not validate today's build.
The same discipline applies to concurrency. Arrange the race and assert that it occurred. If the behavior depends on a request waiting across a round boundary, prove that it waited. Passing a nearby happy path leaves the difficult question unanswered.
Measure: Required behaviors exercised on the final artifact, defects that escape those checks, and mismatches between tested and deployed versions. At runtime, distinguish expected rejection, transport failure, and broken invariants.
5. Learn: fix the workflow that produced the waste
After the talk, we put these ideas to work on a coaching tool that reads engineering sessions through our AI gateway. One review completed 16 model calls, then failed local validation before it could publish the report.
The model had cited an event it was allowed to use. Our validator incorrectly required the reference to appear in two separate lists.
We could have run the review again and hoped for a response the validator liked. Instead, we inspected the saved requests, responses, evidence identifiers, and attempt history. We fixed the validator, added regression tests, and recovered the completed report with zero additional model calls. The original failed run remained unchanged.
That is a useful outcome: no completed model work had to be repeated. It is also a more useful diagnosis than "the model failed."
Explicit offline recovery is now part of the tool. The next engineer encountering that class of recoverable validation problem does not need to rebuild the same recovery process or pay to repeat an entire review.
Learning should leave a specific improvement in the next workflow. "Be more careful" is an obligation you have handed back to the engineer, not an improvement you have made for them.
Measure: Recurrence of the diagnosed failure, whether its regression remains executable, and recovery cost. Include maintenance of the new guidance or check. Remove it if the overhead outweighs the benefit.
What the game results actually showed
These are the construction totals from the three consecutive runs displayed in the presentation. They are the final subset, recorded after the workflow refinement described above. Standard first, then Guided:
Provider calls: 387, then 287. 26% fewer.
Inference cost: $18.12, then $13.83. 24% lower.
Worker time: 50.31 minutes, then 40.34 minutes. 20% lower.
Parent tool completions: 420, then 329. 22% fewer.
Input tokens, including cached: 20.47M, then 13.72M. 33% fewer.
Output tokens: 138,372, then 122,372. 12% fewer.
This was a small, simple game. Three runs do not establish a causal effect, a company-wide saving, or a percentage attributable to each principle. Worker time excludes preparation and independent evaluation; it is not human engineering time. Parent tool completions exclude child-agent tool actions, and cached input is already included in the input total.
The deployed games also worked. Across the retained live event, including pre-talk activity, we recorded 1,482 claim attempts: 1,144 durable claims and 338 expected rejections. Both deployed games passed independent ownership and scoring checks. We deployed the Run 2 pair, where Standard had been cheaper and faster to build.
I would take this evidence seriously enough to keep testing the workflow. I would not put "24% engineering productivity improvement" into a company plan on its strength.
Bring it back to the engineer's actual work
The game gave us a small system we could explain. The Crash Override session work lets us look for the same problems in less tidy engineering tasks.
In retained sessions, a project-specific TypeScript constraint was diagnosed and corrected, with a passing typecheck, then encountered again in a later related helper. Other isolated workspaces lacked the tools needed for the intended tests. Those are specific opportunities: carry forward a checked pattern, or establish readiness in the actual checkout before attempting runtime validation.
A useful coaching report should show the engineer the message and tool result behind its recommendation. It should explain what to change and offer a usable draft spec entry, handoff, checklist, or skill. Our reports retain the original gateway event and session identifiers so the advice can be inspected. An impact ranking helps choose what to try first; its percentages are planning estimates, not measured savings from these sessions.
That is the same premise as everything else we build here: observed, not inferred. Build systems don't hallucinate, and neither do retained session records.
We start in observe, report, suggest mode. The engineer chooses the session and approves any model upload. The tool does not install instructions or rearrange their environment. Advice that cannot survive the engineer examining its evidence has not earned the right to alter their workflow.
The zero-call recovery is measured value. Broader claims about time and money saved need follow-through on later work. I want to turn each recommendation into a testable proposition, rather than another document telling engineers how they ought to behave.
Give your team a better target
Start with one recurring problem. Pick one change to address it. On comparable tasks, record whether the advice was used, whether the result was accepted, and whether the problem recurred. Keep task type and model and harness versions visible so a change in the workload does not masquerade as an improvement.
The useful unit is an accepted change, with its full cost attached. Count model and tool charges, coaching, failed attempts, preparation, review, and recovery. Record engineer active time separately from elapsed time. Check that regressions and relevant runtime failures did not increase.
If the agent uses fewer tokens but reviewers spend longer correcting the result, find out where the work went. If a handoff costs less to maintain than the repeated investigation it prevents, keep it. If you cannot tell whether it does, collect the evidence before making it mandatory. Those are the tradeoffs an engineering leader should be able to discuss.
I am not asking engineers to become better supervisors of an endless stream of generated code. I want us to build working conditions in which more of that code deserves to survive review.
Define the promise. Equip the agent to test it. Remember the decisions. Verify the delivered behavior. Learn enough to make the next attempt better.
The model can do much more of the implementation. We still have to decide what is worth shipping, and we still own what happens after we do.
