The process we wrote was part of the problem
We built a detailed way of working with AI. We are producing more than ever, and understanding less of it.
Last quarter we wrote our engineering standards. Part of that was a fairly detailed process for how we use AI in our engineering practice. Five stages, each with a clear role for the agents and a clear role for the people.
A few weeks ago I told the team we were moving away from it.
The principles still hold. The process wasn’t working, and parts of it were making things worse.
What we were seeing
It started with my most senior engineers. They weren’t complaining about workload. They were worried about the quality of the work, and about something harder to name: a sense that we were losing our grasp of what we were actually building.
Once I started looking, the signs were everywhere. Large backlogs of Claude-generated tickets for work that was never going to get done. Ticket and pull request descriptions far longer than anyone needed to read. Solutions that handled every edge case anyone could imagine, including the ones that will never happen.
The data showed what was happening to our engineering flow. Compared with the year before we rolled out AI coding tools, we are merging about twice as many pull requests a month. But the typical pull request has gone from 40 lines to 165, and more than a quarter are now over 500. The median time from first commit to merge has doubled, from 16 hours to 32. A quarter of pull requests now wait more than two and a half days for a first review. That used to be four hours. And engineers used to have one or two pull requests open at a time. The average is now 4.4.
We are producing a lot more, and it is getting harder to review and harder to understand.
How the process made it worse
Some of this is all of us adjusting to working with AI. But the process we wrote made it worse in three specific ways.
First, it encouraged us to use AI to interrogate our own thinking. One of our standard moves was asking Claude to “grill me” on a piece of work before building it. That sounds rigorous. In practice, the grilling surfaced every possible edge case without the context to know how much each one mattered, or how complex the answer needed to be. Often the right answer to an edge case is to simply fail. A grilling session rarely gets there. It treats every question as something to solve, and the solutions it produced were over-engineered.
Second, the stages applied to everything. A one-line change went through the same flow as a new service. That added overhead to small work, and it taught people that the process was something to get through rather than something that helped them think.
Third, and most important, it moved judgement out of people’s heads and into the steps. When the process has Claude draft much of the approach and a person approve it, the person’s job quietly shrinks to checking. Each step looked reasonable. Together, they removed the moments where someone had to decide.
It was also slow. The detailed process was meant to give us somewhere to add controls later. But we didn’t need the scaffolding for that. Something like a pre-commit check could have gone in on day one. We could have run a dozen small experiments in the time it took to roll out one process, and learnt from each of them in days rather than months.
Get Matt Whetton’s stories in your inbox
Join Medium for free to get updates from this writer.
When I wrote about this process in September, I said the problem was being generated upstream, at the approach stage. I still think that’s true. What I got wrong was believing a more detailed process was the way to fix it.
The path of least resistance
The numbers describe a throughput problem. The judgement problem underneath them is the one that worries me more, and it would worry me even if the numbers were fine.
We all have days when we’re not firing on all cylinders. On those days, accepting what Claude suggests is the path of least resistance. The plan looks reasonable. The breakdown looks thorough. It’s easier to say yes than to think it through.
My sense is that coding agents are turning those days into most days. Nobody decides to switch off. The agent just makes it so easy that it rarely feels like a decision. It’s making us act dumber than we are, and trust it more than we should.
That matters because of what the agent is optimising for. Dex Horthy made a point in a recent talk that stuck with me. Coding agents are rewarded for producing solutions that work now. They don’t have to live with the complexity those solutions create six months later. When we’re engaged, we catch the defensive code nobody needed. When we’re not, the model’s defaults quietly become our decisions.
I don’t think that’s ok, and I think we should be fighting it. Paying attention used to be something the work demanded of us. Now it’s something we have to choose.
The easy conclusion
There’s a conclusion that’s easy to reach from all this. If the code is arriving faster than anyone can read it, then reading becomes the job. Engineers become reviewers.
I don’t think that’s where we should be heading. It’s where you end up by default.
I am worse at reviewing a pull request than I am at writing the code for the solution. Writing gives you confidence because you’ve had to work through the decisions. Reviewing means reconstructing decisions someone else made. If review becomes most of the job, we’ve moved our engineers onto the part of the work that builds the least confidence and produces the most doubt.
That reframed what my senior engineers were feeling. They had started asking “is this ok now?” about things they would once have judged without a second thought. They were being asked to approve decisions that had never really been made, only generated.
What we did instead
We kept the principles. Humans remain accountable. How the code is built matters. We dropped the process, made two targeted changes, and became very explicit about what we expect engineers to understand.
The first is in refinement. A refinement now ends with a person writing a ticket another person can read. Not a generated description, and not a prompt. We won’t accept walking out of a refinement on the basis that Claude will work out how to do it. If nobody in the room can explain how the work will be done, the refinement isn’t finished. That doesn’t mean knowing every implementation detail upfront. Sometimes the decision is to run an experiment. But someone needs to understand what we’re trying to learn and why. Each slice of work then has an owner who carries that understanding through to delivery.
The second is enhanced review triggers. Most changes don’t need anything more than they already get. Past certain thresholds, a change gets a deeper review, which could mean testing it together on a devbox or walking a group through the solution. We’re still working out where those thresholds sit.
We’ve been explicit about what we’re not asking for. We’re not asking engineers to review every line. Knowing exactly which functions were written matters less than understanding how the solution fits together and why it works.
Where this leaves me
I’m not sure yet that two changes are enough. The numbers will tell us whether work is flowing better. Whether we’re making better decisions will take more than a dashboard to establish.
What I am sure of is the shift in focus. Knowing every line and understanding the solution used to be the same thing, because whoever wrote every line had to understand it to write it. They aren’t the same any more, and the second one is the one that matters.
It doesn’t happen on its own. Understanding takes attention, and attention is the thing these tools make easiest to give away.
