By Sean Hsieh, Founder & CEO, Runline
Go back to that afternoon. The coding agent, the two people from the team, the working prototype of something the institution had wanted for two years. I told you the wonder on that leader’s face was real, and it was. I also told you it hides something. This piece is about the specific thing it hides, because it is not a character flaw and it is not a failure of nerve. It is a property of the work itself.
The demo worked. That is the trap. It worked because it was exactly the kind of problem AI is spectacular at, handed to exactly the kind of person who could make that kind of problem sing, on exactly the kind of afternoon where nothing has to survive contact with a regulator, a member, or the next twelve months. Change any one of those conditions and the afternoon does not repeat. Change all three, which is what “put it into production at a credit union” actually means, and you are in a different universe of difficulty. Nobody in the room can see that yet, because the demo gave them no reason to.
Let me show you why the demo was honest and misleading at the same time.
The demo was not a fluke
First, credit where it is due, because underselling this is its own kind of dishonesty. AI genuinely is transforming how software gets built. This is not hype. The numbers here are among the cleanest in the entire field.
GitHub’s controlled study of Copilot found developers completed a real task 55% faster with AI than without, one hour and eleven minutes against two hours and forty-one, with a p-value of .0017. Anthropic studied 132 of its own engineers across roughly 200,000 Claude Code sessions and found that 27% of the work would not have been done at all without the AI, with engineers reporting the model handling around 60% of their tasks. And Google’s CEO told investors on the company’s Q1 2026 earnings call that the share of new code at Google that is AI-generated and then approved by an engineer had climbed from 25% in late 2024 to 50% by the fall of 2025 to 75% by that spring.
Read those together and the conclusion is unavoidable: for software engineering specifically, this is the largest productivity jump since the arrival of the modern IDE. The leader who felt the ground move was not imagining it. The ground moved.
The question this whole series turns on is why the ground moved there, in engineering, and whether it moves the same way when you carry the same tool across the hall into your BSA department. It does not. And the reason is worth understanding precisely, because it is the reason the demo will keep looking easy right up until the day it goes to production and stops.
Four things that make code easy for AI
Software has four properties that make it almost perfectly suited to what large language models do well. None of them are exotic; any software engineer would recognize all four in a heartbeat, which is exactly the problem, because most credit unions do not employ one.
Bounded. A function takes inputs and returns outputs. The task “fix this bug” lives inside a file, or a module, with dependencies that are written down and importable. You can draw a box around the work, and the box holds.
Checkable. The compiler tells you in milliseconds whether the code even parses. The tests tell you whether it does what it is supposed to. The type system catches whole categories of mistake before the program ever runs. The feedback loop is measured in seconds.
Structured. Code lives in files, under version control, with a deterministic build underneath it. The same input produces the same output. You can replay any state, roll back any change, and see exactly what differed.
Verifiable. A pull request is a discrete artifact. A reviewer can open the diff, read it in ten minutes, and say yes or no with real confidence.
Point a capable model at work that is bounded, checkable, structured, and verifiable, and the gain is enormous. That is the entire reason Claude Code and Cursor exist, and the entire reason the demo felt like magic. The magic is real. It is also conditional, and the four properties are the conditions.
Now walk the same four across the hall
Take the tool that just wrote a working prototype in an afternoon and point it at a BSA case investigation, which is precisely the kind of expensive, rules-heavy, back-office work a credit union most wants to automate. Watch all four properties break.
Bounded? A single BSA case reaches into the core, the transaction-monitoring vendor, the case-management system, the OFAC screening service, the wire system, sometimes two different cores left over from past mergers, a custom watchlist the credit union built years ago, and the senior BSA officer’s memory, which holds the actual institutional knowledge about which member patterns are ordinary and which are not. The scope of an investigation is whatever the senior officer decides it is that morning. There is no box. Or rather, the box has no walls.
Checkable? In software the compiler answers in milliseconds. Here, the nearest thing to a compiler is the examiner, and the feedback loop is months. Whether the model was right to treat a member’s $9,400 cash deposit as routine, or as the kind of just-under-the-threshold pattern that deserves a second look, does not get settled that afternoon. It gets settled at the next exam, after the report is filed or not filed, and even then the answer arrives as “the examiner had questions,” not as a green check.
Structured? The written SOP says one thing. What actually happens involves “I always open this spreadsheet first,” and “I email Sarah in deposit ops directly because the case-management notification has been broken for a year,” and a set of exception types the team handles every month that appear in no document anywhere. The distance between the documented process and the real one has a name in this field. It is the conformance gap. In ordinary work it runs thirty percent or more. In exception-heavy work like BSA, I have watched it run past seventy.
I am not speaking in the abstract. At one credit union I worked with, the team handed me the BSA procedure and said, in complete good faith, “this is the process.” Then I sat with the senior officer for two days. The real workflow differed from the document in seventeen places, including an unwritten rule that anything above five million dollars looped in the controller even though the SOP set the line at ten, the spreadsheet she opened before she touched the official system, and seven categories of exception that lived nowhere but in her twenty years of practice. None of that was a failing on her part. It was the accumulated intelligence that let the institution function. But none of it was structured, and a model cannot execute what was never written down.
Verifiable? A pull request can be checked in ten minutes. A SAR narrative cannot. To know whether the model got it right, a senior officer has to read the narrative, cross-reference the underlying transactions, sometimes pull the member’s full history, sometimes call the wire desk to clarify a detail. Verifying one case takes a good officer half an hour. The artifact is not discrete, and the review is not fast.
| Property | Software · holds | A BSA case · breaks |
|---|---|---|
| Bounded | Scope fits inside a file or module | Touches the core, the AML vendor, the wire system, and the officer's memory |
| Checkable | The compiler and tests answer in seconds | The examiner is the compiler; feedback takes months |
| Structured | Version control and a deterministic build | Most of the real process was never written down |
| Verifiable | A pull request is reviewable in ten minutes | A SAR narrative takes a senior officer half an hour |
The line every CFO eventually says
Here is what happens when you ignore all of that and point a generic model at the work anyway, which is precisely what a first prototype does.
The senior officer who used to clear a hundred routine cases a day is now clearing sixty, because she is also correcting the model’s mistakes on the forty it touched. The thing that was supposed to speed the work up has slowed it down, because it produced output that looks confident and is sometimes wrong, and confident-and-sometimes-wrong is the most expensive kind of output there is in a regulated function. A few months in, the CFO asks the question that ends most credit-union AI pilots: “what did our two hundred thousand dollars actually save us?” And the honest answer, more often than not, is nothing. Sometimes less than nothing.
That is not the model failing. The model is doing exactly what it does well, on work that has none of the four properties that let it do it well. The failure is upstream, in the decision to treat back-office judgment work as if it were bounded, checkable, structured, and verifiable code. It is not. And the demo could not have warned you, because the demo was, by its very nature, a small bounded and checkable and structured and verifiable world.
Why the demo lied, exactly
Now put it together with where we started. The person who ran that dazzling prototype was, at most credit unions, not a software engineer, because only the largest credit unions tend to employ any. He was an IT leader, or a head of operations, doing something genuinely impressive with a tool that rewards exactly the bounded, checkable kind of task a prototype is. He made a hard-looking thing look easy, once, in the one setting where all four properties held.
Production is the setting where they do not. The distance between that afternoon and a BSA Runner your compliance officer will stake an exam on is not a distance of effort, or of enthusiasm, or of talent. It is the distance from work that has all four properties to work that has none of them, crossed by people whose job has been to run vendor systems rather than to build and own production software through that crossing. The demo did not lie about the tool. It lied about the work, by showing you the single kind of work where the tool needs no help.
This is not a solo opinion. Matt Pocock, one of the most widely followed teachers of the craft, spent a ninety-minute conference workshop in 2026 arguing that AI is not the clean-break new paradigm everyone assumes, because “software engineering fundamentals, the stuff that’s really crucial to working with humans, also works super well with AI.” His blunter version: “bad code bases make bad agents.” Elizabeth Stone, Netflix’s Chief Product and Technology Officer, drew the same line from the executive chair: “There’s a difference between being able to write lines of code… and understanding how code, computer systems, products work. And I don’t think the latter is going away.” The tool got easier. The understanding underneath it got more valuable, not less.
Everyone who has built software for a living has learned to distrust this exact feeling, the one where the first eighty percent takes an afternoon. We distrust it because we know what the last twenty percent costs, and we know that in regulated work the last twenty percent is the entire product. That is the subject of the next piece: the four specific ways the build dies in that gap, each of which your team will meet, roughly in order, somewhere around month three.
