The Chasm

By Sean Hsieh, Founder & CEO, Runline

There is a picture I keep drawing on whiteboards for credit union leaders, and it is really just McKinsey’s own data with one line added. An organization decides to build. A prototype ships fast and feels like a win, and that good feeling is the eighty percent who say AI made them more productive. Then the curve reaches the place where the work that actually decides value begins: governing the thing, integrating it, maintaining it, securing it. And there the line forks. One branch climbs, and it is the six percent who capture real value. The other flattens and falls, and it is everyone else. The fork is always in the same place. I call the gap beneath it the chasm, because that is what it is: not a steeper stretch of the same road, but a drop between two different kinds of ground.

Everything in the last piece, the four ways the build dies, was a description of falling into this gap. This piece is about the gap itself, and about the single most expensive misunderstanding in enterprise AI right now: the belief that crossing it is just more of the work that produced the prototype. It is not. Prompting a demo and running a regulated, audited, maintained capability are not two points on one road. They are two different disciplines. One is a weekend. The other is a standing commitment with no end date. The distance between them is not effort. It is rigor of a kind most institutions have never had to staff.

I have watched the realization land on a team more than once. They have the prototype. It works in the demo, on clean inputs, in front of the people who approved it. Then someone starts asking the questions that live on the far side of the gap. What happens when the member’s data is not clean? Who gets paged when the output is wrong at two in the morning? How will we know it is still right after the model updates next month? What do we hand the examiner when she asks how it decided? Every question is reasonable. Not one of them came up during the afternoon that produced the demo. And the list of them, taken together, is not a punch list. It is a second project, larger than the first, that nobody scoped.

Let me be concrete about what that rigor actually is, because “production” is a word people nod at without pricing.

What crossing actually requires

I wrote a version of this in “The Loop Isn’t the Hard Part”: the loop, the part everyone is excited about, is the easy part. The hard part is verification, and in a regulated institution verification is not a feature. It is the whole product. Here is the minimum architecture for running an AI agent unattended on real member data, the thing that actually lives on the far side of the chasm.

First, deterministic validation, before anything the agent produces reaches a person. Not a model checking the model. A plain, boring rules engine that confirms the structural facts: the fields are present, the formats are right, the dates fall in range, the totals reconcile. If they do not, the output never goes out, and someone gets paged. This layer exists because a confident, well-formatted, wrong answer is the most dangerous thing an LLM produces, and it should fail loudly before a human ever sees it.

Second, a review pass against a rubric that encodes the actual process, not a generic “does this look right.” At Runline we call that rubric a Playbook: the institution’s procedure written as verifiable criteria rather than prose, so a second agent can check the first agent’s work against the standard the workflow is really held to. This is the layer that closes the conformance gap from two pieces ago, because the Playbook is where the seventeen undocumented steps finally get written down.

Third, an audit trail an examiner can follow. Every input the agent read, every rule it applied, every decision it made or deferred, logged in a form a human auditor can walk and a federal examiner will accept. Not because examiners review every run. They do not. But because the audit trail is what makes the other two layers believable. Without it you are asking a credit union to trust a black box, and no examiner accepts a black box.

Three layers, and the loop everyone got excited about is one line inside the third of them. That is the shape of the far side.

None of this removes the human. It relocates them. Elizabeth Stone, Netflix’s Chief Product and Technology Officer, put the principle simply in 2026: an agent may have written the code or run an analysis, but “it doesn’t make people not have the responsibility that comes with what they’ve created.” On the far side of the chasm a named person still owns every decision a Runner makes, and the whole architecture exists to make that ownership real instead of theatrical. Verification does not replace the officer. It is what lets the officer actually stand behind what the machine did.

And it has to reach your core

Verification is what you build on top. Underneath it is the part no prototype ever shows: getting the agent to your real data, safely. A BSA Runner is worthless if it cannot read the core, the transaction-monitoring system, the wire platform, and the member history, and it is dangerous if it can read them without a permissions model that survives scrutiny. Across the chasm, the agent is wired into the systems of record through a real integration layer, with identity, access controls, and a data foundation that normalizes what each of your cores stores in its own dialect. That plumbing is unglamorous, it never appears in a demo, and for a credit union running on a decades-old core it is frequently the single largest piece of the crossing. It is also where “build so we can control” can quietly turn into “build so we can maintain a dozen brittle integrations forever.”

This is not a credit-union problem. It is the shape of the whole enterprise moment. Cloudera’s 2026 survey of 1,500 enterprise and data architects found that 95% had delayed or canceled AI initiatives over data-governance, compliance, or regulatory hurdles, and 72% said their data architecture needed a significant overhaul before it could carry AI at all. The most common reason serious organizations stall is not the model. It is that the trusted, governed data foundation underneath is not there yet. A credit union on a decades-old core is that same problem in a smaller building, with an examiner attached.

The part that never ends

Everything I just described, you build once. Then you have to keep it alive, forever, on a foundation that will not hold still.

The models underneath change on someone else’s schedule. So crossing the chasm also means owning an evaluation harness that tells you when a model update quietly made your BSA classifier worse, before the examiner does. It means a model-routing layer, so a single deprecation does not take a workflow down. It means an on-call rotation, because at three in the morning when a Runner starts behaving strangely, the question is not how to fix it, it is how to stop it right now and prove that it stopped. That is a kill switch, at the level of the individual agent, the workflow, and the whole institution, and in a regulated shop it is not optional.

And there is one more layer, unique to institutions like yours. The moment an AI system makes or informs a decision about a member’s money, that system becomes a model in the regulatory sense, and model risk management attaches to it. Examiners reference the same interagency principles behind SR 11-7: effective challenge, independent validation, change management, documentation. “Examiner-Ready by Design” made the case that this is not a burden bolted on at the end. It is a design specification, and it happens to be the specification for the far side of the chasm. The institution that treats it as paperwork discovers, in an exam, that the paperwork was the product.

The bill nobody itemized

Now add it up, because this is where “build so we can control” meets arithmetic.

When you buy, all of that, the validation, the Playbook rubric, the audit trail, the eval harness, the routing, the on-call, the model-risk documentation, is built once by the vendor and spread across every institution they serve. You resent the invoice, but the invoice is a fraction, because the cost is shared. When you build, you pay for every layer yourself, in full, and then you pay to keep every layer current, forever, with a team that does not go home when the project ships, because the project does not ship. It runs. Most credit-union AI budgets I have seen priced the prototype. Almost none priced the far side. Cloudera’s survey put a number on the surprise: 84% of enterprises saw their infrastructure costs rise once AI workloads landed. The distance between the prototype budget and the real one is the chasm, expressed in dollars, and it is the number that quietly kills the six-month initiative in the fall.

I want to close on why this is worth getting right, and not merely as an accounting exercise. In “The Loop Nobody’s Verifying” I pointed at TD Bank’s $3.09 billion anti-money-laundering settlement, the largest in U.S. history, and the thing worth remembering is that it was not, in the end, a story about bad actors. It was a story about a loop that ran for years inside a very large institution with a verification layer too thin to catch what was moving through it. That is what an uncrossed chasm looks like at scale, and AI does not shrink that risk. It makes the loop run faster. A credit union that puts five Runners onto an unverified process has not automated its way across the chasm. It has automated its way deeper into it, at machine speed, and the clock on a missed suspicious-activity filing starts the moment the institution had the information, whether or not anyone had checked.

The good news, and it is real, is that the chasm is crossable. The six percent cross it. They cross it on the exact architecture I just described, which is why they compound while everyone else stalls. The only question left is the one no vendor slide wants to ask out loud: do you have the team to build and hold all of that yourself, or not? That is not a question about your ambition. It is a question about your engineering depth, and it is the most honest question in this entire series. We take it up next.