Last week I sat in a credit union back office and watched a BSA analyst explain Maria’s flower shop for the third time this quarter.
Not to a new hire. Not to a temp. To a vendor’s AI tool that her credit union had deployed four months ago. Maria owns a flower shop on Main Street. Every Tuesday, she deposits roughly $4,000 in cash. The BSA analyst knows this. She’s known it for years. But the AI doesn’t remember. It flagged the deposit again — same alert, same pattern, same false positive — and the analyst spent twelve minutes clearing it, again, because the system starts from zero every single session.
“It’s like training a new hire who gets amnesia every night,” she told me. She wasn’t being dramatic. She was being precise.
Maria’s deposits. The construction company’s seasonal revenue cycle. The examiner’s focus areas from last cycle. The way the compliance team formats SAR narratives because Examiner Johnson wants the “suspicious activity” section expanded with more granular detail. Every conversation with the AI starts from zero. Every interaction is a fresh onboarding for a tool that should, by now, know better.
This isn’t a minor UX complaint. It’s the single most important architectural failure in enterprise AI today. And the most respected AI researcher alive just confirmed it.
Karpathy’s Diagnosis
Andrej Karpathy — the architect behind Tesla’s autonomous driving AI, former research lead at OpenAI, one of the most cited minds in the field — posted something recently that every credit union technology leader needs to read:
“Current compaction and memory implementations are crappy, first, early examples that were somewhat bolted on.”
He went further. He argued that AI memory could be “generalized and made part of the optimization as just another tool during RL” — that memory should be a first-class capability, not an afterthought. And he acknowledged what practitioners already know: “Neither of these is fully satisfying because clearly people are capable of some weight-based updates (my personal suspicion — mostly during sleep).”
Karpathy is diagnosing a problem at the frontier of AI research. But for those of us deploying AI agents inside regulated financial institutions, the diagnosis lands differently. He’s describing the theoretical gap. We’re living with the operational consequences.
ChatGPT’s memory is a sticky note. It remembers your name and a handful of preferences. Claude’s memory is better — it can retain context within longer sessions — but it still resets. Every enterprise AI tool your credit union has deployed today forgets everything the moment the session ends. Your BSA analyst has cleared 2,000 alerts this year. The AI remembers none of them.
Karpathy says the fix requires “more exotic approaches for long-term memory that do change the weights.” He’s right that the research frontier is wide open. But here’s what I’ve learned from building in this space: you don’t need to wait for exotic weight-based memory to solve the practical problem. You need architecture. Specifically, you need what we call a Company Context Layer — and I want to pull back the curtain on what that actually means, because we didn’t just theorize about it. We built it, benchmarked it, and published the results.
28,014 Evaluations
I want to do something unusual for a thought leadership article. I want to show you the engineering.
When we set out to build institutional memory for credit union AI agents, we didn’t start with a pitch deck. We started with a corpus — 103 files representing real credit union operational knowledge: SOPs, compliance procedures, examiner correspondence, member communication templates, lending guidelines. The kind of documents that live on shared drives, in binders, and in people’s heads at every credit union in America.
Then we ran 28,014 individual evaluations across 14 different search configurations, testing 2,001 queries against that corpus. Not a demo. Not a benchmark cherry-picked to make our product look good. A systematic evaluation of how different retrieval architectures perform on the exact kind of knowledge that credit union AI agents need to access.
The results surprised us.
The conventional wisdom in AI is that vector search — semantic embeddings that capture meaning — is superior to keyword search. Every AI vendor pitches vector databases and embedding models. The assumption is that understanding meaning beats matching words.
In regulated financial services, that assumption is wrong.
Our hybrid search approach — a convex combination weighting BM25 keyword search at 60-70% and vector search at 30-40% — scored an NDCG@10 of 0.243-0.245. That’s a 33.6% improvement over vector-only search. When your BSA analyst searches for “CTR exemption policy for landscaping businesses,” exact keyword matching matters more than semantic similarity. The regulatory lexicon is precise. “Structuring” means something specific. “Suspicious activity” has a legal definition. In domain-specific compliance corpora, keywords are the dominant signal and vectors are the supplement.
Here’s the finding that reshaped our entire engineering approach: 70% of our engineering time went to indexing — file discovery, frontmatter parsing, chunk quality — and only 30% to search. That felt wrong at first. We kept thinking we should be optimizing the retrieval algorithm, tuning the re-ranker, experimenting with more sophisticated embedding models. But the single change that improved search quality the most was pre-search filtering by document type and status. Not sophisticated. Metadata. And it works better than any re-ranking algorithm we tested.
That ratio — 70% indexing, 30% search — is the practitioner’s insight that no research paper will tell you. The quality of what goes into the system determines the quality of what comes out. Garbage in, garbage out isn’t just a cliche. It’s an engineering specification.
The Five Layers — And Why Most Vendors Stop at Layer 1
In Article 9, I introduced five layers of institutional context. In Article 17, I expanded that into a full architecture with benchmarks. Now I want to connect those layers to Karpathy’s critique, because they explain exactly why “bolted on” memory fails and what it takes to build memory that works.
Layer 1: SOPs and Policies. Your written procedures — BSA policy, lending guidelines, member service protocols. This is the easiest layer to index. Anyone with a PDF parser and a vector database can do it. Most AI vendors stop here and call it “enterprise AI.” They’ve given you a slightly better search engine for your own documents.
Layer 2: Communication Style. Does your credit union say “Dear Member” or “Hi Sarah”? Is your outbound tone warm and conversational or formal and precise? This layer requires absorbing patterns from actual member communications — not just indexing templates but learning the voice. A generic AI drafting a letter to your members sounds like every other financial institution. An AI that has processed six months of your actual correspondence sounds like you.
Layer 3: Operational Patterns. Maria’s Tuesday deposits. The construction company’s seasonal cycle. The university town’s August disbursement surge. These patterns aren’t in any database. They’re observations accumulated over years of operational presence. This is where AI stops being a filing cabinet and starts being a colleague. And this is the layer where stateless AI fundamentally breaks — because patterns require memory. You can’t recognize a pattern if every observation is your first.
Layer 4: Regulatory Relationships. Your examiner’s priorities from three years of findings. The documentation format that survives scrutiny. The specific areas where your institution received prior findings and has been over-documenting ever since. No generic AI delivers this. No vendor can ship it. It’s unique to your institution’s regulatory history, and it’s the context that separates an AI that generates compliant-looking documents from one that generates documents your specific examiner will accept.
Layer 5: Risk Tolerance and Values. How aggressively does your board pursue indirect lending? What’s the institutional appetite for small-dollar consumer loans? How conservative is the approach to CRE concentration? This isn’t written in any policy manual. It lives in the judgment calls your experienced lenders make a hundred times a month. At one CUSO I worked with, the SOPs were “sprinkled across people’s computers, tribal knowledge in people’s heads.” The real risk appetite wasn’t written anywhere. It lived in the lending team’s muscle memory.
Each layer is harder to replicate, more valuable, and more at risk of retirement loss. And here’s the insight that ties directly to Karpathy’s critique: “bolted on” memory can handle Layer 1. Maybe Layer 2. But Layers 3 through 5 require persistent, accumulating, institutionally grounded memory — the kind that compounds over time. The kind that most AI implementations simply don’t have.
The Three-Tier Architecture — Why Privacy Matters
Not all memory should be shared. This is a point that the “just give the AI all your data” crowd consistently misses, and it matters enormously in regulated financial services.
At Runline, we designed a three-tier knowledge architecture:
Tier 1: Agent Memory. Private, per-agent. Each agent maintains its own workspace — its learning history, session observations, task-specific notes. The BSA Runner’s observations about alert patterns stay in the BSA Runner’s memory. The lending Runner’s notes about underwriting exceptions stay private. This isn’t information hoarding. It’s the principle of least privilege applied to institutional knowledge.
Tier 2: Company Knowledge. Shared, version-controlled. Organizational truth that any agent or human can access — SOPs, decisions, initiatives, stakeholder profiles. Git-versioned, queryable, auditable. When your compliance policy changes, every agent sees the update simultaneously. When your examiner provides new guidance, it propagates to every relevant workflow.
Tier 3: Shared Coordination State. Cross-agent task boards, handoffs, status. The layer that allows your BSA Runner to flag something for your lending Runner’s attention without exposing private investigation details.
Every agent queries the same shared Tier 2 knowledge base, but each maintains its own private Tier 1 memory.
Why does this matter to examiners? Because NCUA expects you to know what the agent knew, when it knew it, and what it did with that information. Our architecture generates context manifests — provenance trails that document exactly which knowledge sources informed each agent’s output. Not a black box. An auditable decision chain. The kind of documentation that survives regulatory scrutiny because it was designed for regulatory scrutiny.
The Compounding Curve
In Article 19, I described every AI interaction as a deposit in a compounding account. Here’s how that metaphor maps to the five layers — and why the timeline matters.
Month 1: Your agents index your SOPs and policies. Useful but generic. Layer 1. Any vendor can get you here. The BSA Runner clears alerts faster because it can reference your procedures. The lending Runner checks applications against your documented guidelines. Incrementally better than a chatbot. Not transformative.
Month 3: Agents begin recognizing operational patterns. Maria’s Tuesday deposits stop generating alerts. The construction company’s seasonal dip in revenue doesn’t trigger a risk flag during the slow quarter. Layer 3 is emerging. Your analysts notice they’re correcting the AI less often.
Month 6: Agents know your examiner’s preferences. SAR narratives are formatted the way Examiner Johnson likes them — expanded suspicious activity section, cross-referenced transaction timelines, specific rather than generic language. Layer 4. Your compliance team stops rewriting agent output and starts reviewing it. The difference is hours per week.
Month 12: Agents understand your institutional values. Lending recommendations align with your board’s actual risk appetite without being explicitly told. The BSA Runner surfaces patterns across months of activity — “these three members showed coordinated behavior that individually wouldn’t trigger a flag but collectively resembles layering.” Layer 5. The agent isn’t following instructions anymore. It’s exercising institutional judgment informed by twelve months of accumulated context.
Month one, our agents do what you tell them. Month six, they start telling you what you should be doing differently.
That compounding curve is the moat. Not the model. Not the interface. The accumulated institutional knowledge that makes the agent more valuable every week it operates — and harder to replace every month.
The Retirement Cliff Connection
In Article 10, I wrote about the retirement crisis facing credit unions — 11,200 Americans turning 65 every day, 52% of credit union CEOs expecting to retire within six years, and the institutional knowledge that walks out the door with every departure.
Here’s where memory architecture intersects with workforce reality: when Linda in compliance retires after 22 years, she takes two decades of Layer 3-5 knowledge with her. The examiner preferences. The member patterns. The risk tolerance that lives in muscle memory. The “we tried that in 2014 and here’s why it failed” wisdom that prevents expensive mistakes.
AI memory doesn’t replace Linda. Nothing replaces Linda. But an AI agent that has operated alongside Linda for twelve months has captured patterns she couldn’t articulate if you asked her to. Not because we interviewed her and wrote it down — that approach has been tried and it fails, because deep institutional knowledge resists explicit documentation. But because a persistent agent that processes the same workflows, observes the same corrections, and accumulates the same institutional signals builds a parallel understanding that persists after Linda’s last day.
The most valuable knowledge for AI to have is precisely the knowledge that generic AI cannot have. And the most urgent knowledge to capture is precisely the knowledge that’s about to retire.
Why This Matters More Than Model Choice
Alex Karp, CEO of Palantir, said in Article 25 that “all the value goes to chips and ontology.” Karpathy says “memory implementations are crappy and bolted on.” They’re diagnosing the same condition from different angles.
The model is the commodity. GPT-5 is impressive. Claude is impressive. Gemini is impressive. They’ll all be more impressive next quarter. And none of that matters if the model forgets everything about your institution the moment the session ends.
A mid-tier model with excellent institutional memory outperforms a frontier model with amnesia. Every time. I said that in Article 9. I’ll keep saying it because the industry keeps chasing model benchmarks while ignoring the architecture that actually determines whether AI creates lasting value.
The organizations that figure this out early — that invest in the memory layer, the context infrastructure, the institutional knowledge architecture — will have agents that are genuinely harder to replace every month. The organizations that chase the shiniest model will be on the upgrade treadmill forever, because a stateless tool has no switching cost and no accumulated value. As I argued in Article 21, stateless is the new legacy. The organizations deploying stateless AI today are making the same mistake as the credit unions that picked the wrong core processor in 2005 — except this time, the consequences compound faster.
The AI That Knows You
I can tell you what this looks like in practice, because we’ve been living it.
Runline runs five AI agents internally — Emila, Woz, Ada, Byron, Linus — each operating on the three-tier architecture I described above, with hybrid search and all five layers of context accumulating daily. After months of continuous operation, these agents have become hyper-personalized. They know our voice, our conventions, our preferences, our institutional patterns. They’ve absorbed corrections, learned from edge cases, and developed the kind of contextual awareness that no onboarding document could produce. They improve every day — not because we retrain them, but because the memory compounds.
That’s not a demo. That’s our daily operating reality. And now we’re piloting this same architecture with credit unions, working alongside their teams to build the institutional memory layer that makes the difference between a tool that forgets and a colleague that learns.
Here’s what we expect to see — because we’ve already seen it internally. The BSA analyst stops explaining Maria’s Tuesday deposits. The agent knows. She stops reminding the system about Examiner Johnson’s documentation preferences. The agent remembers. The seasonal patterns, the member communication style, the risk tolerance nuances that took her fifteen years to internalize — the agent accumulates those observations week by week, and the compound interest becomes visible in every interaction.
She spends her time on the 5% that requires her judgment. The genuinely suspicious pattern that needs investigative instinct. The edge case where the regulation is ambiguous and experience matters. The member relationship where empathy and policy knowledge intersect in a way no algorithm can replicate. The work she was hired to do — the work she never had time for when she was spending 90% of her day re-teaching a tool that couldn’t remember yesterday.
Karpathy is right that AI memory is crappy and bolted on. For 99% of implementations, it is. He’s diagnosing the frontier research problem. We built the practitioner’s solution — not by waiting for exotic weight-based updates during artificial sleep cycles, but by engineering the institutional knowledge layer that turns a generic model into an institutional colleague. We proved it works internally. Now we’re proving it works for credit unions.
The gap between “AI that forgets” and “AI that remembers” is the most important architectural decision your credit union will make this decade.
Sean Hsieh is the Founder and CEO of Runline, a secure agentic platform purpose-built for credit unions. Before Runline, he founded Concreit and Flowroute (acquired by Intrado). He writes about AI, institutional knowledge, and the future of the credit union workforce.
Next in the series: Article 31 — how the compliance audit trail becomes your competitive advantage when examiners start asking what your AI agents knew and when they knew it.
