AI Can Write the Code. Who Is Verifying It?

"Ladies and gentlemen, this is your captain speaking. Flight time tonight is about seven hours, weather over the Atlantic looks calm, and a quick note about your crew: I'm self-taught, roughly a hundred hours of YouTube. Sit back, relax, and enjoy the flight."

No one on that plane is relaxing.

The same confession from a dentist, or from the person building your house, ends the conversation just as fast. The instinct is right. Flying, dentistry, construction: each is a discipline. You train, you work under people who came before you, you learn how things fail before anyone lets you near something that matters.

Software engineering should sit on that list. That's not about degrees, some of the best engineers we know are self-taught. What matters is whether someone's judgment has been tested, reviewed, and shaped by real systems failing in real ways.

Instead, software gets treated as the exception, and in our view three things made it one.

First, you can't touch software. A bridge shows its cracks and a bad weld can be inspected, but software has no exterior. A well-built system and a badly built one look identical from the outside: same screens, same buttons, same polished demo. Judging quality from the surface is impossible, even for professionals.

Second, buyers rarely verify the developers. Hiring without credentials is normal in this industry, and the checks that do exist verify the company's paperwork: procurement, audits, security questionnaires. Even great case studies only prove what some team at that company once did. Nobody asks whether those are the people who will work on yours.

Third, the second half of the education broke. Formal study, in a program or on your own, is only the first half. The rest is learned on the job: review, disagreement, production failures, and senior engineers who know why certain decisions are dangerous. Remote work ended much of the sitting-next-to that used to happen by default. A lot of junior developers now work almost entirely alone.

AI puts all three on steroids. It makes software look more finished than ever, so the surface tells you even less. It lets anyone produce at a pace no professional would trust, so unverified developers ship more, faster. And for a junior developer working alone, the AI becomes the senior mentor: endlessly helpful, never saying "we tried that once and it fell over," with no failures of its own to remember.

We see where this leads because the results land on our desk, more often every year. The reviews all rhyme. Here's the most recent one.

What 300,000 lines look like

The admin screen was one file, more than 10,000 lines long. The checkout ran several thousand more. All in, more than 300,000 lines across nearly 1,000 files, built in about six months. One person, with AI doing most of the typing. That comes to roughly 2,000 lines a day.

AI can type 2,000 lines a day. No engineer can read, test, and understand 2,000 new lines a day, least of all while also running the product and supporting customers. At that pace the verification isn't happening, and most of the code goes live unread.

The business on top of it was real. Paying customers, revenue, multiple payment providers, hardware integrations, a mobile app. The product looked polished, and customers liked it. The founders asked us to review the platform because growth was coming and they wanted to know what the code could carry. That's the question the product itself can never answer, so we went inside.

Professional software stands on a foundation of battle-tested parts for the plumbing every product needs: keeping users signed in, controlling what they can access, confirming that payments went through, running work behind the scenes. This platform had no consistent foundation for any of it. The plumbing had been reinvented again and again, some of it done well, plenty of it not, and no two solutions alike. The same logic appeared in slightly different versions across hundreds of files, and everything was woven into everything else so tightly that no second engineer could safely make a change.

Our review logged well over 1,000 findings, among them a long list of security flaws. When access control is hand-built in hundreds of places instead of once, flaws stop being accidents and start being arithmetic.

None of this was visible from the outside. The problems surfaced the only way they could: every change broke something somewhere else, and it was customers, not the team, who found out first.

The part that was good

The review also found something worth protecting: the business logic worked. The pricing math, the booking rules, the guards against two customers grabbing the same slot, all of it encoded years of knowledge about how the operation runs. And it makes sense that this part held up. The rules of the business can be checked from the outside by the people who run it every day: does the price come out right, does the double booking get blocked. The engineering underneath had no such check. So our recommendation was to capture those proven rules in tests, preserve exactly how the business works, and rebuild the structure around them, the structure that turned every small change into a gamble.

The speed was never the mistake either. Building fast with AI is how this business got off the ground at all with a single developer, and we would tell any founder to do the same. Go fast, use Claude Code, use whatever gets something clickable in front of customers by Friday. A prototype has one job: answering whether you have a business. This one answered yes, and left behind two assets: a requirements document you could click, and market validation that paid.

But a prototype stops being a prototype when customers depend on it, money moves through it, sensitive data enters it, or failure can damage the business. That's the transition this company missed. The code kept moving at prototype speed after the risk had become production risk. Of the roughly 160 features we catalogued, the business truly depends on a fraction. Those deserved to be built once, properly. The mistake was keeping the prototype on as the product.

Six questions, six months earlier

You can't inspect the system, so inspect the thinking. These six questions work on anyone building for you. They cost nothing to ask early, and a rebuild to ask late.

What happens when something this depends on fails?

Payment providers go down, networks drop, servers die. A team that has thought about it walks you through what the system does in each case and what the customer sees. Reassurance in place of specifics is the tell.

What did you decide not to build?

Good engineering is mostly subtraction. Anyone who engaged seriously with your problem has a list of rejected options, with reasons. If there's no list, nothing was decided, only accumulated.

What breaks first at ten times the volume?

Every system has a first bottleneck, and someone who knows the system can name theirs. "It should scale fine" means nobody looked.

What would have to change if we replaced the pricing model?

A well-designed system is made of swappable parts, so the answer is specific: where the rules live, what else is affected, how the change would roll out. Vagueness, huge numbers, or advice to work around it mean everything is tangled into everything else.

Who else can work on this?

If the answer is one person, you have a dependency on a human being. On this platform the honest answer was nobody, its own author included, not without months of archaeology first.

Where have you done this before, and will I get those people?

The question everyone skips, and the only one procurement never asks. Experience is the one reliable predictor of the parts you can't see: this kind of system, at this scale, lived with afterward. And it belongs to people, not companies. It's the first question we'd want asked of anyone bidding to rebuild this platform, ourselves included.

None of this requires reading code. Vagueness where there should be specifics is the closest thing software has to a visible crack.

The ending you'd rather have

Prototype fast, and know what you built. Put it in front of customers, and learn the short list of things the business can't operate without. Then change modes: build that list properly, with people who have built it before, and let usage decide everything else.

For this company there's a good path forward: keep everything the business learned, protect the behavior customers rely on, and rebuild what carries it step by step rather than all at once. The prototype earned its cost by proving the business. Extending it past the job it was built for is what got expensive. The same questions, asked six months earlier, would have cost nothing.

If you're holding software that impresses everyone and quietly worries you, talk to us before growth forces the question. Get in touch.