AI Can Write the Code, But Who's Verifying It?

"Ladies and gentlemen, this is your captain speaking. Flight time tonight is about seven hours, weather over the Atlantic looks calm, and a quick note about your crew: I'm self-taught, roughly a hundred hours of YouTube. Sit back, relax, and enjoy the flight."

No one on that plane is relaxing.

The same confession from a dentist, or from the person building your house, ends the conversation just as fast. Flying, healthcare, construction: each is a discipline. You train under experts and mentors, and you learn how things fail before anyone lets you near something that matters.

Software engineering should also sit on that list. We're not just talking about degrees and diplomas. Some of the best engineers we know are self-taught. What matters is whether someone's judgment has been tested, reviewed, and shaped by real systems failing in real ways.

Instead, three factors have led to treating software as an exception.

First, You Can't See Software. A bridge shows its cracks and a bad weld can be inspected, but software has no inspectable exterior. A well-built system and a badly built one look identical from the outside: same screens, same buttons, same polished demo. Judging quality from the surface is impossible, even for professionals.

Second, No One Checks the Builder/Developer. Hiring without credentials is normal in this industry, and the checks that do exist verify the company's paperwork: procurement, audits, security questionnaires. Even great case studies only prove what some team at that company once did. Nobody asks whether those are the people who will work on yours.

Third, Apprenticeship Is Broken. Formal study, whether in a program or self-taught, is only the first half. The rest is learned on the job: code review, disagreement, watching systems fail in production, and senior engineers who explain why a decision is dangerous and why a feature matters to the business. That last part is the one most easily lost. Sound software decisions are business decisions first. Knowing what to build, what to defer, and what to refuse comes from understanding the business the software serves, not from the code itself. Remote work and AI crutches have eroded much of the side-by-side collaboration that used to pass on both kinds of judgment, and many junior developers now work almost entirely alone.

Reliance on AI can make the software look more superficially finished than ever, so the surface view tells you even less. It lets anyone produce at a pace no professional would trust, so unverified developers ship more and ship it faster. And for a junior developer working alone, the AI becomes the senior mentor: endlessly helpful, never saying "we tried that once and it fell over," with no failures of its own to remember and no verifiable sense of which features the business actually needs.

We see where this leads because the results land on our desk, more often every year. The reviews all rhyme. Here's the most recent one.

What 300,000 Lines Look Like

The admin screen was one file, more than 10,000 lines long. The checkout funnel ran several thousand more. All in, more than 300,000 lines across nearly 1,000 files, built in about six months. One person, with AI doing most of the typing. That comes to roughly 2,000 lines a day.

No engineer can read, test, and understand 2,000 new lines a day, least of all while also running the product and supporting customers. At that pace, the verification isn't happening, and most of the code goes live unread and untested.

The business on top of it was real. Paying customers, revenue, multiple payment providers, hardware integrations, a mobile app. The product looked polished, and customers liked it. The founders asked us to review the platform because growth was coming and they wanted to know what the code could carry. That's the question the product itself can never answer, so we went inside.

Professional software stands on a foundation of battle-tested parts for the plumbing every product needs: keeping users signed in, controlling what they can access, confirming that payments go through, running work behind the scenes. This platform had no consistent foundation for any of it. The groundwork had been reinvented again and again, some of it done well, plenty of it not, and no two solutions alike. The same logic appeared in slightly different versions across hundreds of files, and everything was woven into everything else so tightly that no second engineer could safely make a change.

Our review logged well over 1,000 findings, among them a long list of security flaws. When access control is hand-built in hundreds of places instead of once, flaws stop being accidents and start being arithmetic.

Of course, none of this was visible from the outside. The problems surfaced the only way they could: every change broke something somewhere else, and it was the customers, not the team, who found out first.

The Good Part

The review also found something worth protecting: the business logic worked. The pricing math, the booking rules, the guards against two customers grabbing the same slot, all of it encoded years of knowledge about how the operation runs. And it makes sense that this part held up. The rules of the business can be checked from the outside by the people who run it every day: does the price come out right, does the double booking get blocked. The engineering underneath had no such check. So our recommendation was to capture those proven rules in tests, preserve exactly how the business works, and rebuild the structure that turned every small change into a gamble.

The development speed was not the mistake. Building a prototype fast with AI is how this business got off the ground at all with a single developer, and we would tell any founder to do the same. Go fast, use Claude Code, use whatever gets something clickable in front of customers by Friday. A prototype has one job, answering the one foundational question: "Do we have a business here?" This one answered "Yes!", and left behind two assets: a working model of exactly what the product had to do, and paying customers who proved the demand was real.

But a prototype stops being a prototype when customers depend on it, money moves through it, sensitive data enters it, or failure can damage the business. That's the transition this company missed. Development continued in the way you build a prototype, fast and unverified, long after real customers and real money had raised the business risk to production level. Of the roughly 160 features we catalogued, the business truly depends on just a fraction. Those deserved to be built once, properly. The mistake was continuing to use the prototype as the product.

Six Questions, Six Months Earlier

You can't inspect the system, so inspect the thinking. Here are six questions you should ask anyone you've tasked with building the software that makes your business possible. If you ask them early, it costs you nothing. If you ask too late, it costs you an expensive (and time-consuming) rebuild.

1. What happens when something the system depends on fails?

Payment providers go down, networks drop, servers die. A team that has thought about it walks you through what the system does in each case and what the customer sees. Reassurance in place of specifics is the tell.

2. What did you decide not to build?

Good engineering is mostly about subtraction. Anyone who has engaged seriously with your problem has a list of rejected options, with reasons. If there's no list, nothing was decided, only accumulated.

3. What breaks first at ten times the volume?

Every system has a first bottleneck, and someone who knows the system can name theirs. "It should scale fine" means nobody looked.

4. What would have to change if we replaced the pricing model?

A well-designed system is made of swappable parts, so the answer is specific: where the rules live, what else is affected, how the change would roll out. Vagueness, huge numbers, or advice to "work around" means everything is tangled into everything else.

5. Who else can work on this?

If the honest answer is one specific person, then that person leaving, or simply being unavailable for a week, turns a dependency into a crisis. On this platform the answer was no one, not even the original author, without first spending months reconstructing how the code worked, because nothing was documented and the same logic was scattered, in slightly different forms, across hundreds of files.

6. Where have you done this before, and will I get those people?

Experience is the one reliable predictor of the parts you can't see: whether someone has built this kind of system, at this scale, and lived with it long enough to learn how it can fail. That experience belongs to people, not to the company that employs them. A strong track record means nothing if the people behind it aren't the ones assigned to your project. It's the first question we'd ask of anyone bidding to rebuild this platform, ourselves included.

None of this requires reading code. Vagueness where there should be specifics is the closest thing software has to a visible crack.

Happy Ending

To achieve a happy ending, prototype fast and know what you've built. Put it in front of customers, and learn the short list of things the business can't operate without.

For this company, there's a good path forward: keep everything the business learned, protect the behavior customers rely on, and rebuild what carries it step by step rather than all at once. The prototype more than paid for itself by proving the business. Extending it past that job is what got expensive. The same questions, asked six months earlier, wouldn't have cost anything.

If you're holding software that impresses everyone and quietly worries you, talk to us before growth forces the question. Contact us.