What a Cleared Medical Device and an Airport Shuttle Have in Common

An AI device that outlines organs on CT scans before radiation treatment, cleared by the FDA. Shareholder reports for thousands of investment funds. An AI platform built for NATO. A tablet platform for people in prison. Airport shuttle fleets. Film scripts turned into VFX budgets.

Six industries, no obvious connection. What do they have in common?

When the software is wrong, a real person pays for it. An outline that is off by millimeters changes where the radiation goes in a patient's body. A misread number puts a client's money in the wrong place. A risk score decides whether a person with a record gets the job. Building for stakes like these takes a different mindset than most of the software industry ever develops.

The Stakes Are the Reason

Most software can afford to be wrong. A bug ships, users hit it, the team patches it, everyone moves on. Nearly every habit in the industry, and nearly every tool, is calibrated to that cost.

In the work above, an error has a victim before anyone knows it happened. That single fact changes how you build.

Assume the system will be wrong, and catch it first. The accuracy in the sales deck is not the accuracy you will get: demos hit 95% on clean documents, your own documents drop it to 60%. So the question that matters is whether a wrong answer reaches the patient, the analyst, or the hiring manager before anyone notices. A serious system knows its own confidence, sends anything uncertain to a person, and gets better from every correction.

Know exactly what it did. By the time a question comes up, the radiation was already delivered and the trade already settled. Either the system recorded which version ran, what went in, and who approved the exception, from its first day, or that answer does not exist and cannot be recreated. Self-protection is the least of it: the patient whose treatment the software touched deserves better than "we believe it was working."

Read the regulation as recorded harm. FDA design controls exist because devices have hurt people. Privacy law exists because records have leaked. The stalled projects we get called into treated those rules as paperwork for later: build the clever part first, handle compliance before launch. The pilot worked. Four months later it died in a review it was never designed to pass.

Distrust the demo. Two systems can look identical on screen when only one of them can catch its own errors and account for its own actions. Buying on the strength of the demo is how companies end up owning the other kind.

Questions That Reveal the Mindset

You cannot see any of this from outside, but you can find it by asking. None of these require reading code.

"What happens when a system you depend on goes down?" Pick any of them: the payment provider, the data feed, the model vendor. In defense work this question comes with lives attached: a unit in a multi-domain operation acting on stale information is worse off than one that knows it is blind. A team with the right mindset walks you through exactly what happens and what your customer sees while the system is down. Reassurance in place of specifics is the tell.

"How will we know it is still working in six months?" The answer should be machinery, not confidence: checks that run on every change, monitoring that catches problems before a customer does. On the device we took through FDA clearance, one automated check caught a misconfiguration before anything deployed. Found by an inspector instead, it would have been a formal violation on the record.

"What exactly is inside it?" The right answer is a list, available on demand, down to every borrowed component. Ours passed 200 entries, so it rebuilds itself automatically; when a security flaw was published in a component three layers deep, we knew within hours. If nobody can produce the list, nobody is watching what is on it.

"Where does our data go, and when does it leave?" Every outside service that touches the data is part of the answer: the model vendor, the storage, the analytics. On the tablet platform we built for use inside prisons, this question was the whole project: a device in a cell must not become a channel for information to move in or out unseen. Your version is milder, and the discipline is the same. A team with the right mindset names every service, shows each is covered by your agreements, and says what is kept, for how long, and how it is deleted when a regulation or a customer requires it. "We would have to check" means the data is already somewhere it should not be.

"Who governs the AI?" A model is not installed once; it changes. Someone has to decide when it changes, prove the new version is safe before it touches real cases, and watch for accuracy drifting afterward. On the cleared device, that governance was agreed with the FDA in advance: a bounded set of changes, each with an agreed way to prove it safe. "We just take the latest version" is the answer to worry about.

"Who did what, and when?" The answer should be a named person, backed by a record the system itself cannot edit. Anything vaguer means that when something goes wrong near a payment or a person's job, the answer will be a shrug.

These skills transfer in one direction. A team that has shipped an FDA-cleared device can build your internal tool in an afternoon. A team raised on internal tools cannot make the trip in reverse, and you will not see the gap in a demo. You will see it when something is at stake.

The Saturday Build

Someone in operations wires up n8n or Power Automate over a weekend, and by Monday documents are being read, classified, and routed. It looks like a six-month project delivered early and free. For meeting scheduling and internal notifications, where the worst case is a missed reminder, it genuinely is the right call.

Ask it these questions and you find out what it actually is. Nothing in it measures confidence, so a wrong answer sails through. Its only record of itself is a log the tool can edit, so what it did last month is anyone's guess. Your data goes wherever each integration sends it, and nobody can say what those integrations contain, because every one is someone else's code updated on the maker's schedule. All of it runs under the personal login of whoever built it, with all of their access, kept when they change roles and nobody remembers to look.

Strip away the polish and the Saturday build is a prototype that moved into production without telling anyone. That is harmless while it schedules meetings. The day it starts reading patient records, routing payment instructions, or storing an unreleased script, your most sensitive process is running on the least accountable software in the building.

And they multiply. What we find at clients is rarely one of them. It is hundreds, with no list of them anywhere, builders long gone, credentials still live. We wrote about getting ahead of that separately.

Rigor Without the Slowdown

The usual objection to doing it properly: surely that means slow. Frozen requirements, binders of sign-offs, a release every other quarter. It did mean that, back when the rigor was carried by hand.

Machines carry it now. The compliance checks finish in minutes. The list of what is inside rebuilds itself. The evidence assembles as a side effect of normal work instead of being written up at the end. Being careful costs minutes per release, and the team ships the way modern product teams ship. That is how the contouring device, which began as a research prototype at Penn Medicine, reached FDA clearance in months at less than half the cost of the traditional route, and why it keeps improving after launch without a fresh approval cycle each time.

One Discipline

This is what the list at the top of this page has in common. The fund platform traces every number back to its source page, because analysts sign what it produces. R3 Score will not emit a score nobody can explain; a job hangs on it. The studio examined what our pipeline was made of before asking whether its budgets were right; a leaked script does damage no refund covers. The airport's service numbers survive disputes because they come from a record of what each shuttle actually did, and penalties ride on them.

Different industries, same discipline, and it is what clients are actually buying: software built by people who take the stakes seriously from the first day, and ship at full speed anyway. The engineering behind all of it is on our engineering blog.

If your process has real stakes behind it, get in touch.