Field Notes · The Framework

The Five Fault Lines

Where AI deployments break, and why the model is rarely the reason.

If you have shipped AI into the real world, you probably know this story. The model topped its benchmark. The demo was clean. Everyone signed off. Then it went into the field, and it did not fail loudly. It just started being wrong: quietly, confidently, in ways no test had flagged, because the test had never asked the right questions.

It is tempting to read that as a modeling failure and reach for a better model. It almost never is. The model was fine; a strong model is necessary but rarely sufficient. What broke was the gap between the conditions a model is measured under and the conditions it has to live in. And that gap has structure. After enough of these projects, we have found that it opens along five predictable fault lines, and knowing which ones your application crosses, before you build, is most of the work.

The place to start is why benchmarks mislead. A state-of-the-art score is earned inside a quiet fiction: the inputs are curated and clean, the metrics were chosen because they are standard and easy to compute, the output format is fixed by convention, runtime constraints are seldom stated explicitly, and the data is assumed to reach the computing resources intact. None of that is dishonest; it is just not your deployment. Researchers have said as much for decades: no single model can be best at everything, and generality always has a price. The model did not lie about its score. The benchmark lied by omission about everything the score leaves out. The five fault lines are the five places where that omission catches up with you: what comes out, what goes in, how success is judged, when the answer is due, and where the thing runs. None of them is a flaw inside the model; each one is a seam between the model and your world. And each one is, at bottom, an assumption that went untested. The fix is rarely a better model, but is turning those assumptions into deliberate decisions.

FAULT LINE 01Output representation

The question what does it need to produce?

A standard model emits a standard set of outputs: exactly the fields it was trained to produce, and nothing else. Applications rarely want exactly that set. We worked on a vision-based worker safety system that needed body joints that a standard model simply does not output. No prompt fixes this: a model’s outputs, its training data, and its annotation scheme are one bound object. Getting a different set of outputs is not a configuration change; it means new annotation and new training. The teams that accept this on day one plan for it. The teams that assume the model already covers their needs discover it in month three, after the architecture is set.

The same fault line runs the other way: asking for a richer answer than the decision needs. Suppose you need to count bricks in a picture of a pallet. The bricks are all the same size, and they all look alike, yet a general-purpose detector will hunt each one, draw a box of arbitrary size around it, and perhaps label its every pixel. All that machinery exists because, in general, objects come in every size and shape. Yours do not. Counting research settled this more than a decade ago (see Learning to Count Objects in Images, NeurIPS 2010): problems with this structure can be solved with far simpler formulations, on far less data, at far lower cost. Paying for generality you do not need is still paying.

FAULT LINE 02Data & conditions

The question what will it actually run on?

A benchmark is run on clean, curated data. Your system runs on whatever your world actually produces. In one workplace deployment, the model we started with was excellent on benchmark frames. It lost most of its accuracy the moment a worker was partially occluded, and occlusion is the normal case in the field, not the exception: cameras end up wherever the site allows, and people work behind shelves, pallets, and each other. So, we collected data where the system would actually live, and then came the step that separates real work from optimism: accepting what the failures were telling us. The occlusion problem was not an annoyance to patch over. It was the thing the whole method had to be built for.

A second trap is less obvious, and it waits within the data collection itself. When teams go to collect data, the easiest way is to stage it: ask workers to carry boxes for the camera, and film that. Staged footage looks like deployment data, but it is not a reliable reflection of the real data distribution, especially in your metrics. A worker performing a task faces the camera, moves cleanly, completes every motion by the book; a worker actually working turns away, gets blocked mid-lift, hurries, improvises. Staged data still has its uses, as one ingredient in training, but it should not dominate your validation set. When the validation is mostly staged, your metrics can look excellent right up until users touch the system: the numbers stop tracking the reality they are supposed to measure.

FAULT LINE 03Metric & objective

The question how will you know it worked?

This is the fault line that quietly decides the others. The metric a model is optimized for is almost never the metric your application lives on. One system we shipped had a backbone model, conventionally scored frame by frame: how precisely it located things in each image. The question the system was bought to answer lived on a different axis: not “what is in this frame”, but “how often does this pattern repeat”, a number you can only get by following the footage over time and counting the repetitions. A model can be state-of-the-art on the first question and useless on the second. Frame-level scores do not even see the failure: predictions that flicker from frame to frame can each look right alone and add up to counts that are simply wrong.

So the first thing we built was not the model. It was the yardstick: the one measure that actually decides success, and the instrument that reports it. That choice ends up mattering more than any single modeling decision, because the build that follows takes its direction from it. Not by coincidence, it is also what lets us stand behind an outcome instead of a deliverable: a promise that “it works” means nothing until someone has built the thing that can tell.

FAULT LINE 04Timing & latency

The question when does it need to answer?

Offline, near-live, or hard real-time: the timing budget quietly sets the rules of what is possible. An application that can tolerate an offline pass lets you spend compute freely: look at the whole data sequence at once, run more than one pass, use methods that need to see both before and after a given moment. The same problem under a hard real-time budget forbids most of that. The two scenarios carry the same name and are not the same problem.

We once found this fault line sitting in competitors’ public demos. Their claimed advantage was running in real time; nobody had asked whether the task needed it. We did something plainer instead: we walked through the users’ actual day. The specialist films a batch of videos on the floor, then goes back to the office to study them and think through what to change; the results are not needed until then, and the office is where the thinking happens anyway. We took this read to the client’s own domain expert, who saw no flaw in it, so we built for offline and used methods that read the footage in both directions, which the real-time constraint would have ruled out. The end users, who had tried multiple products, found ours simply worked better. The requirement deserves the same scrutiny as the data, which is why studying how the answers will actually be used, by whom, on what clock, is part of the engineering, not a courtesy that precedes it. Some “real-time”, once you check, turns out to mean “by the end of the shift”, and every such discovery is accuracy you gain for free.

FAULT LINE 05Systems & hardware

The question where does it run, and what has to cross the wire?

The last fault line is the one people see last, because it has nothing to do with the model at all. We have watched a perfectly good drone-monitoring system come apart, not because the analysis was wrong, but because the video never arrived intact. A moving wireless link drops frames, and you cannot analyze frames you did not receive. No transport setting could rescue it. On a moving link, you can pick two of reliability, fidelity, and low latency; the analysis needed all three. The answer was architectural, not algorithmic: move the compute to the data. Run the analysis on the drone, on the video stream as the camera hands it over, and let the network carry only loss-tolerant copies for the live view; if one goes missing, the original is still sitting on the drone. Sometimes that means specifying the hardware yourself. This fault line is the deepest one, because it can take you below the model entirely, into the physical system it runs on.

Depth is the moat

The fixes line up by depth; consider one per stage. Defining the right evaluation is the shallow end. Collecting data that matches reality sits below it. Rebuilding the modeling stage sits deeper still, in and around the model: retraining for new outputs, changing the architecture, building the supporting modules around the backbone. Tailored hardware, redrawing where the data flows, sits at the bottom, beneath the model. Depth here is not difficulty: the shallow fixes are the ones you can walk back, and the deep ones commit you. The shallowest one, the yardstick, is also the easiest to get wrong. It costs nothing to change, so few teams stop to question it; yet it is the one decision every deeper fix is steered by. The further down a problem pushes you, the harder it is to solve by wrapping an off-the-shelf model and calling it a solution. The hard fault lines are usually the ones a one-size-fits-all provider cannot fix. And they rarely announce themselves: what the buyer sees is a product that does not work well, not the fault line underneath it.

That reframes the question an AI project actually turns on. It is rarely which model should we use. It is where is this going to break. Knowing the answer is mostly pattern recognition: deployments like yours have cracked before, in places we have learned to check first. The teams that ship tend to be the ones who go looking for their fault lines before they choose a model, not after the demo embarrasses them. The model is rarely where the system breaks. The work is in the five seams where the model meets your world.

That is also why every engagement we take starts with a diagnostic: we walk your deployment along all five fault lines and name where it will crack, before anything is built. And it is why we can stand behind results in a way most firms cannot. Because fixing the third fault line means building the very instrument that defines “works”, we do not have to guess whether a system succeeded: the yardstick was built, on the terms that mattered, before anything shipped. So we can put our stake on the outcome, not merely the delivery.

Find your fault lines first. The model can wait.