The Tool Isn't the Problem. The Workflow Is.

A 52-year-old man gets arrested for trying to lure a child at a McDonald's five hours from his home, a restaurant he has never visited, in a city he has never been to. An automatic license plate reader shows no record of his car anywhere near the scene. He lives 300 miles away. None of that stops the warrant from being issued.

The facial recognition software said it was probably him. That was enough.

This is not a story about bad technology. It is a story about bad process wrapped around mediocre technology, deployed by people who were never trained to understand what the output actually means.

I run enterprise cybersecurity for a large state government. We use AI-assisted tools constantly. Anomaly detection, behavioral analytics, threat correlation, pattern matching at scale. These tools are genuinely useful. I am not going to argue that government should stop using them. But every single one of those tools shares a critical characteristic that most of the people using them do not fully grasp: they produce probabilities, not answers.

Facial recognition software does not tell you who committed a crime. It tells you who looks similar to the person in the image, within whatever tolerance the algorithm was tuned to accept. The output is a ranked list of possible matches with associated confidence scores. When a grainy surveillance photo gets fed into that system, the confidence scores drop, the false positive rate climbs, and what you get back is a list of people who resemble each other more than they resemble the actual suspect.

That is not a flaw. That is the math working exactly as designed.

The flaw is treating that probabilistic output as an investigative conclusion instead of an investigative starting point.

Aviation safety culture builds multi-layer checks into every critical decision precisely because any single instrument or sensor can fail silently. A pilot who acts on one anomalous reading without cross-checking against three others is not being decisive. They are being reckless. The whole architecture of modern cockpit design is built to force cross-validation before action. One data point does not authorize a decision that cannot be undone.

Law enforcement agencies using facial recognition are skipping that architecture entirely. What happened in this case is a textbook single-point-of-failure scenario. The facial recognition match fed directly into a photo lineup. The photo lineup fed directly into a warrant application. The warrant application omitted the exculpatory evidence. A man went to jail.

This is not an AI problem. This is a workflow design problem. And it is exactly the kind of problem that good governance is supposed to catch before it generates a lawsuit.

Here is what the workflow should look like. Facial recognition output gets treated the same way a cybersecurity analyst treats a SIEM alert: as a signal that requires corroboration, not a finding that authorizes action. You cross-check it against physical evidence, timeline analysis, behavioral indicators, and independent witness identification conducted without contamination from the biometric match. If the corroborating evidence conflicts with the match, you hold. You do not proceed to warrant based on a single contested data point.

The Robert Dillon case had multiple conflicting data signals available before the warrant was ever sought. License plate reader data showing no presence near the scene. A witness describing the suspect as a "regular" at a restaurant the suspect claims never to have visited. A distance calculation that put the suspect five hours away from the crime. Any one of those should have triggered a hold and a second review. All three together should have stopped the investigation in its tracks.

Instead, investigators anchored on the facial recognition match and used the photo lineup to confirm what the algorithm had already told them. That is confirmation bias with a technology veneer on top. It is not investigation. It is post-hoc justification dressed up in software.

The Pinellas County system at the center of this case is a statewide tool touching law enforcement agencies across Florida. When a single tool touches that many agencies and cases, the design of the guardrails matters more, not less. Scale amplifies both the benefits and the failure modes. If the workflow is broken, deploying it at scale means breaking things at scale.

What needs to change is not the technology. What needs to change is the doctrine governing its use.

Government agencies need mandatory corroboration requirements before any biometric match can be cited in a warrant application. They need explicit disclosure requirements: if facial recognition was used in an investigation, that fact and the confidence score must appear in the warrant application, not get buried or omitted. They need independent review at a defined checkpoint before an arrest is made, specifically in cases where the biometric match conflicts with any physical or timeline evidence.

These are not novel ideas. The intelligence community has used structured analytic techniques for decades to prevent exactly this kind of single-source, confirmation-biased reasoning from driving irreversible action. The requirement to cite alternative hypotheses, to document what would have to be true for the primary hypothesis to be wrong, to explicitly identify the weakest link in the evidentiary chain. None of that is magic. It is discipline.

What Robert Dillon experienced is what happens when a powerful probabilistic tool gets integrated into a high-stakes decision process without any of that discipline attached to it. The technology worked exactly as designed. The process failed him completely.

The agencies involved in this case owe him an apology and owe the public a redesigned workflow. What they should not do is respond by either doubling down on the tool or abandoning it. Both of those responses miss the actual problem.

The question every agency using these systems should be asking right now is not "does this technology work?" It is "does our workflow prevent the technology's failure modes from becoming someone's wrongful arrest?"

If you cannot answer that question in detail, you are not ready to use the tool.