AI's Trust Crisis: Why Hallucinations and Black Boxes Block Adoption

The essay opens by charting AI's extraordinary recent progress: systems like ChatGPT, Claude, and Gemini now converse fluently, draw on humanity's written knowledge, and increasingly reason using language as a substrate for thought. For the first time, the arrival of what some call Artificial General Intelligence (AGI)—defined in OpenAI's charter as 'highly autonomous systems that outperform humans at most economically valuable work'—seems plausibly imaginable. Yet despite this progress, the author argues, 'something fundamental is broken.' The systems being built are powerful but intrinsically untrustworthy, and this trustworthiness problem creates what the author calls a 'trust ceiling' where trillions of dollars of value are stuck. No business will allow AI to fully automate core functions if at every step the system might hallucinate an outrageously false answer; no regulator will approve widespread deployment when every decision is a black box subject to unpredictable error. Until this trust problem is addressed, 'the most economically valuable work' will remain in human hands.

To understand the issue, the essay steps back to examine two fundamental software paradigms. Symbolic software—traditional programming with algorithms, data structures, and databases—operates on the principle that fully correct is the standard. A spreadsheet adding two cells must produce the right answer every time; if it does not, it is a bug to be found and fixed. These systems are deterministic, consistent, explainable, and fully controllable. The second paradigm is statistical machine learning, particularly large neural networks ('deep learning'). These systems are not programmed but learn from data; they are large mathematical formulas initialized randomly and improved through training until they generate useful results. Crucially, 'the results however are never guaranteed.' They are approximations with performance always below 100%—best guesses. What makes modern AI powerful is that these deep learning systems improve as they scale: more data for training, more compute, larger models mean continued performance increases. When neural networks first saw commercial use in the late 1980s, they had tens of thousands of parameters and trained on megabytes of data. Today's frontier models have hundreds of billions to trillions of parameters and train on a meaningful fraction of everything humanity has ever written. This astronomical increase in data and compute, plus innovation, has driven similarly astronomical expansion of AI capability—the engine behind current progress. But with this power comes a fundamental problem: neural systems are intrinsically untrustworthy.

The essay defines trustworthiness not as mere willingness to rely on a system (which people may do out of convenience even when they should not) but as 'the properties that make reliance warranted.' For serious work, those properties go beyond being right; they include knowing where an answer came from, getting the same answer to the same question every time, being able to reconstruct and contest a decision, and knowing which rules governed it. Measured against that standard, neural systems fail in four significant ways. First, accuracy: in AI research, progress is celebrated when a model improves from, say, 87.3% to 89.2% accuracy—but this also means the system is wrong more than one in ten times. Benchmarks are declared saturated in the mid-90s, and the community moves to harder ones; saturation never means 100%. When errors affect brand, reputation, financial decisions, legal outcomes, or safety, even small error rates prevent use. And at scale, even tiny error rates produce a torrent of bad results. Second, hallucination: large language models frequently produce answers that are fluent, detailed, completely believable but outrageously false. This is 'the worst kind of wrong' because the system sounds authoritative while fabricating facts. This failure mode cannot be engineered away; it follows from how the machine works. An LLM is built to produce the most plausible continuation of text, and when it does not know the answer, it produces the most plausible continuation anyway: fluent, confident, and fabricated. Research continues to reduce hallucination frequency, but OpenAI's own researchers have shown that factual error is a statistical inevitability of next-word prediction, and hallucination persists because training and evaluation reward confident guesses over admissions of uncertainty. Their proposed remedy—training models to say 'I don't know' more often (calibrated abstention)—is real progress, but 'it reduces the bluffing; it makes nothing checkable.' Several years of recognition of hallucination as a major industry problem and billions spent on R&D have reduced it, not removed it; state-of-the-art models from every provider continue to hallucinate. Trust does not average out: so long as any answer might be fabricated, every answer must be checked.

Third, neural systems are not explainable in any meaningful sense. Modern neural networks consist of billions or even trillions of parameters, initialized at random and shaped by training on vast datasets with astronomical compute. We can observe their behavior but cannot explain why they arrive at a particular answer. A formula with a hundred billion weights derived this way is 'arguably one of the least explainable human-produced things we can imagine: the truest black box it is possible to be.' Even the researchers who dedicated their careers to these systems do not fully understand their internal workings. An entire scientific field, mechanistic interpretability, now works to open the box, and its findings are illuminating; but 'a science of the black box is not the same as an explanation you can rely on.' Partial accounts assembled after the fact are different from a full, faithful, human-understandable record of reasoning behind every answer. Additionally, the essay notes (citing a peer-reviewed paper presented at ICML's 2026 Trust Workshop), reasoning traces from the latest LLMs are fundamentally different from what a good explanation should be. A real explanation points to facts that made the difference to the outcome, but neural networks do not represent data as independent facts, so there is nothing to point to. The paper's conclusion: 'where complete explainability is essential, some part of the system must be something other than a neural network.' Finally, neural systems are not reliably consistent. The same system can produce different, contradictory answers to the same question. Even with randomness switched off ('temperature zero'), outputs shift with trivial rephrasings and reorderings of input that change nothing about meaning. In isolation this may seem minor; in practice, it erodes trust rapidly. A system that cannot make the same decision twice under identical conditions cannot be relied upon. Taken together, these issues amount to a single central problem: modern AI systems cannot be trusted.

The essay traces the current impasse in AI deployment to a fundamental divide between two software paradigms. Traditional symbolic software—algorithms, databases, spreadsheets—operates on the principle that fully correct answers are the standard; when errors occur, they are bugs to be found and fixed permanently. Neural networks, by contrast, are statistical approximations trained on massive datasets using astronomical compute, and error is not an aberration but an inherent property. The author acknowledges that neural networks excel at tasks symbolic software cannot: they improve as scale increases, and this scaling—more data, more compute, larger models—has driven the current wave of progress. But this power comes with a cost: the very properties that make neural networks powerful (their data-driven, learning-based nature) make them intrinsically untrustworthy for high-stakes decisions.

The four trustworthiness failures the essay identifies are not temporary engineering challenges but consequences of how neural networks fundamentally work. Residual error rates are not steps toward 100% accuracy; benchmarks are declared 'saturated' in the mid-90s and the field moves to harder tasks, accepting that error is permanent. Hallucination follows from the LLM architecture—the system produces the most plausible text continuation whether or not it has grounded knowledge—and even training models to say 'I don't know' more often merely reduces the rate of confident guessing rather than making answers checkable. Explainability is impossible at scale: a formula with a hundred billion weights derived from random initialization and training on vast data is a black box that even its creators do not fully understand. Consistency failures (identical inputs yielding different outputs) erode trust by making systems unreliable for repeated decisions. Taken together, these issues prevent AI from handling the most economically valuable work—decisions affecting brand, finance, law, and safety—where even small error rates at scale produce a torrent of bad outcomes.