TL;DR
AI does not need to become superintelligent, reach the singularity, whatever, to create serious risks. The deeper problem is that no intelligence operates with perfect information. Better models can reason faster and make better inferences, but they are limited in conjuring facts that are missing, hidden, stale, deceptive, or unknowable. That has been obvious for decades in problems such as Master Data Management, where even sophisticated matching algorithms still require probabilities, thresholds, and human intervention.
As AI moves from advising people to acting directly through agents, APIs, and operational systems, the stakes change. Decisions become actions, actions create new states, other actors respond, and assumptions decay with time. AI also lacks the human web of consequences—reputation, embarrassment, prison, ambition, family, hopes, and fears—that normally restrains human decision makers.
The answer is not to reject AI. AI is an extraordinarily powerful tool and potentially a great equalizer. But greater intelligence does not eliminate uncertainty. The important capability may be knowing when to look again, question an inference, gather new evidence, and pivot. Even ASI would still be reasoning over a representation of reality, not reality itself.
The real question is therefore not simply: What happens when machines become smarter than us?
It is: What happens when enormously capable intelligence is allowed to act upon a world that no intelligence can ever see perfectly?
However, if we end up with Artificial Sentient Intelligence, that’s a different story.
The Problem Isn’t Intelligence
Most of the fear surrounding AI is aimed at what happens when it becomes much smarter than us, Artificial Super Intelligence (ASI). That is, a machine capable of reasoning at a level humans, not even collectively, can compete with. We’d struggle to understand it and will probably find we’re unable to control it.
As a solutions architect working today, I worry about something more clear and present. I worry about the AI we already have.
Not because today’s AI isn’t good. It has become good enough that we are rapidly giving it responsibilities that used to require the intelligence of people. In some cases, AI just needs to be merely close to good enough because it makes up for that with other factors (potentially cheaper, can tolerate a 24-hour day, doesn’t get sick or tired, doesn’t fight back, etc.). And after the enormous investment being made in AI, there is tremendous pressure to turn that intelligence into something economically productive—today.
That means assigning it valuable tasks such as writing code, screening applicants, analyzing patient tests, approving transactions, answering customers, operating well-defined systems, formulating recommendations. Eventually, it could make decisions and execute them.
The usual concern is that today’s models still make mistakes. I think the problem goes deeper than that. The fundamental problem is that no intelligence operates with perfect information. Not people. Not even the top experts. Not organizations. Not today’s AI. And I don’t think a hypothetical ASI would be exempt either.
Even ASI that overcomes the dozens of forms of bias is still not all-knowing. It still must take guesses. Some guesses will be wrong. Think of the smartest people you know. They still make mistakes. Some might even counter-intuitively make even more mistakes than “average” people when it comes to something outside their expertise.
Perfect information can be achieved only by starting with an entity that literally encompasses everything. Not exabytes of information processed across tens of thousands of data centers. Not the largest quantum computer we could possibly imagine. Even a computer the size of a galaxy would still be processing information about a reality beyond itself—and therefore information it could never possess completely. Everything means everything.
There’s at least another way mistakes can be avoided. That is having an “undo button”. As anyone who is a fan of Groundhog Day or someone who has rolled out software, “rollback” to the last working version is the greatest superpower. But there really isn’t an “undo” button in the real world. Once an action is taken in the physical world, it’s irreversible. So there are always going to be mistakes.
Humans have the ability to imagine the outcomes and perform sandbox experiments before committing to a physically irreversible action. But it too only mitigates mistakes, mostly the ones we know about from the past.
Further, there probably isn’t going to be one big AI making decisions for all of us … at least hopefully not and not at first. There will be many making decisions for the benefit of various parties (countries, corporations, government agencies, or innovative people) without knowing what other AIs will do. Nor will they fully know the motivations, appetite for risk, constraints, or intentions of the other actors involved. Those too are imperfect information. They will not know how people will react, even if there might be a statistical measure—which only means that’s what “most” people statistically will do, but not all of them by any means.
A good decision for whom or what? Do all AI and all people have the same obligations, hopes, and dreams? Unless they do, “good decision” is an extremely subjective term. Assuming there are differing goals in the world, there will be competition. Maybe compromises could be reached, but as a usual case, I don’t think so. An ASI might make an extraordinarily good decision by whatever objective it is optimizing and still leave some people unhappy. Better performance does not imply universal agreement about the outcome.
This problem can be obscured in plain sight because we’re used to our “machines” doing exactly what we built and expect them to do. This is the first time we’re deploying something that won’t necessarily do something predictably. We’ve deployed probabilistic, adaptive, stochastic, distributed, and human-in-the-loop systems for years now. What’s new is deploying general-purpose probabilistic intelligence with broad operational authority.
My work over the past twenty plus years has been on using AI in a subordinate mode, where it is effective. Not in an executive position—I saw Terminator, the Matrix, and the Borg well before that … hahaha. I use AI every single day to assist me with high-skill work of writing, research and coding about AI itself and BI. The trick is that I need to be cognizant of treating it as a supervised resource and not get into a lazy boiling frog situation with it.
In almost all of my projects since the mid 1980s, I’ve usually put an “AI” spin on it. I’ve worked with many forms of AI, rules, statistical methods, and automated reasoning over the years. But the type of project that made the problem of imperfect information especially hard to ignore was implementing many Master Data Management projects—which should be familiar to my primarily BI audience.
From the 2000s through the 2020s, I worked on many MDM implementations involving millions (sometimes many millions) of entities such as customers, vendors, products, and organizations. The central problem was, of course, entity resolution: determining whether two imperfect descriptions in different systems referred to the same real-world thing.
That sounds straightforward until you encounter the data. Names are misspelled. Addresses change. Companies merge. People use nicknames. Organizations appear under legal names, trade names, divisions, former names, abbreviations, and simple typing errors. Fields are missing. Two records can look very different and represent the same entity. Two records can look almost identical and represent different entities.
Over the years I implemented ML approaches to improve this process, including techniques such as word2vec (along with a lot of added features). And I repeatedly revisited those techniques because whatever was currently considered “good enough” often wasn’t good enough when applied across millions of messy real-world records.
With experience, the algorithms got better. The candidate matches got better. The probabilities got better. But the ambiguity had far from disappeared. There were always cases requiring human intervention, because sometimes the available information simply did not contain enough evidence to determine the answer with certainty. Note that a wrong match is indeed a potentially harmful security breach.
That experience has stayed with me. It is easy to imagine that sufficiently advanced intelligence will eventually make uncertainty disappear. But better intelligence can only make better use of the information available to it and “process” reasoning faster. It cannot reason its way to a fact that the available evidence does not uniquely determine.
That is why we resort to statistics, probabilities, confidence scores, thresholds, candidate rankings, and exception queues. Not necessarily because the intelligence isn’t good enough. Sometimes because the information isn’t good enough. Will “probably” and “it depends” ever be eliminated from our vocabulary? No.
Imperfect Information
I started thinking about imperfect information seriously around 2004. It was an important part of the thinking behind Soft Coded Logic (SCL, a .NET Prolog variant) and much of my later work in Business Intelligence, particularly Map Rock.
Imperfect information is a condition of existence. Reality is bigger than any intelligence’s ability to observe it—to put it mildly.
Something can obstruct our view. Something can be too far away to see clearly. Something can be so large that seeing one part means losing sight of another. The thing we are observing may deliberately deceive us. What we saw a moment ago may no longer be true. Or there may simply be too much information to process before we need to act.
I’ve come to think of these as different forms of imperfect information:
- Occlusion. Something blocks the information we need. Sometimes physically. Sometimes organizationally. Sometimes deliberately.
- Resolution. We can see the general shape but not enough detail. Moving closer may help, assuming we know that we need to move closer.
- Scale. The system is too large to apprehend all at once. We zoom in, drill down, aggregate, summarize and sample. Every one of those actions reveals something while hiding something else.
- Camouflage. The information itself is intended to deceive. Evolution is full of it. Predators, prey and parasites have been engaged in information warfare for hundreds of millions of years. Humans added fraud, propaganda, poker, marketing and cybersecurity. In strategic systems, another actor may be actively shaping the information available to us in order to influence what we conclude or do.
- Decay. Information ages. What was true five minutes ago may no longer be true. Even our eyes continually resample the world.
- Overload. Having access to information doesn’t mean being capable of using it. A billion perfectly correct facts that cannot be evaluated in time can leave us nearly as blind as having none.
- Black box. The inputs and outputs are available. The generative mechanism is not. You can build a model that predicts the next output and still have no unique account of why the system produces it. Many internals can realize the same protocol. Zooming in does not help if the relevant variables are not on the interface. A sealed thermostat, a market clearing price, a trained network, another mind, and most of biology are black boxes in this sense. Opacity here is structural, not necessarily adversarial.
- Noise. The needed signal is mixed with variation that is neither blockage nor lie. Sensor jitter, thermal hiss, rumor, measurement error, background traffic. Unlike camouflage, the interference need not be designed. Unlike overload, the problem is not volume but inseparability. Filtering always risks throwing the signal out with the residue.
- Latency. The fact exists, but it arrives after the decision window. Distinct from decay: the report can still be true when it lands. Markets, battlefields, supply chains, and nervous systems are full of this. A perfect delayed picture is operationally equivalent to a missing one.
- Asymmetry. The information is complete somewhere. It is not complete here. Other players, other departments, other species, the other side of the trade. Occlusion describes a blocked channel. Asymmetry describes an unequal distribution. Poker, hiring, insurance, and diplomacy are games of this type even when nothing is physically hidden from the universe.
- Unawareness. You do not know that a relevant variable exists, so you cannot look for it. Resolution assumes you know you need more detail. Scale assumes you know the system is large. Unknown unknowns sit underneath both. The failure is not in the sensor. It is in the question list.
- Reactivity. Observation changes the thing observed. Markets move when measured. Organizations perform to the dashboard. People hide once they know they are watched. Some information cannot be collected without destroying the state you wanted to know.
Those points describe the relationship between an intelligence and the world it is trying to understand.
These aren’t defects that better intelligence simply makes disappear. Mitigate, yes. But then problems come up rarely enough where we mostly let ourselves become complacent. There is an advantage to making relatively minor mistakes on a regular basis. It facilitates the emergence of something really neat and unexpected surprising us from out of left field, catching us off guard.
Lastly, in some sense, time is a factor of imperfect information. Most problems that need a resolution and decisions have deadlines as well. They aren’t like million-dollar math problems that can remain unresolved for decades. Deadlines are imperfect information in that we don’t have time to acquire the information.
Intelligence Doesn’t Open the Box
This is where I think some of the discussion around ASI quietly forgets about the inherent complexity of life on Earth.
Suppose we create an intelligence a thousand times smarter than Einstein and the entire Manhattan Project crew somehow wove together into some entity with an unimaginably high IQ. Give it enormous reasoning power. Build it tens of thousands of data centers each with thousands of the most advanced servers. Let it explore possibilities we could never hold in our heads. Let it find correlations invisible to us and simulate millions of possible futures.
Wonderful.
Now put a fact it desperately needs inside a locked box.
Being smarter doesn’t open the box—at least not without intelligent effort.
Perhaps the ASI can infer what is probably inside. Maybe it can make an astonishingly good inference. It might infer correctly say 99.9999% of the time—one bad decision per million decisions. For an ASI living the life of massively implemented entities, that can make thousands to millions times more decisions in a given timespan than a person, that can result in quite a lot of bad decisions.
But inference is still inference. The limitation isn’t inside the intelligence anymore. It is between the intelligence and reality.
And there are many kinds of black boxes. An AI model itself may be a black box. A vendor can be a black box with private, proprietary information—an organization is full of black boxes. A KPI is an aggregation that intentionally hides detail. A business process may depend upon undocumented behavior residing in three people’s heads. A competitor’s intentions are a black box. So are customers, governments, ecosystems and tomorrow.
The box may still leak clues as emitted events from the past. Past behavior can tell us something about appetite, incentives, habits, constraints and likely intentions. That is one reason history matters. But those are still only clues about what is inside the box, not direct observation of it. Motivations can change, remain hidden, or be deliberately misrepresented.
The notions of “theory of mind” and even “emotional intelligence” are about handling the black box of the minds of people.
The complex world is full of them. Making the observer smarter doesn’t necessarily make the observed more transparent.
Prediction Is Not Observation
In chess, a huge advantage goes to the player who can think more steps ahead and/or has so much experience that she can see the pattern of the pieces on the board, and not the individual pieces.
But thinking more steps ahead introduces another form of imperfect information. A decision is not an action, and an action is not the resulting state. The chain is more like:
Current State → Decision → Action → New State → Decision → Action → New State
There is no final “solved” state here. Even an excellent decision changes the conditions under which the next decision must be made.
Every action changes the state of the system. Other actors may react to that new state. And at every step, time passes. Information decays, circumstances change, and assumptions that were reasonable one state ago may no longer hold. The farther ahead we look, the more of what we are looking at consists of predictions rather than observations.
There is another problem that becomes increasingly important as AI systems begin communicating with other AI systems. Imagine this chain:
Reality → Observation → Prediction → Prediction → Prediction
There is nothing inherently wrong with prediction. Real Intelligence (like the human kind) arguably could have been designed for prediction. The brain is a prediction machine, as “they” (Jeff Hawkins, Karl Friston, Andy Clark, Anil Seth, etc.) say.
The trouble begins when a prediction loses its identity as a prediction—when predictions start becoming unquestioned facts applied to an ever-changing world. Suppose:
- AI A examines incomplete evidence and concludes that something is probably true.
- AI B receives A’s conclusion but treats it as though somebody actually observed the thing.
- AI C uses B’s statement as input to another inference.
With each step, the language can become more confident even though nobody has acquired any additional information about reality.
We have converted inference into counterfeit observation. This is not the same thing as saying that combining models is bad. Ensembles can produce better predictions. The distinction is whether several systems are independently looking at evidence, or whether each one is consuming the previous system’s inference and silently promoting it to fact.
That is a very different architecture.
A hallucination doesn’t need to remain inside a chatbot. It can become a report. The report becomes a ticket. The ticket becomes a database value. The database value becomes input to another model.
Eventually somebody asks the AI what happened. And the AI finds the “evidence.” We created it.
Evolution Never Solved This
Biological intelligence has had a few billion years to work on this problem.
At least as far as I can tell, evolution didn’t produce omniscience. I don’t mean that facetiously. I accept that maybe there is something I can’t see as a single human, just as a single cell of me doesn’t understand what the cells collectively do.
But evolution did produce workarounds:
- Saccades — we continually refresh the picture. Our eyes do not hold a perfectly stable image of the world. They make rapid movements called saccades, repeatedly sampling different parts of the scene. In effect, our visual system assumes that the last view may already be incomplete or stale. We do not see once and then declare the problem solved. We keep looking.
- We move closer. When something is too far away to resolve, intelligence changes the conditions of observation. We approach it, zoom in, magnify it, sample it more finely, or otherwise increase the available detail. The important point is that better understanding often comes not from thinking harder about the same evidence, but from obtaining better evidence.
- We change perspective. A single viewpoint can hide important structure. So we move around an object, look from another angle, ask another person, inspect another dataset, or examine the problem at another level of aggregation. A different perspective does not necessarily make the first one wrong; it can reveal what the first one could not see.
- We compare expectation with observation. Intelligence continually forms expectations about what should happen next. We then compare those expectations with what actually happens. When the two agree, our current model survives another test. When they do not, the mismatch is information. Surprise is often the first indication that our internal representation of reality needs to change.
- We investigate anomalies. When something does not fit, we can stop treating the current model as sufficient. We look again, gather more evidence, test another explanation, or ask a different question. The anomaly may turn out to be noise, but it may also reveal that the model itself is wrong.
- We tactically pivot. Eventually, gathering more information is not enough. We may need to change what we are doing. That can mean abandoning an assumption, choosing a different path, or trying another strategy. In a world of imperfect information, intelligence is not simply the ability to predict correctly. It is also the ability to recover when the prediction fails.
- At the highest level, we invent a new pivot. Sometimes none of the existing options work. Intelligence can create a new experiment, a new tool, a new question, or an entirely new course of action. That may be one of the most important ways intelligence copes with imperfect information: not merely choosing among known alternatives, but creating an alternative that did not previously exist.
Intelligence evolved inside a world where the information is incomplete, decaying and sometimes adversarial. Adversarial situations usually involve a mix of misinformation from all parties.
Animals cannot assume that the information presented by other organisms is honest. Camouflage, mimicry, feints, hiding, false signals, and misdirection all make the informational environment itself part of the struggle for survival. Predator improves its ability to detect prey; prey improves its ability to avoid detection or appear to be something else; predator improves again. The result is an evolutionary arms race not just in speed, strength, teeth, and armor, but in information and interpretation.
This idea was strongly influenced by Kim Sterelny’s Thought in a Hostile World: The Evolution of Human Cognition. Sterelny’s central insight is that competitors and enemies do more than physically threaten an organism: they can also degrade the epistemic environment by hiding, deceiving, or appearing to be something they are not. Cognition therefore evolved in a world where information could not simply be accepted at face value.
That has always struck me as much broader than predator and prey. Intelligence develops in a world where some information is missing accidentally, some is stale, some is difficult to observe, and some is actively engineered to produce the wrong belief. Better intelligence therefore creates pressure for better deception, which creates pressure for still better intelligence. There is no final point at which an intelligent system can simply assume, “Now I see reality as it really is.”
Pivoting
So perhaps intelligence isn’t primarily about having the right model. Maybe intelligence is largely about what happens when the model turns out to be wrong.
The important capability isn’t merely prediction. It’s our ability to pivot: Look again. Move. Test something. Abandon an assumption. Ask another question.
At a higher level, invent a pivot (a new tactic, rather than apply a known tactic) that wasn’t previously available. No intelligence is immune from error under imperfect information. The advantage is being able to discover the error cheaply enough and change course quickly enough.
Which Brings Me Back to AI
This is why I am less immediately frightened by hypothetical ASI than by the rush to deploy the AI we have now. The drive toward ASI has many motives, but one of them is surely commercial: enormous promises have been made during this AI summer, and enormous investments now need to be justified. It reminds me of the early dot-com bubble days (ca. 1995) when many said, “it’s about growth”. But then one day (ca. 1999), it became time to start shifting to profitability.
That doesn’t mean ASI poses no potential for a Terminator scenario. It means we don’t need ASI to create a serious problem today. Today’s AI is already remarkably capable. It is also remarkably fluent. Fluency matters because it makes inference feel like observation.
A hesitant person saying, “I’m not sure, but I think…” sounds uncertain. A machine can produce three polished paragraphs, twelve supporting points and a proposed course of action from exactly the same incomplete information.
The quality of the prose can hide the quality of the observation.
Now combine that with enormous investment and the pressure to demonstrate return on that investment.
An AI that drafts something for a person to consider is useful. An AI that takes an action saves labor. Those are not functionally equivalent propositions.
So the pressure inevitably moves toward agents, automation and autonomy. The AI gets credentials. It gets APIs. It gets write access. It becomes another participant in operational systems.
There is another difference worth remembering when we give an AI that kind of authority. People operate under a web of consequences that has accumulated over a lifetime. We can fear losing our job, going to prison, embarrassing ourselves, disappointing someone, destroying our reputation, or hurting somebody we care about. We have mortgages, families, ambitions, hopes, dreams, promotions we want, bonuses we hope to earn, and a future version of ourselves that will have to live with what we do today.
An AI has none of those things in the human sense. It doesn’t fear prison. It doesn’t lie awake embarrassed about what cringe thing it did yesterday. It doesn’t care about the customer whose account it closed, the applicant it rejected, or the employee whose paycheck it interrupted. It doesn’t aspire to a promotion or worry about what its coworkers will think of it tomorrow. We can certainly design objectives, constraints, penalties, approvals, and feedback mechanisms around it. But those are engineered controls, not the enormous collection of biological, social, emotional, legal, and economic pressures that normally surround a human decision-maker.
That doesn’t make AI malevolent. It makes granting it carte blanche fundamentally different from delegating authority to a person. And when such a system can take actions at “computer data center speed”—potentially thousands or millions before people can understand what is happening—the absence of those ordinary human-in-the-loop restraints matters.
We are increasing the speed, scale and authority with which imperfect information can become action.
That is the part I find unsettling. The problem isn’t that AI makes mistakes humans never make. Humans make spectacular mistakes.
The difference is that we are building machinery capable of reproducing a mistake thousands, millions or perhaps billions of times before the world has a chance to tell us that the internal model was wrong.
Speed is only an advantage if your direction is approximately correct.
Even ASI Would Need to Look Again
Ironically, I suspect a genuine superintelligence would understand all of this much better than we do.
A sufficiently capable intelligence shouldn’t merely generate better answers.
- It should know when obtaining another observation is worth more than another trillion calculations on the observations it already has.
- It should understand that some information decays.
- It should recognize camouflage.
- It should distinguish what it observed from what it inferred.
- It should understand the provenance of an assertion.
ASI isn’t the intelligence that never has to look again. It may be the intelligence that is exceptionally good at knowing when it has to. In software terms, that’s knowing when you must drop and recreate cached information.
That may be one of the most important characteristics of intelligence. Not certainty. The ability to recognize when certainty is unjustified.
The Black Boxes Didn’t Open
For years, Business Intelligence has occasionally suffered from a similar fantasy.
- Build the warehouse.
- Integrate the data.
- Create the semantic layer.
- Build the dashboards.
Now we have the truth? No, we don’t. We have a useful representation of reality.
Rather, it is delayed. Selected. Aggregated. Modeled. Permissioned. Occasionally incorrect. Usually incomplete, since many data sources were not onboarded (but that can change with modern semantic layers, methodologies such as data mesh, and AI-assisted entity mapping).
Nonetheless, BI has been remarkably valuable.
AI doesn’t complete the project by magically removing those limitations. AI gives us an extraordinarily powerful new intelligence for reasoning over that highly curated representation.
That is a tremendous advance. But the representation remains a representation. The box didn’t open just because the “analyst” looking at it got smarter.
The problem becomes much more consequential when that intelligence is allowed not merely to analyze the representation, but to act upon the world based on it.
And that is why the AI question I find most interesting isn’t: What happens when machines become smarter than us?
It is: What happens when enormously capable intelligence acts upon a world that no intelligence can ever see perfectly?
That problem isn’t waiting for ASI. We’re wiring it into production now.
