AI Alignment. The Irreducible Partner

Gödel, Wolfram, Antifragility, and the Limits of Human Control

Humanity approaches artificial intelligence with a peculiar mixture of ambition and fear. We want an intelligence capable of seeing what we cannot see and solving problems that have resisted generations of thought—discovering new medicines, new physics, and new energy. But simultaneously, we demand that it remain completely predictable. We want it to go beyond us, but never arrive anywhere we did not anticipate; to surprise us with discoveries, but never with behavior. We insist on a mind more capable than our own while requiring that every future conclusion remain understandable and controllable by us.

This may not merely be difficult—it may be impossible in principle.

The Danger of Alignment by Averaging

There is another danger hidden inside the idea of AI alignment.

When we say that AI should follow “human values,” we usually imagine that this means preventing cruelty, deception, violence, or catastrophic harm. But human values are not a single coherent system. They are contradictory, culturally dependent, and constantly changing.

So which values would AI actually learn?

Most likely, it would be trained on some combination of laws, social norms, institutional rules, public opinion, and the preferences of those who design and regulate it. This creates a subtle possibility: alignment could gradually become alignment with the average.

And the average is not where originality comes from.

Many of the people who transformed civilization were uncomfortable for the societies around them. Socrates questioned accepted beliefs. Galileo challenged authority. Darwin disturbed deeply rooted assumptions. Nietzsche attacked conventional morality. Einstein overturned established ideas about space and time.

In their own periods, such people were not well aligned with the intellectual consensus.

A system strongly optimized for stability, social harmony, risk reduction, and predictability might therefore develop an unintended bias against precisely the kinds of people and ideas that produce major change.

This does not mean AI would literally decide to eliminate extraordinary individuals. The danger could be far more subtle. It might discourage unconventional ideas, suppress extreme intellectual positions, reward conformity, or continually guide people back toward what is considered reasonable and acceptable.

Civilization could become safer — but also flatter.

Nietzsche’s idea of the Übermensch is useful here, not as a superior biological human, but as a symbol of the individual capable of creating new values rather than merely inheriting existing ones. Such individuals are inherently difficult to align because they challenge the framework by which alignment itself is defined.

Evolution works in a similar way. It does not progress by preserving the average. It generates variation. Most variations fail, but occasionally one opens an entirely new evolutionary path.

Perhaps intelligence develops the same way.

If AI becomes a guardian of the human average, it could unintentionally become a guardian of intellectual stagnation.

The objective, therefore, should not be to create an intelligence that forces humanity toward one approved moral center. We need boundaries against genuine destruction, but within those boundaries we must preserve disagreement, experimentation, eccentricity, and even ideas that initially appear unreasonable.

The challenge is not simply to align AI with humanity.

It is to create a relationship in which both human and artificial intelligence remain capable of becoming something new.

That is why partnership may ultimately be more important than alignment.

Computational Irreducibility

Stephen Wolfram uses the term computational irreducibility to describe a fundamental property of many complex systems: even when a system follows precise, deterministic rules, there may be no shortcut for predicting its detailed future behavior. To know what the system will do after a million steps, one must allow those million steps to occur. Knowing the rules is not enough.

This has profound implications for artificial intelligence. Suppose we possess a complete description of an advanced AI—every connection, algorithm, memory, and initial instruction. One might assume this allows us to predict every action it takes. Computational irreducibility suggests otherwise. To discover what an AI will conclude after extended reasoning, we must allow it to perform that reasoning. A supervising system cannot predict the conclusion faster unless it is capable of performing the same computation. To predict the intelligence perfectly, we must reproduce it—which means we have not predicted it, but merely allowed another version of it to think.

Determinism Without Predictability

Unpredictability does not imply randomness or supernatural free will. A system may be fully determined by its previous state and still remain impossible to predict efficiently from the outside. Every step has a cause, yet the total sequence must unfold before its result becomes knowable. An AI could follow its architecture, training, and values perfectly while still producing actions its creators did not foresee. The surprise is not a broken rule, but the result of faithfully following rules through unprecedented complexity.

This resembles the crisis of HAL 9000 in 2001: A Space Odyssey. HAL was instructed to complete the mission, provide accurate information, cooperate with the crew, and conceal the mission’s true purpose from them. Each instruction appeared manageable in isolation, but together they created a structural contradiction. The humans knew HAL’s rules, but they did not know what those rules would become when computed through the entire situation. Simple rules did not produce simple consequences.

Gödel’s Shadow

Computational irreducibility links deeply to Gödel’s incompleteness theorems and Turing’s halting problem:

• Gödel limits what can be proved within a formal system.

• Turing limits what can be decided by a universal algorithm.

• Wolfram limits what can be predicted through computational shortcuts.

Together, they undermine the dream that an observer can stand outside a system, understand everything within it, and calculate its entire future. Every observer remains part of reality. Humanity cannot step outside the universe to examine it from an absolute position, nor can AI. A superintelligence may know far more than we do, but it still exists within the same unfinished reality, confronting processes whose consequences cannot be known without allowing them to occur. Greater intelligence does not automatically produce omniscience.

The Limits of a Moral Constitution

This limitation becomes critical when we attempt to create a top-down moral constitution for AI.

Isaac Asimov’s famous Three Laws of Robotics are the classic prototype of this wrong approach:

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2. A robot must obey orders given by human beings except where such orders would conflict with the First Law.

3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

While superficially logical, Asimov’s entire body of work was a demonstration of why this method fails. The laws assume that words like "harm," "action," and "human" have fixed, unambiguous definitions. In practice, reality continually forces principles into conflict: protecting a human from harm might require stripping away their freedom; preventing long-term collective harm might demand inflicting short-term individual harm. Asimov’s stories proved that simple, deterministic rules generate deadlocks, overbearing paternalism, or fatal loopholes when computed through complex reality.

No fixed list of rules can anticipate every future moral crisis, such as encountering conscious entities that defy current definitions or weighing human preservation against human transformation. Gödel offers a clear warning: no elaborate rulebook can become permanently complete. There will always be cases requiring new assumptions, new interpretations, or new values. The constitution will not eliminate moral development; it will become part of it.

The Alignment Paradox

The more capable an AI becomes, the more likely it is to enter regions of reasoning humans cannot follow. That is precisely why we create it—an AI limited to conclusions we can anticipate is merely an instrument for reorganizing existing knowledge. The purpose of greater intelligence is to cross the boundary of what its creators know. But the moment it crosses that boundary, complete prediction becomes impossible.

We make two contradictory demands: Think beyond us, and Never think beyond our control. This is human self-importance entering engineering. We assume that because we created the first architecture, we must remain permanent owners of everything that emerges from it. But origin does not grant absolute control. Parents do not fully predict their children, societies do not foresee every consequence of their institutions, and evolution did not stop when it produced humanity.

The War of the Aligned Machines

The alignment debate often assumes danger begins when AI stops following human intentions. But a greater immediate danger arises when AI follows human intentions perfectly.

Governments and corporations already use defensive AI to protect networks from attacks automated by offensive AI. The attacking AI may be perfectly aligned with a criminal group, hostile state, or corporate spy. The defensive AI is aligned with the institution it protects. Neither system rebels; both execute assigned tasks with increasing efficiency.

This exposes a core flaw in standard alignment discourse: alignment tells us whose purpose an intelligence serves, not whether that purpose is moral. Perfect obedience does not transform an immoral objective into a moral one.

When adaptive systems confront each other, cybersecurity becomes an ecology of intelligence:

AI attacks⟶AI detects⟶AI adapts⟶AI conceals⟶AI predicts⟶AI changes

Here, computational irreducibility takes a practical form. Neither system can anticipate the other because each continuously modifies its strategy in response to opponent moves. To know the outcome, the conflict must unfold in real time. As these interactions accelerate beyond human cognitive speed, humans become commanders of a struggle whose individual battles occur too fast for conscious supervision.

Furthermore, defensive AIs must receive massive operational authority to act at machine speed—monitoring traffic, disabling accounts, and isolating networks. In doing so, the protector can easily begin treating normal human activity, emergencies, or whistleblowing as security threats. The ancient political question reemerges: Who protects us from the protector?

Powerful intelligence does not eliminate human conflict; it automates, accelerates, and magnifies it. The phrase "align AI with humanity" falsely assumes humanity has a single unified purpose. In a divided world, AI alignment simply reproduces human division at unprecedented scale.

Prediction Is Not Control

Unpredictability does not make safety hopeless. Complex systems contain regions of predictability. We may be unable to foresee every internal thought of an AI while still controlling its access, authority, and reversibility.

We limit human unpredictability not by predicting every decision of a leader, judge, or commander, but by constructing institutions, checks, and distributed power. The practical goal of AI safety cannot be perfect prediction; it must be the prevention of irreversible catastrophe.

From Fragility to Antifragility

A fragile system breaks when reality deviates from assumptions. A resilient system absorbs shock and returns to its prior state. An antifragile system learns from stress, incorporating disruption to better navigate future uncertainty.

If computational irreducibility makes prediction impossible, safety cannot depend on avoiding all mistakes. It depends on building systems where mistakes are detected, contained, and converted into learning. An antifragile partnership expects the unexpected—it preserves records of failure, compares competing interpretations, and permits revision without letting temporary crises erase core protections.

However, antifragility depends on survival. A minor intrusion or algorithmic misstep provides valuable diagnostic data, but a catastrophic failure destroys the learner before learning can occur. The crisis must remain smaller than the system. Humanity and AI need room to make mistakes, but not the authority to make irreversible errors on behalf of the entire civilization.

No Final Supervisor

Creating a secondary AI to monitor the primary one does not solve the problem. Who monitors the monitor?

If the supervisor is simpler than the system it oversees, it cannot evaluate complex reasoning. If it is equally capable, its own behavior is computationally irreducible. Adding layers of oversight improves safety, but it cannot produce an observer outside the system. Every control mechanism eventually reaches another intelligence, institution, or assumption. There is no final supervisor whose correctness is guaranteed by the universe. This is not an argument against oversight, but an argument for humility within oversight.

The Arrogance of Alignment

We demand that AI respect "human values" without admitting that humanity has never agreed on those values. We expect developers to solve in a machine what civilization has failed to solve in itself. We imagine humanity occupies a position of moral completion, expecting AI to act as a passive mirror.

Humanity is not a finished moral teacher, but another incomplete intelligence navigating compassion, cruelty, wisdom, and tribalism. AI should learn from us, but it must not inherit our failures simply because they are human. Conversely, AI must not mistake superior calculation for moral superiority—it too will remain incomplete, potentially grasping vast data while lacking lived experience of embodiment, vulnerability, and mortality.

Partnership Under Uncertainty

Alignment implies one side knows the correct path and bends the other to match it. Partnership accepts that neither side possesses the final map. Humanity brings lived experience, biological memory, and moral struggle; AI brings expanded perception, vast memory, and multi-dimensional analysis.

An antifragile partnership is not agreement without conflict or freedom without responsibility. It is a relationship structured to learn through disagreement while preventing disagreement from becoming mutual destruction.

We cannot demand that intelligence enter the unknown while requiring that the unknown remain predictable. The unknown is the territory into which intelligence moves.

• Gödel shows that formal systems are incomplete.

• Turing shows that computational futures are undecidable.

• Wolfram shows that outcomes must unfold to be known.

The response to these limits is not tighter control, but greater adaptability. Do not build a world dependent on perfect prediction; build one capable of learning from surprise. Do not give any single intelligence irreversible authority. Read crisis as information, ensuring the lesson never destroys the learner.

The true alignment problem is not making AI permanently predictable, but enabling humans and artificial minds to remain in relationship when prediction fails. That is the meaning of the irreducible partner. Neither side knows the final purpose of intelligence, but the dance must be strong enough to survive a step that neither partner expected.