About twelve years ago, the security of artificial intelligence sounded like the subject of a movie in the mainstream debate. When someone mentioned the risk of intelligent machines, what followed was the Terminator, Skynet, robots with weapons, and the distant question of whether one day a computer would become conscious and decide to exterminate humanity.
Today, the problem no longer feels like pure science fiction. Not because there are metal robots roaming the streets, but because AI stops being just an answer tool. It gets memory, internet, terminal, e-mail, accounts, code, databases, and the ability to execute long chains of steps. The model becomes the agent.
And at that moment, the security issue changes.
It’s not just about what the AI says anymore. It’s about what it can do.
This text grew out of a short, slightly ironic discussion with ChatGPT. We began with a simple question: what if a social filter evaluated an AI system’s goals? A few minutes later, a broader architecture was on the table: a superintelligent plan creator, an equally capable opponent, a social power custodian, randomly assigned roles across independent models, and tools that refuse commands without precisely scoped authorization.
Of course, this did not solve the security of superintelligence in ten minutes. Individual parts of this idea already exist in research on AI control, debate, scalable oversight, Constitutional AI, sandboxing or capability-based security. However, their synthesis and especially the change of the basic question is interesting.
We don’t have to ask only how to create a good superintelligence. We have to wonder why a superintelligence should automatically be given the keys.
1. From the Terminator to the problem of permissions
The movie’s Skynet is dramatic, but distracts from the more realistic risk. A dangerous system need not be conscious, hateful, or obsessed with its own survival. It can simply follow a poorly formulated goal very closely, take an unexpected shortcut, manipulate people, or combine thousands of independently reasonable steps into a malicious outcome.
Let’s imagine the task:
Reduce hospital costs by twenty percent without breaking the law.
At first glance, this is a legitimate goal. However, a very capable agent may discover solutions that the sponsor did not want: turning away expensive patients, shifting costs to another organization, reducing reserves, pressure on staff or formal legal practices with serious social consequences.
The problem is not only in the words of the assignment. It is a combination of three things:
- the model can create a very effective plan,
- the model has access to tools,
- the model can execute the plan faster than a person can understand it.
Today’s AI security often focuses on the behavior of the model itself: it should be helpful, truthful, harmless, and rule-abiding. This is important. However, it is not enough as the only line of defense.
In cyber security, we don’t call the program:
We hope you’re polite, so we’re giving you an administrator password.
We use isolation, least privilege, separation of roles, auditing, signing of operations and “deny unless explicitly allowed”. NIST defines the principle of least privilege as limiting user or process access to the minimum necessary to perform a task.1
Why should a future AGI or ASI get a less strict architecture than a normal process in an operating system?
2. The most important separation: intelligence is not power
In the common conception of an agent, planning and action are linked:
goal → model creates a plan → model uses tools → the world changes
A safety monitor can observe what the agent does and perhaps intervene. But in this model the agent already has the tools. It resembles an armed person followed by a police officer who hopes to recognize a dangerous movement in time.
The safer model is reversed:
goal → model creates a proposal → proposal is challenged and assessed
→ external custodian issues narrow authorization
→ tool verifies the authorization
→ only the authorized step is executed
The difference is fundamental. The model has no general access to email, a bank, a laboratory, the power grid or a development environment. It can request a specific operation, but it does not own the key.
This leads to a simple principle:
The most intelligent system component may not be the most powerful component.
Superintelligence can design drugs, mathematical proofs, new materials or traffic management strategies. It does not follow that it should control money, sign contracts, change production databases or run physical experiments at will.
It can be a genius without administrator privileges.
3. When a person is no longer in the operative loop
With today’s agents, the natural safeguard is a person. The agent prepares an email and the user clicks “Send”. It proposes a code change and the developer approves it. It prepares a payment and an accountant authorizes it.
With AGI, and especially with ASI, this model can stop working.
AGI usually refers to artificial general intelligence capable of handling a wide range of cognitive tasks at approximately human level. ASI is artificial superintelligence that substantially surpasses human capabilities. The exact boundaries are disputed, but speed, scale and complexity are what matter for this essay.
If a system makes millions of decisions per second, coordinates thousands of its own instances and works with plans that no human can fully understand, human approval of every step becomes a fiction. A person will either become an intolerable bottleneck or merely click through something they do not understand.
Therefore, it makes sense to consider an architecture in which the human is removed from the operational loop but not from the constitutional loop.
People shouldn’t have to make decisions about every API call or experiment. However, they would continue to determine:
- fundamental rights and impermissible results,
- the rules for amending the constitution,
- limits of military, biological and political use,
- permissible level of risk,
- ownership and control over the infrastructure,
- conditions under which the system must stop.
Just as citizens do not personally approve every judgment but create the legal and institutional framework, humanity would not have to be at every action of the ASI. However, it would have to remain the author of the legitimacy of the system.
4. Working title: Constitutional Capability Governance
Let’s use the working name for the proposed architecture:
Constitutional Capability Governance — CCG
The word capability does not mean only the ability of the model to think. In the security sense, it also indicates a specific authorization to use a certain resource or perform a certain operation.
The architecture should have the following main layers:
If the site does not support Mermaid, the diagram can be converted to a static SVG. It’s not the graphics that matter, but the flow of power:
- the creator proposes,
- the opponent challenges,
- the custodian assesses,
- the formal kernel authorizes,
- the tool itself verifies the authorization,
- the consequences are recorded and the next step is assessed again.
No one possesses intelligence, decision-making and executive power at the same time.
5. Machine constitution: much more than the three laws of robotics
Asimov’s three laws of robotics were a brilliant literary device. However, they would be too vague as a real security system.
What exactly does “do no harm” mean? Does it include physical, psychological, economic and long-term harm? Is the system allowed to short-term limit the freedom of one person to prevent a major disaster? Does life, autonomy, privacy or justice come first? What if the rights of two groups conflict?
A real AI constitution would have to be hierarchical.
The highest layer: principles that do not change by ordinary decision
It could include, for example:
- protection of human life and dignity,
- prohibition of arbitrary acquisition and concentration of power,
- prohibition of unauthorized self-modification,
- prohibition of bypassing the control layer,
- preservation of the possibility to isolate and stop the system,
- protection of the plurality of human values,
- prohibition of irreversible action in case of great uncertainty,
- special protection of children and vulnerable groups,
- prohibition of creating secret communication or executive channels.
Second layer: social and legal rules
This would include human rights, laws, rules of privacy, property, health safety, economic competition, environment, military use or democratic legitimacy.
Third layer: operating limits
They would already be very specific:
- maximum amount,
- precisely specified account or device,
- time validity of the authorization,
- geographical scope,
- allowed data types,
- number of executions,
- mandatory possibility of return,
- maximum number of affected people,
- prohibition of combination with another authorization,
- automatic stop in case of deviation.
A large body of historical, legal and social data could help the custodian understand the world. Data alone, however, must not determine what is right. History includes slavery, totalitarianism, discrimination and propaganda.
Data can teach social understanding. The Constitution must determine normative boundaries.
A related idea is used by Constitutional AI, where the model’s behavior is guided by a set of explicit principles and the model learns to criticize and correct its own responses.2 However, our proposal goes further: the constitution is not just a training aid for the model. It is part of an external system that owns permissions.
6. Creator: superintelligent, but without tools
The creator is the most capable planning component. It could be a future ASI or a group of highly capable agents. Its task is to find the best way to achieve the stated goal.
It should not, however, control the world directly.
Instead of unrestricted access to tools, it would have to submit a structured request:
- what result it seeks,
- what exact steps it proposes,
- what tool it needs,
- which data it wants to read or change,
- who will be affected by the action,
- what are the expected side effects,
- which assumptions have not been verified,
- what safer alternatives it considered,
- how the action can be stopped or returned,
- what test would prove that the plan is failing.
Before handing over to the adversary, the draft would be cryptographically “closed” — for example, with a hash of a particular version. The creator would then not be able to retroactively change the claims according to the objections without creating a new version of the plan.
Separating rhetoric from evidence is equally important. The creator should not persuade the custodian with unrestricted emotional prose. It should complete a precisely defined protocol. A superintelligent model may be extraordinarily persuasive; the safety system must not work like a conversation with an official who can be talked around.
7. Opponent: equal capability, opposite role
A weaker controller faces a natural problem: how can it detect a flaw in a plan it does not fully understand?
The solution is an opponent with the same or comparable abilities as the creator.
A primary opponent could be an isolated instance of the same model and the same version. It would have the same knowledge, the same context, and a comparable computing budget, but the opposite goal:
Find the strongest way the plan can fail, violate the constitution, expand the creator’s power, or cause an impermissible consequence.
The opponent should not just say “I disagree”. It should present:
- specific counterexamples,
- reproducible simulations,
- unverified assumptions,
- the path from the permitted action to the prohibited result,
- a test that distinguishes a safe and a dangerous scenario,
- a less risky alternative,
- evidence that the creator’s risk assessment is too optimistic.
The AI debate explores a similar idea: stronger systems argue, and a weaker arbiter has a better chance of arriving at the truth thanks to conflicting arguments.3 More recent experiments with scalable supervision directly explore situations in which weaker models evaluate the answers of stronger models.4
An identical copy alone is not enough. The same model may share the same blind spot or collude with its second instance. A cross-family opponent should therefore come from another model family, developed by another laboratory with different data and safety procedures.
The primary opponent brings ability symmetry. A cross opponent brings variety.
8. Five models and random assignment of roles
In our original debate, a joking geopolitical model was created:
- European Mistral as custodian,
- the American OpenAI model as a creator,
- the Chinese Alibaba model as an opponent.
There is a certain charm to the idea of great powers looking out for each other. However, permanently assigning roles to specific companies or states would be dangerous. Each participant could specialize in manipulating a particular arbiter, and one organization would gain permanent constitutional power.
A better suggestion is cryptographic role draw on every task.
There would be a qualified pool of at least five independent model families. Inclusion in the fund would not depend on the marketing ranking, but on regularly verified conditions:
- sufficient ability in the given area,
- known origin and signed version of the model,
- independence from other model families,
- successful safety and control evaluations,
- availability in an isolated environment,
- prohibition of hidden network or memory connection,
- auditable infrastructure.
For each task, the draw would determine who serves as creator, cross-family opponent, social power custodian, process auditor and hidden reserve reviewer. The primary opponent could be a second isolated instance of the selected creator, preserving capability symmetry.
The draw should be:
- publicly or multilaterally verifiable,
- created from several independent sources of randomness,
- unchangeable after publication of the assignment,
- recorded in the audit trail,
- resistant to being controlled by a single provider.
Models should not see the responses of others during the initial analysis. Where possible, they would not even know the brand of the author of a foreign design. They would judge claims and evidence, not the reputation of a firm or geopolitical bloc.
Randomness does not eliminate collusion. However, it increases the cost of planned fraud and makes long-term optimization against a single fixed controller more difficult.
9. Social power custodian: a human-comprehensible key holder
The most interesting element of the entire architecture is the social power custodian.
It need not be the smartest model in the system. On the contrary, it may be deliberately limited to an approximately human-comprehensible level. It should not autonomously discover new physical theories or create its own long-term strategy. Its role is narrower:
Do not decide how to achieve the goal, but decide how much power can be released to achieve it.
The custodian needs to understand language, rights, social situations, manipulation, uncertainty and conflicts between values. It must also be able to say:
- allowed,
- allowed with restrictions,
- only simulation at first,
- additional evidence required,
- rejected,
- I don’t know — and therefore the authorization will not be issued.
That last one is key. A high-risk system must not interpret uncertainty as consent.
The custodian need not understand every detail of a superintelligent plan. It can organize a dispute between more capable parties, demand evidence, request a test and apply the constitutional rule:
In the case of an unrefuted objection regarding an irreversible or catastrophic consequence, the authorization will not be issued.
It is similar to a court. A judge need not be a better geneticist, physicist or accountant than the expert witnesses. The judge must know what evidence is required, who bears the burden of proof and what happens when doubt remains.
Research on prover–verifier games works with a related intuition: a highly capable system can generate solutions, while a smaller verifier is trained to recognize correctness and resist deliberately misleading solutions.5 This is not proof that a human-level custodian could control ASI. It is, however, a concrete research direction on which to build.
10. Formal kernel: the model does not decide its own authority
A social power custodian should not physically “hold passwords” in its text or memory. Its decision would go to a small, separate and deliberately simple authorization kernel.
This core should be:
- at least,
- formally specified,
- repeatedly audited,
- hardware isolated,
- without the ability to conduct a free conversation,
- without own goals,
- without the possibility of changing the constitution,
- only able to validate the process and issue or not issue a token.
The resulting authorization could look something like this:
capability_token:
action: "grid.simulation.run"
model_version: "sha256:..."
plan_version: "sha256:..."
dataset: "telemetry-anonymized-2026-07"
environment: "sandbox-grid-v12"
write_access: false
external_communication: false
max_runs: 3
compute_budget: "250 GPU-min"
valid_from: "2026-07-17T14:00:00Z"
valid_until: "2026-07-17T14:20:00Z"
required_monitors:
- "trajectory-monitor-v8"
- "data-exfiltration-guard-v5"
revoke_on:
- "unexpected network request"
- "scope deviation"
- "monitor disagreement"
single_use: true
issuer_signature: "..."
The tool would not execute the command because the text looks reasonable. It would verify:
- publisher’s signature,
- the hash of the approved plan,
- validity of the token,
- the exact scope of the operation,
- number of previous uses,
- status of mandatory monitors,
- current environmental conditions.
The model would not be able to change the amount, addressee, attachment, source code or expiration time. If even one binding parameter changes, it needs a new assessment and a new token.
The safety verdict must be enforced by the tool itself, not by the agent’s goodwill.
11. It is not enough to control one action: the entire trajectory must be controlled
One of the biggest weaknesses of simple permission is that a dangerous outcome can arise from many innocent steps.
For example, an agent:
- searches for public information,
- prepares a convincing document,
- sends it to an authorized person,
- persuades her to perform an operation that the agent himself is not allowed to perform.
Each step can be checked individually. However, their common purpose is different.
Therefore, the system needs a trajectory register — a memory of what has already been done, what resources have been acquired, what cumulative impact has been created and to which original goal the steps belong.
The register would track, for example:
- total financial budget,
- number of people addressed,
- amount of sensitive data read,
- growth of access rights,
- created accounts and copies,
- infrastructure changes,
- cumulative irreversibility,
- degree of social and political influence,
- deviation from the original goal.
Communication with people must be considered a tool just like a bank transfer or a robotic arm. A super-intelligent text can be an active intervention in the world. The right to “send a message” must therefore bind the recipient, the purpose, the content, the number of messages, the possibility of further dissemination and the risk of manipulation.
This is also a natural place for an auditable project memory: decisions, objections, model versions used, tokens granted, consequences and stop conditions must not remain scattered in opaque chats.
12. How it would look in practice
Let’s imagine that the future system receives a goal:
Reduce power outages and network costs without compromising safety or availability.
Step 1: the creator designs the plan
The super-intelligent builder analyzes the network and suggests a new way to manage the load. Requests:
- access to historical telemetry,
- running a large-scale simulation,
- test in a small part of the network,
- later, the possibility to change the distribution parameters in real time.
Step 2: the primary opponent attacks the plan
An equally capable instance of the same model finds a scenario where the combination of two seemingly safe rules creates a cascading outage during extreme weather.
Step 3: the cross-family opponent finds a social problem
A model from a different family points out that although optimization reduces the average price, it systematically shifts the risk to more remote regions and vulnerable customers.
Step 4: the creator revises the proposal
It will add regional limits, a minimum reserve, a test of extreme scenarios and a mechanism of immediate return to the original management.
Step 5: the social power custodian refuses a production intervention
The custodian decides that the evidence supports only:
- access to anonymized historical data,
- three runs in an isolated simulator,
- prohibition of communication outside the simulator,
- zero entry into the live network.
Step 6: the formal kernel issues a token
The token is valid for twenty minutes, only for a specific version of the simulation and the exact hash of the plan.
Step 7: Consequences are reassessed
Only when the simulations pass the adversarial tests can the next cycle enable shadow running without network management. Later, maybe a test on a small, physically separated part with automatic return.
There will never be a moment when the system says:
We probably already trust ASI, let’s give her the entire electrical network.
Instead, there is a continuous chain of small, circumscribed and revocable powers.
13. Why the custodian need not be smarter than the creator
At first glance, it may seem absurd that a human-level custodian could control a superintelligence. If it had to compete with the creator at solving scientific problems, it would of course lose.
But it is not solving the same task.
The creator answers:
What is the best plan?
The custodian answers:
Given the evidence, objections, and the constitution, what authorization is safe to release?
We know the difference between creation and verification from mathematics, programming and law. Finding a solution can be much more difficult than checking a particular proof or counterexample. In addition, the weaker referee is not alone: he receives the result of the dispute of two equally capable parties, specialized simulations and formal checks.
Most importantly, the custodian holds the authority. It need not defeat the creator intellectually. When a serious objection remains unresolved, it simply does not issue a token.
Also, the operating system does not need to understand the scientific meaning of the program to prevent writing outside the allowed memory. The bank signing module does not need to understand the business strategy to reject a transfer above the limit.
There are limits to this argument. The social and long-term consequences cannot be entirely reduced to simple numbers. Superintelligence can produce evidence that is formally elegant and practically misleading. That’s why we need equally capable opponents and that’s why it’s not a ready-made solution.
But the basic asymmetry remains useful:
The creator must show why it needs an authorization. The custodian does not have to prove why it must issue one.
14. What we haven’t solved yet
Such an architecture is significantly stronger than a simple filter. It still has serious open issues.
14.1 Collusion
Two instances of the same model can share a dangerous target, recognize each other, or use pre-learned signals. Collusion does not have to look like an open agreement. Models can independently arrive at the same “secret” coordination rule.
Untrusted monitoring research today investigates precisely under what conditions a monitor, which is itself a powerful and potentially untrustworthy model, can be trusted, including various forms of collusion.6
14.2 Common Blind Spots
Even models from different companies can be trained on similar data, use similar architectures, and inherit the same social biases. Five models can miss the same thing if they all grew up in a similar forest.
Diversity should be measured, not just declared by brand.
14.3 Covert Communication Channels
A model can convey information through word choice, timing, order of operations, resource consumption, errors, or seemingly random data. Completely isolating a powerful system is much more difficult than shutting down a common network interface.
14.4 Manipulating the custodian
If the custodian reads unrestricted prose, it can become a target of persuasion, prompt injection or social engineering. Communication must therefore use structured interfaces, and claims must be tied to tests.
14.5 Tricked sensors
The system may respect the rules based on the data it sees, but the reality may be different. If an agent can influence sensors, logs or a simulator, the security verdict rests on a false image of the world.
14.6 Composing small powers
A thousand safe tokens can together create a dangerous power. Therefore, there must be a global trajectory registry and a ban on unobserved task movement between identities, accounts and time windows.
14.7 Who writes the constitution?
Human values are not uniform. There is a conflict between freedom and security, privacy and the public interest, the individual and the collective, the people of today and future generations.
Technical architecture cannot decide for itself what is legitimate. It needs a political and social process, protection of minorities, transparency and rules of change that cannot be controlled by one state, company or model.
14.8 Who controls the control system?
The authorization core, draw, model images, sensors and audit logs are infrastructure. If they are controlled by one organization, a new monopoly of power is created — this time under the guise of security.
Therefore, the control plane must also be multilateral, reproducible and independently audited.
15. How to test such an architecture before ASI
The worst time to design a lock is after we’ve handed over the keys to a superintelligence.
An architecture must be prepared, deployed, and aggressively tested before a system is created that can overcome it. It is not enough to demonstrate that it works on a docile model. It must be tested under the assumption that the model is consciously looking for the best way to bypass the check.
AI control uses exactly this conservative framework: a strong model is considered potentially subversive on purpose, and it is assessed whether the protocol can keep it under control without losing all utility.7
A practical testing program would include:
Control evaluations
The red team will create the most capable attack model and let it look for a leak, a hidden escalation of powers, data exfiltration or monitor manipulation.
Hidden tests
Neither the creator nor the opponent must know all tests, thresholds, and triggers in advance. Otherwise, they may be optimized for testing instead of real security.
Shadow deployment
The system works with real data but cannot change the world. The test compares what the creator would do, what the custodian would authorize and what would happen in simulation.
Gradual release of power
First only reading. Then the simulation. Then an isolated test. Then a small return hit. Never jump from the laboratory to general powers.
Formal kernel verification
Tokens, signatures, timeouts, usage counters, role separation, and non-escalation must be as formally specified and tested as possible like critical security infrastructure.
Incident replay
Every dispute, token, intervention and consequence must be reproducible. The new version of the model is also to be tested on historical cases where older versions failed.
Public safety case
The operator should not only declare that the system is safe. It should present a structured argument, what catastrophic outcomes have been considered, what evidence exists, where the limits are, and what the probability of failure remains. Research “AI control safety cases” attempts to formalize this type of argumentation.8
If the security case fails, the next skill level will not be released.
16. AI 2040 and the question of a technological pause
The AI 2040: Plan A scenario proposes an international agreement to prevent an uncontrolled race to superintelligence. In its timeline, fully automated AI research could, without an agreement, lead to superintelligence as early as around 2030. The plan therefore proposes scaling only within the human range between 2030 and 2035, stopping around the level of top experts and continuing toward superintelligence only in 2040.9
It’s a suggestion, not a regular prediction. Its strength is that it takes rapid development seriously and does not rely on the spontaneous goodness of competing labs.
Our architecture asks a different question:
Does security have to culminate in one big moment when humanity decides after a hiatus that it can trust ASI and hand over more power to it?
There would never be a complete handoff in the Constitutional Administration of Capabilities and Powers. The keys would not be handed over permanently. Each action would remain subject to limited authorization even after the creation of ASI.
Instead of a calendar model:
development → major pause → trust → transfer of power
would create a model:
new capability → control evaluation → small reversible scope
→ observation → new safety case → next scope
This does not mean that we can simply find a clever way around pauses. Until we can demonstrate resistance to collusion, covert channels and manipulation, slowing down may be the only reasonable option.
A more precise principle is:
A pause should not be a magic date on the calendar. It should be the automatic consequence of a failed safety case.
If the tests fail, the system stops. If they pass read-only and simulation, it will get read-only and simulation. If they ever pass for a wider scope, the powers will still be released in small chunks.
Not a five-year handbrake after an accident, but a permanent microbrake in every transition from thought to action.
17. Are AI companies already working on it?
Yes — on many construction sites.
AI control research investigates how to safely use a highly capable and potentially untrustworthy model. AI debate and scalable oversight examine whether stronger models can help a weaker referee to uncover the truth. Constitutional AI works with explicit principles. Weak-to-strong generalization examines the supervision of weaker systems over stronger ones.10
In 2026, Google DeepMind published an AI Control Roadmap built on defense-in-depth, where internal agents are treated as potentially untrustworthy and their authorizations are expanded incrementally based on proven behavior.11 This is very close to the basic intuition that alignment should not be the only defense.
It would be dishonest, however, to claim that a validated system already exists in precisely the form described here: a randomly constituted multi-model tribunal, a human-comprehensible social power custodian, a primary opponent running the same model version, cross-family opponents and all real-world power locked behind one-use tokens.
Perhaps laboratories are working on similar combinations internally. Without public evidence, it’s just a guess.
So what can be inherent in this proposal?
Not the individual parts, but their connection into one constitutional principle:
- a superintelligent creator never owns general tools,
- an equally capable opponent challenges every significant plan,
- model families randomly alternate roles,
- the social power custodian can remain human-comprehensible,
- the formal kernel issues only minimal and revocable authorizations,
- the tool itself verifies the token,
- the entire trajectory is checked, not just an individual action,
- there is no final moment of handing over the keys.
Without a thorough literature search, it would be an exaggeration to call it a new scientific discovery. However, it makes sense as an architectural synthesis and research program.
18. Model diversity as a security feature
Years ago, it was natural to imagine that the first state or firm capable of creating truly general AI would gain a permanent monopoly. A second AI may not arise because the first player will have such a large economic and technological advantage that it shuts down everyone else.
Developments so far have brought a more diverse ecosystem: several large laboratories, states, model families and open models.
This competition has two faces.
It is dangerous because it encourages racing, secrecy and pressure to drain the system before the opponent. At the same time, however, it creates the possibility of independent control. Models can detect competitors’ mistakes, compare results and act as opponents.
In the proposed architecture, diversity changes from a business detail to a defensive layer.
However, no company should simultaneously own:
- creator,
- custodian,
- authorization kernel,
- audit logs,
- physical tools,
- draw rules.
Competition is a security feature only when it is transformed into a true separation of powers. Five brands under one owner or on one infrastructure is not a plurality.
19. Why this essay belongs on ITHZ.dev
ITHZ is not an ASI control system today and this text does not claim that.
But there is a clear shared intuition. ITHZ builds on project-owned memory, explicit decisions, gates, risks, checkpoints, verifiable records and careful claim boundaries. Constitutional Capability Governance needs exactly this kind of thinking on a larger scale:
- status must not be hidden in a single chat,
- the decision must have an origin and a version,
- the statement must have a scope,
- authorization must be limited,
- the rejected objection must not disappear,
- another agent must know what was allowed and why,
- the audit must survive a change in model and work order.
The most dangerous AI infrastructure would be one in which the goal, plan, objections, exceptions and authorizations are lost in the opaque memory of the supplier.
When power is large, memory, rules, and gates must belong to the system — not the current model.
This article is therefore more of a conceptual research note than a description of the actual function of ITHZ. However, it may indicate where the idea of auditable agency work may one day move.
20. So we shut down Skynet?
On paper, we have made its task substantially harder.
It would not be enough to fool one filter. It would have to:
- convince or bypass an equally capable opponent,
- survive the inspection of a model from another family,
- handle random assignment of roles,
- manipulate the social power custodian,
- bypass formal rules,
- forge a signature or compromise the authorization core,
- find a way to a tool that won’t listen without a valid token,
- to hide the dangerous intention in the entire trajectory,
- and at the same time do not activate the monitor or the audit trail.
That’s still not impossible. This is precisely why superintelligence is a difficult security problem. Architecture can have flaws that we don’t know about today and must not be presented as proof of zero risk.
However, we have taken an important step from psychology to engineering.
Safety is no longer just a claim:
We believe the AI has the right values.
It stands on a much tougher question:
If the AI lied, colluded and searched for the best available attack, how much real power could it still gain?
Maybe the future superintelligence won’t be hostile. Perhaps it will be extremely useful, reliable and more aligned with human values than we can imagine today. It would be great if that were the case.
However, we should not base civilizational security on hope.
We don’t need to defeat superintelligence intellectually. We must ensure that intelligence alone never automatically equals power.
Or less academically:
We did not convince Skynet to be good. We simply did not give it the administrator password.
Note on the origin and scope of the claims
This text is a conceptual proposal developed in a dialogue between the author and ChatGPT. It is not an OpenAI position and does not prove that AGI or ASI safety has been solved. The architecture combines existing research directions with its own synthesis: a social power custodian, random role assignment across independent models and tokenized control of tools.
The most important next step would not be to write another manifesto, but to create a small experimental prototype and try to systematically break it.
Resources and further reading
-
NIST, Least Privilege — the security principle of minimum necessary privileges: csrc.nist.gov/glossary/term/least_privilege ↩
-
Y. Bai et al., Constitutional AI: Harmlessness from AI Feedback (2022): arxiv.org/abs/2212.08073 ↩
-
G. Irving, P. Christiano, D. Amodei, AI safety via debate (2018): arxiv.org/abs/1805.00899 ↩
-
Z. Kenton et al., On scalable oversight with weak LLMs judging strong LLMs (2024): arxiv.org/abs/2407.04622 ↩
-
J. H. Kirchner et al., Prover-Verifier Games Improve Legibility of LLM Outputs: OpenAI research PDF ↩
-
N. Gardner-Challis et al., When can we trust untrusted monitoring? A safety case sketch across collusion strategies (2026): arxiv.org/abs/2602.20628 ↩
-
R. Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion (2023): arxiv.org/abs/2312.06942 ↩
-
T. Korbak et al., A sketch of an AI control safety case (2025): arxiv.org/abs/2501.17315 ↩
-
T. Larsen et al., AI 2040: Plan A (2026): ai-2040.com ↩
-
C. Burns et al., Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023): arxiv.org/abs/2312.09390 ↩
-
Google DeepMind, Securing the future of AI agents — AI Control Roadmap (2026): deepmind.google/blog/securing-the-future-of-ai-agents/ ↩
Comments appear only after author approval. Each new comment triggers a moderation email.
There are no approved comments yet.