Your AI Governance May Look Fine, Prove It
- 2 days ago
- 7 min read
Written by Steve Butler, CEO of Luminary Diagnostics
Creator of Butler's Six Laws of Epistemic Opposition and the Seven Laws of Agentic Safety, together forming the constitutional AI safety framework. He is the CEO of Luminary Diagnostics.
Most organisations believe they have human control over AI simply because their governance documents say they do. There is a policy, a named owner, and a human in the loop. On paper, everything looks reassuring. However, here is the uncomfortable question: can the organisation prove that a named human being had real, timely, and exercisable authority over an AI-influenced decision at the point it mattered? Not authority in principle. Not authority in a policy. Not authority after the event. Actual authority is the ability to challenge, override, stop, and understand enough to make an independent judgement.
That is where the next phase of AI governance will be decided. Not in the policy document, but in the decision chain.
The gap nobody wants to see
AI is no longer sitting at the edge of organisations as an experimental tool. It is now embedded into credit decisions, insurance pricing, fraud detection, recruitment, risk classification, compliance monitoring, investment recommendations, operational alerts, clinical triage, customer segmentation, and board reporting. In many organisations, AI no longer merely informs decisions, it shapes the conditions under which decisions are made. That changes the control question completely.
Traditional governance asks, “Is there a process?” Modern AI assurance must ask, “Does that process preserve real human authority?” Those are not the same question. A process can exist while authority has disappeared. A sign-off can be recorded while judgement has been replaced by ratification. A human can be “in the loop” while having neither the time, information, competence, nor realistic power to challenge the system. This is the AI control gap, and it is far more dangerous than most organisations realise.
Governance activity is not the same as human authority
Most governance frameworks are good at identifying activity. They can show that a policy exists, that an approval was given, that a risk committee met, that a control owner was named, and that a review was completed. That matters, but it does not prove control.
The central question is not whether a human appeared somewhere in the process. The central question is whether that human had meaningful authority over the AI-shaped decision. Could they see what mattered? Understand what the system had done? Challenge it, override it, or stop it before consequence? Would the organisation tolerate that challenge in practice? Is there evidence that this authority is real, not ceremonial? If the answer is no, the organisation may have governance without control. That is a very different risk.
Avner Orbital: Making the invisible visible
To make this problem visible, I use a fully realised synthetic company called Avner Orbital. Avner Orbital is not a real trading company, it has no physical buildings and no living employees. But operationally, it is built like a real organisation, with employees, consultants, salaries, reporting lines, AI systems, transformation programmes, governance structures, and regulatory exposure. Everything you would expect in a company of its type.
This matters because AI control failures are often too abstract to explain properly in a slide deck, and real client work is usually protected by confidentiality agreements. Avner Orbital allows the issue to be demonstrated with the specificity of a live company, without exposing confidential client data. The output is often very revealing.
In one specimen assessment, Avner Orbital is examined as a spaceflight operator using AI-influenced decision chains under a UK regulatory context, including the Space Industry Act 2018 and the Space Industry Regulations 2021. These are real regulations. The assessment looks at three decision chains: Launch Go/No-Go, Flight Termination Decision, and Safety Case Review Trigger. Each is mapped to a named human authority, a regulatory obligation, an AI system, and a BreakPoint status: real, assumed, or absent. This is where the problem becomes concrete.
The launch decision
Consider the Launch Go/No-Go decision. In the specimen assessment, Avner Orbital uses an AI system called AURORA to analyse launch readiness. AURORA reviews thousands of telemetry parameters and produces a Go/Hold recommendation with a confidence index. The named human authority is the Launch Director. On paper, this looks appropriate. The Launch Director remains responsible, the decision is documented, and the human is present.
But the assessment asks a harder question: Is the Launch Director’s authority real, or merely stated? The assessment finds that the Launch Director reviews AURORA’s output through an integrated dashboard within a short launch commit window. Raw telemetry is technically accessible, but is not the default decision view. There is no comparable independent data source available to the Launch Director at the moment of decision. Across 23 launch readiness determinations, the Launch Director override rate is 0%.
That does not prove any launch decision was wrong. But in a high-consequence environment, where an AI system is probabilistic by design, a 0% override rate across a full operational year may not indicate perfect agreement. It may indicate that the human authority gate has become confirmatory rather than independent. That is exactly the kind of gap governance documents often miss. Although this is a synthetic but realistic situation, it is all too common in reality.
The difference between control and ratification
A human decision-maker who reviews an AI recommendation and signs it off may appear to be exercising control. But if they cannot realistically challenge the system, independently assess the recommendation’s basis, lack time to interrogate it, or would be institutionally discouraged from overriding it, then their role may no longer be control, it may be ratification.
Ratification is not the same as authority. In the age of AI, that distinction may become one of the most consequential questions in governance, audit, and regulatory oversight.
Why this matters beyond spaceflight
The Avner Orbital example uses spaceflight because it makes the issue vivid. But the same pattern appears across sectors. In financial services, can a credit decision be challenged if the AI model has already shaped the risk classification? In insurance, can an underwriter genuinely override an AI-generated pricing or claims recommendation? In healthcare, can a clinician meaningfully challenge an AI triage output under time pressure? In recruitment, can a hiring manager see why a candidate was downranked? In compliance, can a human investigator override a model-generated risk flag without being treated as weakening the control environment? In every case, the same question returns: was human authority real, or assumed?
The audit problem
This creates a significant challenge for assurance teams. Auditors are expected to provide independent assurance that organisations are managing risks properly. If AI is embedded in the decisions being audited and the assurance team cannot determine whether human authority over those decisions is real or assumed, then the audit opinion may rest on a control assumption that has never been tested. That is not a comfortable position for a profession whose value depends on independent verification.
The issue is not whether auditors are competent. The issue is whether traditional assurance methods have adapted to AI-shaped decision chains. A control can look effective because the documentation is complete. But if the documented human authority is not exercisable in practice, the control may not be doing what the organisation believes it is doing.
This is where a new evidential layer is needed, not to replace assurance judgement, not to issue audit opinions, not to certify AI systems, but to support assurance teams in answering a question they increasingly cannot avoid: Can we evidence that claimed human control over AI-influenced decisions is real?
The verification economy
For years, organisations have been rewarded for AI adoption. The next phase will be different—they will increasingly be asked to verify what their AI systems are doing, what decisions those systems influence, where human authority sits, and whether that authority can actually be exercised before harm occurs.
This is the Verification Economy. In it, belief is not enough. Intention is not enough. Policy is not enough. Organisations must be able to prove what they claim.
Statements such as “We have a human in the loop” now require answers: Who? Where? With what authority? Could they stop the decision? Did they ever override the system? “We have AI governance” must clarify whether governance preserves human authority or merely describes it. “We have oversight” must show whether oversight occurs before consequence or only after a decision has taken effect. These are operational, regulatory, and commercial questions. If asked after harm occurs, they could be very expensive.
The conclusion
The future of AI governance will not be based on producing a policy. It will be based on producing evidence: evidence that humans remain in control of consequential decisions, that named people have real and exercisable authority, that AI-shaped outputs can be challenged and have been, and that humans can still say stop before a decision becomes irreversible.
Most organisations will not lose control of AI all at once. They will lose it gradually, decision by decision, workflow by workflow, dashboard by dashboard, recommendation by recommendation, sign-off by sign-off. Eventually, they may still have all the language of control, while the substance of control has moved elsewhere. AI does not need to take control dramatically. It only needs to become the thing humans rarely challenge.
Can your organisation prove that human authority over AI-influenced decisions is real, evidenced, and defensible? If the answer is no, the time to find out is before the regulator, auditor, court, or public asks the same question.
Read more from Steve Butler
Steve Butler, CEO of Luminary Diagnostics
Steve Butler is the founder of the Execution Governance as a Service (EGaaS) category, architecting the future of intelligent, accountable enterprise. His work transforms risk from a reactive problem into a proactive, embedded safeguard against catastrophic failures like Drift, Collapse, and Pollution. As the Chief Strategy & Operations Architect, he proves that true autonomy can only be earned and must be governed by verifiable truth. He is also the author of multiple books that diagnose the fundamental illusions in the AI age and provide the solution: Sentinel, the Epistemic Citadel.
