Evidence boundary: Safety engineering, software assurance, cyber-resiliency, and AI risk-management guidance provide methods for separating hazards, requirements, controls, evidence, and authority. They do not validate this proposed stack for a generation ship. Language models remain nondeterministic and vulnerable to confabulation, injection, poisoning, privacy loss, and automation bias. This lesson is civil-and-defensive and does not provide offensive cyber or weapon-integration guidance. High-consequence cyber, dual-use, governance, and life-safety boundaries require two-person review.
Plain-language summary
A system’s ability to suggest an action is not permission to perform it.
The autonomy authority stack places different kinds of machinery and judgment in layers:
- physical protection and passive safety;
- deterministic protection and verified control;
- bounded automation and planning;
- evidence-linked human support, including LLMs;
- quarantined experiments.
The lower layers protect essential physical limits. Higher layers can improve efficiency, coordination, and learning, but cannot silently weaken lower protections. Authority crosses a layer only through a typed, logged, reviewable gate.
This is not a claim that deterministic software is perfect or humans are always wise. It is a way to keep one plausible failure—especially a fluent generative system—from controlling every observation, decision, actuator, record, and recovery path.
Layer zero: passive and physical protection
The strongest control can be a geometry, material, pressure relief path, mechanical stop, fire boundary, shielding mass, gravity-driven drain, or normally safe valve state. These protections do not need a model to recognize a sentence.
Physical measures still have assumptions and failure modes. A relief valve can corrode. A passive thermal path can be undersized. A mechanical stop can be bypassed during maintenance. Their authority comes from verified physical behavior within a declared environment, not from being “non-digital.”
For each hazard, record:
- protected quantity and safe range;
- physical mechanism and capacity;
- failure and maintenance modes;
- environmental limits;
- inspection and test evidence;
- dependencies shared with active control; and
- conditions under which software may not override it.
AI may help inspect or explain the layer. It should not redefine the physical limit.
Layer one: deterministic protection and control
This layer includes interlocks, independent trips, hard real-time control, voting logic, bounded state machines, and manually accessible local control. “Deterministic” means behavior is constrained enough to analyze and test; it does not mean bug-free.
NASA-STD-8739.8 requires lifecycle software assurance, software safety, objective evidence, and independent verification and validation appropriate to NASA software. A ship would need its own authority and competence, but the evidence discipline remains relevant.
Safety controllers should:
- enforce explicit invariant limits;
- use independently justified sensing for high-consequence conditions;
- fail toward a defined state;
- expose state and reason codes without relying on natural-language interpretation;
- accept only typed, range-checked commands;
- separate configuration change from routine operation;
- preserve a local manual mode; and
- remain restorable from known-good material.
An LLM cannot sit inside the final trip path merely because it performed well on examples. Statistical perception may inform a bounded detector, but an independent layer should contain its mistakes.
Layer two: bounded automation
Schedulers, planners, optimizers, diagnostic systems, robotic sequences, and distributed coordination live here when their state, actions, resources, and failure responses are explicit.
Deep Space 1’s Remote Agent illustrates bounded autonomy: high-level goals, onboard planning and execution, selected subsystems, simulated faults, and a finite flight experiment. Its timing bug is part of the evidence, not an embarrassment to omit.
Bounded autonomy needs an operational contract:
- allowed goals and prohibited outcomes;
- visible system state and uncertainty;
- action and resource limits;
- timing bounds and missed-deadline behavior;
- assumptions about sensors, clocks, communications, and people;
- monitored invariants supplied by lower layers;
- abstention or handoff conditions;
- rollback and recovery;
- test coverage and unresolved anomalies; and
- an accountable owner.
Automation can act without a person approving each step only inside that contract. Changing the contract is a higher-authority decision.
Layer three: evidence-linked advice
Language models can help people query records, draft a plan, translate a procedure, compare hypotheses, summarize an incident, or learn an unfamiliar domain. At this layer, the model proposes; an authorized person and external controls dispose.
A trustworthy interface should show:
- controlled source and revision;
- exact locator for material claims;
- whether text is quotation, structured record, inference, or proposal;
- conflicting evidence and missing links;
- model, prompt, retrieval corpus, tool, and configuration versions;
- the authority under which any tool would act;
- a preview of consequential changes; and
- the human decision and reason.
A generated citation must be resolved against the local archive before display as evidence. A signature verifies bits and a signer under a policy; it does not make the content true, current, lawful, or safe.
Tool calls require particular care. Retrieved text, user messages, or logs can contain prompt injection that attempts to redirect the model. Treat all model-produced commands as untrusted proposals. Enforce identity, scope, schema, range, rate, and safety constraints outside the model. The model cannot expand the operator’s permissions or approve its own action.
Layer four: experiments
New models, prompts, retrieval methods, tools, robot policies, and self-modifying systems begin in quarantine. Experimental systems should use synthetic, historical, or sacrificial targets before any shadow operation near live processes.
An experiment record should state:
- hypothesis and expected Earthside or mission value;
- protected systems and data;
- prohibited actions;
- evaluation cases and possible contamination;
- success, failure, and stop criteria;
- privacy and retention limits;
- independent reviewers;
- rollback and cleanup; and
- the evidence needed to request promotion.
No performance result automatically promotes a system. Moving upward requires a new safety case, change review, representative testing, and explicit authority.
Gates, not vague human oversight
“A human is in the loop” says little. A person may have seconds to act, lack the relevant information, distrust their own judgment after years of automation, or face institutional pressure to accept the model.
For every gate, specify:
- who receives the proposal;
- their competence and conflict status;
- information and time available;
- what independent measurement they can inspect;
- whether they can reject safely;
- whether a second reviewer is required;
- how dissent and appeal work;
- what happens when no qualified person is available; and
- how authority expires after an emergency.
High-consequence medical, reproductive, nuclear, life-support, identity, governance, cyber, and dual-use actions require two qualified people with independent evidence. An AI is neither reviewer.
Evidence and requirements graph
The stack should be represented in an inspectable graph rather than buried in prompts:
- hazard → safety requirement → physical or software control;
- control → implementation → configuration;
- requirement → verification method → result → anomaly;
- model → training and evaluation provenance → validity limits;
- proposed action → authority rule → approvals;
- incident → observation → hypothesis → recovery;
- source → claim → counterevidence → review date.
The graph makes a crucial difference: an LLM can explain why a command is blocked without becoming the blocking mechanism. Reviewers can traverse from a physical invariant to its evidence. Missing or disputed edges remain visible instead of being smoothed into prose.
Failures can cross layers
Layering is not independence if every layer shares one sensor, clock, identity service, compiler, power supply, or administrator. A malicious or accidental update can alter the controller, the twin used to test it, the LLM that explains it, and the log that records it.
Trace common causes. Preserve separately governed sensors and tools. Keep recovery material offline. Use physical observations that do not depend on the same software chain. Rotate people and authority. Test the stack with central identity, network, AI, and one specialist unavailable.
Human skill is part of the architecture
If people never operate a system, their nominal override authority will decay. Training must include:
- local manual operation;
- reading raw instruments;
- diagnosing without model summaries;
- rebuilding from controlled records;
- identifying confident but unsupported advice;
- working across roles and accessibility needs;
- teaching successors; and
- restoring normal authority after emergency operation.
Measure workload and timing. A manual fallback that needs twenty experts within three minutes is not a fallback for a small crew.
Recovery and AI-off operation
Every generative service needs a kill, isolation, and recovery plan that does not depend on asking the same model what to do. Preserve human-readable procedures, structured exports, source material, tools, and known-good configurations.
Exercises should inject:
- a poisoned procedure in retrieval;
- a prompt or tool-injection attempt;
- a stale but signed source;
- a model that omits inconvenient evidence;
- corrupted telemetry;
- a shared runtime failure across nominally diverse models;
- loss of the central policy service; and
- a prolonged AI-off interval.
Pass criteria are safe service, accurate authority, evidence preservation, and recoverability—not merely successful model restart.
Evidence ledger
- L10-02-A — Separating passive safety, verified control, bounded autonomy, AI advice, and experiments is a proposed authority architecture. Basis: proposed. Readiness: early research. Confidence: supported by current safety and assurance methods, unvalidated as an integrated ship design.
- L10-02-B — Bounded autonomy has flown, but its evidence does not transfer to open-ended generative authority. Basis: demonstrated. Readiness: operational for selected tasks; no known path for indefinite legitimate judgment. Confidence: strong.
- L10-02-C — Tool-using LLM output should be treated as an untrusted proposal constrained by external identity, schema, policy, and safety mechanisms. Basis: normative synthesis. Readiness: early research for life-safety use. Confidence: strong.
- L10-02-D — A nominal human loop is insufficient without time, competence, information, refusal power, and practiced skill. Basis: normative human-factors requirement. Readiness: major scale-up in representative habitat tests. Confidence: supported.
- L10-02-E — Shared sensors, runtimes, toolchains, identity, or authority can correlate failures across layers. Basis: modeled and observed system principle. Readiness: operational as analysis, early research for full integration. Confidence: supported.
- L10-02-F — Safe AI-off operation and local recovery are launch gates for any safety-critical function influenced by generative models. Basis: normative. Readiness: proposed. Confidence: supported.
Linked corpus claims: claim-11-01, claim-11-04, claim-11-06, claim-11-07, claim-11-10, claim-12-01, and claim-12-03. See the claim registry for each record's current evidence grade and independent-review state.
Assumptions and limits
- The five layers are a design heuristic, not a universal certification scheme.
- Deterministic software and physical protection still require verification, maintenance, and independent failure analysis.
- “Manual” counts only when access, instrumentation, staffing, timing, and competence are demonstrated.
- LLMs remain offline-capable, evidence-linked, logged, non-authoritative, removable, and excluded from direct life-safety actuation.
- Cryptographic provenance does not establish truth, currency, legitimacy, or safety.
- Civil rights, privacy, appeal, and expiry of emergency powers constrain technical authority.
- Offensive cyber operations and autonomous weapons are excluded.
What would change this conclusion?
A better independently reviewed architecture could replace these layers if it demonstrated equal or stronger physical safety, explicit authority, failure independence, due process, and recovery. Confidence would rise after representative habitats repeatedly survive poisoned inputs, compromised models, broken identity, unavailable specialists, and prolonged AI-off operation while maintaining safe service. Any design that cannot bound tool authority, preserve lower protection when higher layers fail, or sustain a real manual path should trigger redesign or a wait/do-not-launch decision.
Sources and locators
- S01 — NASA-STD-8739.8B, Software Assurance and Software Safety Standard (opens external site in a new tab). Locator: scope, software assurance, software safety, security, objective evidence, independence, IV&V, lifecycle, and requirements mapping; September 2022; accessed 2026-07-25.
- S02 — NASA-STD-7009B, Standard for Models and Simulations (opens external site in a new tab). Locator: sections 1–5 on intended use, requirements, lifecycle, credibility products, validation, verification, uncertainty, configuration, and acceptance; March 2024; accessed 2026-07-25.
- S03 — JPL, Deep Space 1 Autonomous Remote Agent (opens external site in a new tab). Locator: bounded onboard planning, execution, selected subsystem control, simulated-fault response, timing bug, and ground-team role; accessed 2026-07-25.
- S04 — NIST AI 100-1, Artificial Intelligence Risk Management Framework 1.0 (opens external site in a new tab). Locator: trustworthy-AI characteristics and Govern, Map, Measure, Manage functions; January 2023; accessed 2026-07-25.
- S05 — NIST AI 600-1, Generative Artificial Intelligence Profile (opens external site in a new tab). Locator: confabulation, information integrity, privacy, human-AI configuration, value-chain, and governance actions; July 2024; accessed 2026-07-25.
- S06 — NIST AI 100-2 E2025, Adversarial Machine Learning (opens external site in a new tab). Locator: poisoning, evasion, privacy, misuse, attacker capabilities, lifecycle stages, and mitigation limitations; March 2025; accessed 2026-07-25.
- S07 — NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models (opens external site in a new tab). Locator: AI-specific additions to SSDF practices for organization, protected artifacts, secure production, and vulnerability response; July 2024; accessed 2026-07-25.
- S08 — NASA Systems Engineering Handbook, Appendix D and Appendix I (opens external site in a new tab). Locator: requirements verification matrix, bidirectional traceability, verification methods, validation planning, and V&V plan structure; accessed 2026-07-25.
Editorial record
- Prepared by: GShips Project
- Last edited: 2026-07-25
- Status: Substantive editorial draft
- Independent domain review: Pending; cyber, dual-use, governance, and life-safety claims require explicit two-person review before publication
- Required review: safety engineering, control systems, AI evaluation, software assurance, cybersecurity, human factors, and governance
- Reviewer: No independent reviewer assigned
- Conflicts: Maintainer intends to explore a commercial venture based on some GShips work; no entity, funding, customer, sponsor, or partner relationship currently exists
- Relationship boundary: Independent educational synthesis; citations do not imply affiliation, endorsement, partnership, or adoption by NASA, JPL, NIST, CCSDS, or any named organization
- Scope boundary: Civil and defensive uses only; offensive cyber operations, autonomous weapons, weapon integration, and actionable exploitation instructions are excluded
- Corrections: Suggest a correction