Knowledge Graph Infographic

NASA's Rules for Code That Can't Fail

Gerard Holzmann's ten rules for safety-critical code, generalized beyond C, extended for distributed systems, and annotated for the specific ways AI code generation violates them.

Author: David Lee

leestack.dev · Mar 18, 2026 (updated Apr 26)

10Rules — small enough to memorize, strict enough to enforce with static analysis

2006Holzmann's original 'Power of Ten' paper at NASA/JPL

32Execution paths in a function with 5 levels of nesting — tests cover 3

2Minimum assertions per function — make assumptions executable

The Rules

Ten Rules for Code That Can't Fail

From C in embedded flight systems to Go/Rust in backend services — each rule maps to a specific distributed-systems failure mode and a specific AI generation weakness.

Keep Control Flow Linear

No deep nesting. No hidden jumps. Five levels of nesting = 32 possible paths. Tests cover three — the 29th fires in production.

Bound Every Loop & Retry

Explicit ceiling in code, not documentation. Unbounded retries cause the most predictable catastrophic outages.

Declare Resource Lifetime

Every resource opened must be closed on every exit path. Don't acquire what you can't account for under partial failure.

One Function, One Job

Small enough to hold in your head, describable in a sentence without "and." A 300-line handler can't be unit tested at any component.

Make Assumptions Executable

Preconditions, postconditions, invariants as code — not comments. A violated assertion at the point of violation is worth 10 hours of post-hoc debugging.

Never Swallow Errors

Every error handled, logged, or propagated. A bare catch{} is active suppression of diagnostic information. DEBUG-logged and discarded = months of corrupted state.

Minimize State Scope

Data as close to use as possible. Shared mutable state is the primary source of race conditions and test pollution. Parameter passing over globals.

Visible Side Effects

I/O, mutations, network calls obvious at the call site. formatResponse() that writes to an audit log is a correctness trap. Pure transforms in one layer, writes in another.

Limit Indirection Layers

Four middleware layers with their own error handling — an error in layer 3 is swallowed by layer 2, surfaced as generic 500. Test full stack error composition.

Zero Warnings, Always

Warnings are future bugs the toolchain has already found. Linters and static analyzers as hard CI gates. Every warning suppressed is a bet on behalf of on-call.

AI Weaknesses

How AI Systematically Violates Each Rule

LLMs optimize for happy-path task completion — they are systematically bad at bounded behavior, explicit error handling, and legible control flow.

Deep Nesting

AI generates deeply nested conditionals because training rewards task completion over structural clarity. Count nesting levels post-generation — more than two deep = refactoring task.

Unbounded Retries

AI retry logic lacks caps, jitter, and dead-letter handling. Ask explicitly: "What is the maximum? What happens at the limit? Is there jitter?"

Error-Path Resource Leaks

AI closes connections on success, leaves them open on failure. Trace every return err — confirm cleanup on every path.

Skipped Validation

AI skips input validation when surrounding code implies data is already clean. That implication is often wrong. Prompt for explicit preconditions.

Swallowed Errors

AI generates empty catch blocks, `_ =` discards, and DEBUG-level continues. Enforce: every error must be WARN+, raised, or explicitly returned with context.

Missing Linter Config

AI projects almost never include linter config. Set up static analysis as scaffolding before AI writes anything — the tooling catches what review misses.

Case Study

The Retry Storm — Rules 02, 06, 09 Violated

11:47pm — Fraud API Returns 503

Database failover. Payment service's fraud detection API returns 503 for 90 seconds. The retry wrapper with exponential backoff and no ceiling begins retrying.

11:49pm — Goroutine Accumulation

By retry 7, delay is 6.4 seconds. 8,000 goroutines in flight waiting on retry timers. Memory: 400MB → 3.2GB. Payment requests queuing.

11:51pm — OOM Cascade

Pod OOMs. Restarts. Immediately begins processing queued requests. Three instances restart in sequence. Fraud API, now recovered, receives 24,000 requests in 30 seconds — falls over again.

Root Causes

Rule 02: retry cap of 3 with max delay 2s would have failed fast. Rule 06: WARN-level logging would have made the storm visible. Rule 09: transparent middleware hid the missing cap from code review.

Mapping

Single-Process Rules → Distributed Systems

Bounded Behavior (01, 02)

Maps to retry storms, cascading timeouts, work amplification. Bounded loops with jitter are the distributed equivalent of loop ceilings.

Resource Accounting (03)

Connection pool exhaustion, file descriptor limits, goroutine leaks. Finite at service level AND cluster level.

Observable Assumptions (05)

Contract testing, schema validation — Protobuf, JSON Schema, OpenAPI as executable assertions for distributed contracts.

Error Propagation (06)

Structured logging, distributed tracing, error classification. Errors crossing service boundaries must carry origin context.

Side Effect Visibility (08)

Event publication must appear in the handler, not buried in a service layer. Hidden writes trigger chains of downstream consequences.

Indirection Limits (09)

Service mesh sidecars, API gateways, load balancers — each adds latency and failure modes. Know the full request path.

AI Guardrails

Working With AI Tooling

Rule 11: You Are the Engineer of Record

AI-generated code has not been tested by someone who cares whether it works. The model has no incident response rotation. Read every line. Trace every error path. Treat AI output like a PR from a capable contractor who has never been on-call.

Rule 12: Specify Failure Modes First

"Here are the requirements, here are the expected failure modes, here are the edge cases: write tests that cover them first, then implement." This forces the model to reason about failures before writing happy-path code.

"Never commit code you haven't read in full. You are responsible for what ships. The model has no incident response rotation."

— David Lee, leestack.dev

FAQ

Twelve Questions the Graph Answers

Ten rules for safety-critical C code by Gerard Holzmann at NASA/JPL — small enough to memorize, strict enough to enforce with static analysis, targeting failure modes that actually kill software.

Each rule maps to a distributed-systems failure mode: unbounded retry storms, resource leaks under partial failure, silently corrupted state, opaque middleware stacks.

LLMs optimize for happy-path task completion — systematically bad at bounded behavior, explicit error handling, minimal scope, and legible control flow.

Unbounded retry logic without caps or jitter. Looks correct but retries indefinitely — a single loop cascades into a cluster-wide thundering herd.

Explicit maximum count in code. Backoff with ceiling. Exhausted? Dead-letter queue, error with full context. Named constants like maxRetries=5, maxDelay=5s.

Fraud API returns 503 for 90s. 8,000 goroutines in flight. Memory 400MB→3.2GB. Pod OOMs. Three-instance cascade. Preventable with retry cap of 3.

AI has no incident response rotation. Read every line. Trace every error path. Check every resource. Treat AI output like a PR from a contractor who has never been on-call.

Specify failure modes upfront: max retry count, jitter, dead-letter behavior, context cancellation. Write failing tests first, then implement.

At least 2 per function. Preconditions/postconditions/invariants as executable code, not comments. An assertion at violation point = 10 hours of debugging saved.

Limit indirection layers. Know what every layer does and how errors compose. Test the full middleware stack's error behavior, not interceptors in isolation.

Every warning suppressed = a bet that this one won't matter. Static analysis as scaffolding before code exists. Zero warnings as hard CI gate.

Set up static analysis as project scaffolding before any code. golangci-lint with errcheck, govet, staticcheck, gosec. Zero warnings as merge gate. AI's first output will violate — that's the point.

Glossary

Key Concepts

Safety-Critical Code

Software that cannot fail — from Apollo to backend services where failure carries unacceptable cost.

Distributed Systems

Networked services where failure modes include retry storms, cascading timeouts, pool exhaustion.

AI Code Generation

LLM-produced code — optimizes for happy path, systematically weak on error handling and bounded behavior.

Static Analysis

Automated tooling enforcing rules before execution — linters, type checkers, CI analyzers as hard gates.

Retry Storm

Cascading failure where unbounded retries amplify a partial outage into a full cluster-wide collapse.

HowTo

Apply NASA Rules to AI-Generated Code

1. Set Up Static Analysis Before Code

Configure linters as project scaffolding. The AI's first output will have violations — that's the point. Tooling catches what review misses.

2. Specify Failure Modes First

State retry caps, jitter, dead-letter behavior, context cancellation upfront. Write failing tests before implementation.

3. Review Every Resource Path

Trace every function that acquires resources. AI closes on success, leaves open on failure — this pattern is nearly universal.

4. Flatten Nested Control Flow

Count nesting levels. More than two deep in a handler = refactoring task. Extract branches to named functions.

5. Enforce Error Propagation

Grep for empty catch blocks and `_ =` discards. Every error: WARN+, raised, or explicitly returned with context.

6. Separate Pure Transforms from Writes

AI buries writes in utility functions. Extract to named, visible calls. Read path and write path must be structurally distinct.

7. Read Every Line Before Committing

Non-negotiable. AI has no incident rotation. Trace every error path. Verify every bound. If you can't explain what every line does under partial failure, don't ship it.