fortress-development

name	fortress-development
description	Repository-wide engineering policy and project context for Fortress Rollback. Use when implementing, diagnosing, reviewing, testing, documenting, or releasing changes in this repository.

Fortress Rollback -- LLM Development Guide

Canonical source of truth for project context. All other LLM instruction files point here.

TL;DR

Fortress Rollback is a correctness-first fork of GGRS, written in 100% safe Rust. Peer-to-peer rollback networking for deterministic multiplayer games.

Five Pillars

1. Zero-panic production code -- All errors returned as Result, never panic
2. >90% test coverage -- All code must be thoroughly tested
3. Formal verification -- TLA+, Z3, and Kani for critical components
4. Enhanced usability -- Intuitive, type-safe, hard-to-misuse APIs
5. Code clarity -- Readable, maintainable, well-documented

Quick Commands

cargo fmt && cargo clippy --workspace --all-targets --features tokio,json && cargo nextest run --no-capture  # Full local validation
cargo c && cargo t                        # Aliases from .cargo/config.toml
typos                                     # Spell check (CI enforced)
cargo test --features z3-verification -- --nocapture  # Z3 proofs (slow)

Test output rule: Always use --no-capture (nextest) / -- --nocapture (cargo test) so output is visible on failure, and NEVER pipe test output through tail/head -- redirect to a temp file instead:

cargo nextest run --no-capture > /tmp/test-results.txt 2>&1  # Then read the file
for i in $(seq 1 10); do cargo nextest run --no-capture >> /tmp/flaky-check.txt 2>&1; done  # Repeated runs

CLI Tools (Dev Container)

Use	Instead of	Key flags
rg	grep	-l list files, -C 3 context, --type rust
fd	find	-e toml extension, --type d dirs
bat --paging=never	cat	-n line numbers, -r 10:20 range
eza	ls	-la, --tree, --git
sd	sed	sd 'old' 'new' file, -F literal
dust	du	-d 2 depth
tokei	wc -l	Code stats by language
hyperfine	time	Statistical benchmarking

Rules: always bat --paging=never (bare bat blocks); never redirect to /dev/null; use rg --no-ignore for gitignored files.

Non-Negotiable Requirements

• 100% safe Rust -- #![forbid(unsafe_code)]
• ZERO-PANIC POLICY -- Production code must NEVER panic; all errors as Result
• All clippy lints pass -- clippy::all, clippy::pedantic, clippy::nursery
• No broken doc links -- All intra-doc links must resolve
• Accurate panic docs -- Rustdoc # Panics lists all reachable panic conditions; boundary panic docs should have matching tests when practical
• Public items documented -- Rustdoc with examples
• Overflow checks in release -- Integer overflow caught at runtime
• Deterministic behavior -- Same inputs must always produce same outputs
• Bounded allocation -- Never trust a length from the wire; validate config at the boundary. Bound total bytes, element counts, raw receive attempts, and socket receive-poll batches before allocation (for example, decoded bytes split into many Vec buffers need a frame/count cap too; built-in sockets cap receive polls at 256 raw attempts and 256 decoded messages). Dynamically-sized allocs in src/ need an // alloc-bound: justification (enforced by check-unbounded-alloc); see defensive-programming.md
• Network input width -- Config::Input for network sessions must serialize to at least one byte and should serialize to the same byte length for every value. Prefer fixed-width structs of integers/bools; variable-length enums, strings, vectors, maps, and similar payloads can collapse delta-compressed frame streams and must be rejected or encoded through a fixed-width wrapper.

// FORBIDDEN in production:  value.unwrap(), .expect(), array[i], panic!(), todo!()
// REQUIRED:  value.ok_or(FortressError::MissingValue)?, array.get(i).ok_or(...)?,
//            if !valid { return Err(FortressError::InvalidState); }

Zero-panic key principles: never swallow errors (use ?), validate all inputs, prefer .get() over indexing, exhaustive match (no _ => on enums), enums over booleans, doc examples must use ? and Result. See defensive-programming.md for the complete guide.

Code Design Principles

Follow SOLID, DRY, and Clean Architecture. Rely on descriptive names; comment only the "why." Prefer zero-cost abstractions (generics/traits over dynamic dispatch), value types, and minimal allocations.

Design patterns used: Builder (SessionBuilder), State Machine (protocol/connection states), Strategy (input prediction), Iterator (combinators over manual loops).

Before writing new code, search for similar existing patterns. Extract shared utilities; avoid copy-paste.

Skills Reference

Load focused skills from the sibling directories under .agents/skills/ when the task needs deeper guidance. Their discovery metadata covers defensive programming, testing, formal verification, rollback netcode, determinism, performance, WASM, CI/CD, API design, documentation, and release workflows.

Project Architecture

Repository Structure

src/
  lib.rs                           # Public API entry point
  error.rs                         # FortressError types
  frame_info.rs / hash.rs / rle.rs / rng.rs  # Core utilities
  time_sync.rs / sync.rs / checksum.rs / telemetry.rs
  input_queue/
    mod.rs                         # Input buffering
    prediction.rs                  # Input prediction strategies
  sync_layer/
    mod.rs                         # Core synchronization (SyncLayer)
    game_state_cell.rs             # Thread-safe game state
    saved_states.rs                # Circular buffer for rollback
  network/
    compression.rs / messages.rs / network_stats.rs
    chaos_socket.rs / udp_socket.rs / codec.rs / tokio_socket.rs
    protocol/
      mod.rs / event.rs / input_bytes.rs / state.rs
  sessions/
    builder.rs                     # SessionBuilder pattern
    p2p_session.rs                 # P2P gameplay
    p2p_spectator_session.rs       # Spectator mode
    sync_test_session.rs           # Determinism testing
    config.rs / player_registry.rs / sync_health.rs

Session Types

• P2PSession -- Standard peer-to-peer gameplay
• SpectatorSession -- Observe but don't participate
• SyncTestSession -- Verify determinism by running simulation twice

Critical Determinism Rules

1. No HashMap iteration -- Use BTreeMap or sort before iterating
2. Control floating-point -- Use libm feature or fixed-point math
3. Seeded RNG only -- rand_pcg or rand_chacha with shared seed
4. Frame counters, not time -- Never use Instant::now() in simulation
5. Sort ECS queries -- Bevy queries are non-deterministic; sort by stable ID
6. Pin toolchain -- Use rust-toolchain.toml for reproducible builds
7. Audit features -- Check for ahash, const-random leaks with cargo tree -f "{p} {f}"

Kani Essentials

The #1 cause of Kani CI failures: All loops with symbolic bounds require #[kani::unwind(N)] where N = max_iterations + 1. CI uses --default-unwind 8 via --quick mode.

The #2 cause: Proofs that assert the wrong thing (e.g., wrong enum variant).

The #3 cause: format!() inside macros (e.g., report_violation!) creating explosive CBMC state space. The report_violation! macro handles cfg(kani) internally (borrows arguments to suppress unused warnings without format!()) while preserving public format! token syntax such as named arguments. No additional gating needed when calling it. See kani.md for details.

cargo kani --harness proof_function_name    # Run specific proof
./scripts/verification/verify-kani.sh --tier 1 --quick   # Fast proofs (~15 min)
./scripts/verification/verify-kani.sh --list             # List all proofs and tiers
./scripts/verification/check-kani-coverage.sh            # Validate proof registration

New proofs must be registered in scripts/verification/verify-kani.sh:

• Tier 1: Fast (<30s) -- simple property checks
• Tier 2: Medium (30s-2min) -- moderate complexity
• Tier 3: Slow (>2min) -- complex state verification

Pre-commit validates registration only, NOT that proofs pass. Run affected proofs locally before committing.

TLA+ Tools Version

The repository-pinned TLA+ tools version lives in .tla-tools-version. scripts/verification/verify-tla.sh, the devcontainer image/setup hook, and CI cache keys must read that file instead of hardcoding release URLs or duplicate version values.

The .tla-tools/ directory is a runtime cache (tla2tools.jar plus a generated .version sentinel) populated by those scripts. It is .gitignored and must never be committed — the check-no-tla-tools-cache pre-commit hook (scripts/hooks/check-no-tla-tools-cache.py) enforces this. Committing the sentinel desynchronizes it from the actual cached jar and silently disables the staleness guard.

Safety CI Checks

Check	Purpose
Cargo Careful	Extra runtime safety checks (nightly)
Overflow Checks	Release builds with -C overflow-checks=on
Debug Assertions	Release builds with -C debug-assertions=on
Panic Patterns	Counts unwrap, expect, panic!, todo!
Strict Clippy	Nursery lints enabled
Documentation	Doc build with -D warnings

Also: ci-rust.yml (Miri), ci-security.yml (cargo-geiger, cargo-deny).

Dependabot auto-merge policy: this repository is squash-only and the auto-merge job MUST wait for every non-self CI gate on the PR head SHA to reach an explicitly accepted state (success, skipped, neutral) before enabling merge. Anything else -- including failure, timed_out, cancelled, action_required, startup_failure, stale, missing/null conclusions, or future GitHub-added states -- refuses the merge (allow-list semantics). GitHub also requires protected-branch rules (for example, requiring pull requests before merge) on the PR base branch before enablePullRequestAutoMerge can succeed; the script classifies that single case as soft-fail exit code 20, and the workflow downgrades only that code to warning. Use scripts/ci/enable-dependabot-automerge.sh -- never inline gh pr merge in workflows, never re-introduce a "one-shot" / bypass path, never wrap the script in a sub-job-level timeout. Regression-tested in scripts/tests/test_enable_dependabot_automerge.py.

CI fails on: unformatted code, clippy warnings, broken doc links, markdown lint errors, workflow syntax errors, unregistered Kani proofs.

Development Workflow

Before Writing Code

1. Read relevant source files and tests for context
2. Check existing patterns for consistency
3. Consider impact on other components
4. Plan tests first -- define expected behavior

When Fixing Bugs

1. Write a failing test that reproduces the bug
2. Root-cause analysis -- understand why, not just what
3. Fix at the right level (production bug vs test bug)
4. Add regression tests; check for similar issues elsewhere

Asking Clarifying Questions

When clarification is required before proceeding, use the question template to keep questions focused, actionable, and forward-moving.

Design Entrance Gate

Before implementation, run the design entrance checks in dev-pipeline.md to confirm determinism, zero-panic handling, session impact, and broad test coverage.

Review and Hardening Gates

Before opening PRs, run review-readiness.md. For high-risk changes or post-incident hardening, use adversarial-handoff.md with adversarial-review.md.

Test Writing

Use Arrange-Act-Assert pattern. Name tests: what_condition_expected_behavior (e.g., parse_empty_input_returns_none).

#[track_caller]  // Shows actual test location on failure
fn check_parse(input: &str, expected: Option<Ast>) {
    let actual = parse(input).ok();
    assert_eq!(actual, expected, "parse({:?})", input);
}

Consolidate integration tests into a single crate (tests/it/main.rs). Anti-patterns: assert!(result.is_ok()) (use assert_eq!), sleep-based synchronization, testing implementation details, and if let Ok(..) = session.advance_frame() patterns that swallow all frame-advance errors instead of matching only expected errors.

For protocol tests that poll in loops or assert timer effects, always inject TestClock/ProtocolConfig.clock and advance it each poll iteration (for example with POLL_INTERVAL_DETERMINISTIC); interval-gated sends (retries, quality reports, keepalives, pending output) will not fire reliably if wall-clock time does not advance. Multi-process real-UDP child waits must derive from each PeerConfig.timeout_secs plus documented overhead and stay below nextest budgets; scripts/hooks/check-network-timing-invariants.py enforces this.

Changelog Policy

Quick decision: "Does this affect pub items or user-observable behavior?"

• YES -- Add entry (use Breaking: prefix if API signature changed)
• NO (pub(crate), private, tests, CI) -- Skip

Include: new features/APIs, user-visible bug fixes, breaking changes (with migration guidance), performance improvements, dependency updates affecting compatibility.

Exclude: internal refactoring, test improvements, doc-only changes, CI/tooling, lint fixes.

Unreleased code rule (enforced; agent preflight wired)

Never add separate ### Fixed entries, or non-**Breaking:** ### Changed entries, for code that has not yet been released. Fixes and tweaks to unreleased features must be folded into the existing ### Added entry describing that feature. Only **Breaking:** entries (for already-released types -- e.g., new variants on a non-#[non_exhaustive] enum) belong in ### Changed of [Unreleased]. A fix to behavior that already shipped in a released version DOES get its own ### Fixed entry, prefixed **Pre-existing:** (the self-declaration the hook accepts, mirroring **Breaking:**). The changelog describes the final shipped state, not intermediate development history.

Release bump floor (enforced)

Release preparation derives a minimum semantic-version bump from the complete [Unreleased] section and refuses a smaller operator selection. Added, Deprecated, and non-breaking Changed entries require at least a minor bump. **Breaking:** and Removed entries require a minor bump before 1.0 and a major bump at or after 1.0. A patch is valid only when the release contains Fixed/Security entries. An empty [Unreleased] section is not releasable. Wire-version changes belong in Changed with **Breaking:**, even when they accompany a pre-existing behavioral fix.

Validate locally: python3 scripts/hooks/check-changelog-unreleased.py. Also runs automatically in scripts/ci/agent-preflight.py whenever CHANGELOG.md is in the changed set, and as the changelog-unreleased-rule pre-commit hook.

Version sync rule: If Cargo.toml is X.Y.Z, the matching changelog header must be ## [X.Y.Z] - YYYY-MM-DD (ISO date required). Keep ## [Unreleased] undated. Validate with bash scripts/sync-version.sh --check; auto-fix with bash scripts/sync-version.sh --changelog-only.

Workspace lock rule: Cargo owns one checked-in lock per discovered workspace root. Use python3 scripts/release/workspace_locks.py sync after manifest version changes and python3 scripts/release/workspace_locks.py check for full freshness validation. Never bypass --locked or use --no-deps as a lock oracle. Generator fixes made on release branches must also land on main.

Mandatory Linting

• After Rust changes: cargo fmt && cargo clippy --workspace --all-targets --features tokio,json (or cargo c)
• After workflow changes: actionlint (no exceptions)
• After doc changes: cargo doc --no-deps
• After markdown changes: npx markdownlint 'file.md' --config .markdownlint.json --fix
• Before finalizing agent work: python3 scripts/ci/agent-preflight.py --auto-fix (changed-file-aware early checks; if output includes Falling back to --all checks., resolve git-state issues and rerun)
• After shell-script changes: bash scripts/ci/check-shell-portability.sh
• Hook policy: pre-commit/pre-push target <10s; slow full-repo checks belong in manual hooks, agent preflight, or CI.
• After Agent Skills changes: Run python scripts/hooks/validate-agent-skills.py; every SKILL.md must stay within the 500-line open-format recommendation
• Doc/link validation: python3 scripts/docs/check-links.py plus bash scripts/ci/check-doc-claims.sh for Rust semantic claim drift (also flags rustdoc intra-doc links whose backticked text names an item the crate::/super::/self:: target's last segment does not -- they resolve but land on the wrong page; see doc-code-sync.md). Runs in agent preflight when .rs/.md files change.
• Spell check: typos (also runs in agent preflight via agent-typos.py)
• Vale (advisory): vale docs/ -- checks prose quality, non-blocking in CI. Cheat-sheet of recurring swaps and weasel words: .agents/skills/user-facing-docs/SKILL.md. Agent preflight surfaces per-file counts via the vale-advisory check.
• Full local validation: cargo fmt && cargo clippy --workspace --all-targets --features tokio,json && cargo nextest run --no-capture

Shell regex portability rule: avoid PCRE-style escapes in grep -E/sed -E (\b, \s, \w, etc.). Use POSIX-safe classes like [[:space:]], [[:alnum:]_], and token boundaries (^|[^[:alnum:]_])word([^[:alnum:]_]|$).

Skill Code Examples

Code examples in .agents/skills/ must follow zero-panic rules with these exceptions:

• build.rs: .unwrap() OK (comment // build.rs:)
• Test code: .unwrap() OK (comment // test: or // In tests:)
• Fuzz targets: .expect() OK (comment // Fuzz target:)
• Loom tests: .unwrap() on .join() OK (comment // Loom test:)
• #[allow] examples: showing lint suppression is the point
• Also accepted: // proptest:, // allowed:, // SAFETY:, #[test], #[fixture], #[cfg(test)] attributes

Additional rules: catch_unwind closures must use AssertUnwindSafe; fully qualify ambiguous types (e.g., arbitrary::Result<T> not bare Result<T>); no 2>/dev/null in shell examples. Run scripts/docs/check-agent-skills.sh after modifying Agent Skills (also enforced in CI via ci-agent-skills.yml).

Breaking Changes Checklist

• CHANGELOG.md updated with Breaking: prefix and migration guidance
• docs/migration.md updated with before/after examples
• README.md and docs/user-guide.md updated if affected
• All examples/*.rs compile: cargo build --examples
• Rustdoc examples compile: cargo test --doc
• Search for old API usage: rg 'old_name' --type rust --type md

Documentation Sync

When changing public APIs, update: rustdoc comments (source of truth), README.md, docs/user-guide.md, examples/, CHANGELOG.md. Search with rg 'function_name|StructName' --type rust --type md.

For docs/wiki mirrors, use a first-line <!-- SYNC: ... --> header with explicit direction: docs pages point to their wiki mirror (wiki/...), wiki pages point to their docs source (docs/...). Never self-reference the same file in a SYNC header.

Quality Checklist

• cargo fmt and cargo clippy --workspace --all-targets --features tokio,json pass
• All tests pass (cargo nextest run)
• New functionality has tests and Rustdoc examples
• Code is 100% safe Rust and handles every error path
• Public APIs have no duplicate methods and document feature dependencies
• Changelog reviewed for pub/user-observable changes

For Agents

When spawning sub-agents or using Task tools: the sub-agent MUST run cargo fmt and verify cargo clippy --workspace --all-targets --features tokio,json passes on any modified files. If the sub-agent cannot run these, the parent agent must run them after receiving changes.

GitHub Access in Devcontainers

Use callable VS Code integrations first: the built-in Git extension for source control and GitHub Pull Requests and Issues (GitHub.vscode-pull-request-github) for GitHub operations; UI presence alone is not callable access. If they cannot perform the operation, use local git for repository work and the connected GitHub connector/app for platform work. gh is an absolute fallback, allowed only when the VS Code connector, local Git, and connected GitHub connector/app are unavailable or incapable. gh auth status is not a prerequisite when another supported path exists: probe Git reads with git ls-remote origin HEAD and writes with git push --dry-run origin HEAD:<branch>; SSH usually uses VS Code-forwarded SSH_AUTH_SOCK and HTTPS its credential helper. Never print or persist credentials.

---

License: MIT OR Apache-2.0