core-internals

# Git Repo Core Internals Engineering Guide ## Executive Summary Welcome to the authoritative engineering guide for Git Repository Management within the git-repo codebase. This repository of folk knowledge exists to safeguard the intricate orchestration of our multi-repository synchronization toolchain. Historically, complex operations such as parallelized network fetches, layered manifest overrides, and interactive terminal rendering have been vulnerable to subtle concurrency races, deadlocks, and fragile system states. This guide captures these critical failure modes—ranging from IPC serialization bottlenecks and filesystem lock contentions to non-hermetic test pollution—and establishes rigid engineering constraints to prevent their regression. To maintain system stability, this guide enforces strict architectural boundaries across the tooling ecosystem. It mandates stateless execution for multiprocessing pools, guaranteed atomicity for worktree layout modifications, and deterministic error translation for all standard Git subprocesses. Furthermore, it defines the standard operating procedures for canonical manifest object deduplication, the modernization of our hermetic testing frameworks, and the delivery of consistent, machine-readable CLI interfaces. By adhering to these paradigms, incoming engineers will ensure the reliability, performance, and extensibility of git-repo's core repository management infrastructure. ## Summary | Chapter Theme / Title | Scope & Objective | | :----------------------------------- | :------------------------------------ | | **Concurrent Synchronization & IPC** | This domain governs the stable | : : orchestration of parallel network : : : fetches, local checkouts, and : : : interleaved subprocess routines. It : : : strictly enforces safe : : : multiprocessing IPC, deterministic : : : Git locking via exponential backoff, : : : and state synchronization across : : : concurrent pool workers. : | **Filesystem Atomicity & Worktree | This domain governs the deterministic | : Layout** : creation, migration, and cleanup of : : : internal repository structures and : : : Git worktrees. It relies on ephemeral : : : temporary directories, atomic rename : : : operations, and robust error recovery : : : to prevent corrupted states during : : : unexpected interruptions. : | **Subprocess Git Integration & Error | This chapter defines the constraints | : Translation** : for wrapping, executing, and : : : translating standard Git subprocesses : : : within the Repo tooling ecosystem. It : : : mandates the use of centralized : : : command abstractions, strict version : : : gating, and deterministic stream : : : handling to guarantee reliable : : : repository state management and : : : actionable error reporting. : | **Manifest Object Model & | This chapter governs the parsing, | : Deduplication** : validation, and canonicalization of : : : XML manifest components. It strictly : : : enforces semantic immutability via : : : NamedTuple implementations, defensive : : : copying for hierarchical override : : : scoping, and deterministic : : : JSON-backed file tracking across the : : : subsystem. : | **Hermetic Testing & Test | This domain governs the migration of | : Modernization** : legacy unittest suites to modern : : : pytest functional paradigms and the : : : establishment of hermetic session : : : fixtures. It enforces strict : : : environment isolation to prevent : : : global state pollution (e.g., : : : developer .gitconfig bleeding) while : : : safely intercepting standard streams : : : and filesystem paths. : | **CLI Argument Parsing & UX | This chapter governs the lifecycle, | : Consistency** : validation, and execution of : : : command-line arguments, enforcing : : : strict standardization for : : : machine-readable serialization, : : : unified logging, and deterministic, : : : thread-safe terminal interactions. : | **Repo Hooks Framework** | The Repo Hooks Framework governs the | : : execution, parameter validation, and : : : lifecycle management of user-defined : : : scripts within the repository : : : ecosystem. It ensures seamless : : : integration of extensions like : : : post-sync or pre-upload while : : : strictly isolating their execution : : : failures from core operational : : : workflows. : -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- ## Chapter: Concurrent Synchronization & IPC **Context:** This domain governs the stable orchestration of parallel network fetches, local checkouts, and interleaved subprocess routines. It strictly enforces safe multiprocessing IPC, deterministic Git locking via exponential backoff, and state synchronization across concurrent pool workers. ### Summary | Rule ID | Principle / Constraint | Priority | Primary Symptom / Trap | | :-------- | :------------------------ | :------- | :------------------------ | | **T1-01** | Stateless Class Methods | High | Passing bound instance | : : for Parallel Pool Workers : : methods to a parallel : : : : : executor, dragging : : : : : unnecessary state into : : : : : the multiprocessing : : : : : serialization pipeline. : | **T1-02** | Buffered Serialization of | High | Executing terminal print | : : Concurrent Standard : : calls directly from : : : Output : : within a parallelized : : : : : worker routine. : | **T1-03** | Interleaved Sync Path | High | Dispatching parallel | : : Validation : : checkout jobs across all : : : : : project variants without : : : : : verifying layout types. : | **T1-04** | Guaranteed | Critical | Placing save operations | : : Synchronization State : : at the very end of an : : : Persistence : : execution path without : : : : : exception guards. : | **T1-05** | Jittered Exponential | Critical | Firing `git submodule | : : Backoff for Git : : init` concurrently : : : Configuration Locks : : without retry logic for : : : : : `config.lock` failures. : | **T1-06** | Exponential Backoff for | High | Failing a git command | : : Concurrent Git Config : : immediately without : : : Locks : : evaluating standard : : : : : streams for transient : : : : : lock errors. : | **T1-07** | Hard Termination on | Critical | Logging an error state | : : Interleaved Sync Stalls : : inside a `while` loop but : : : : : allowing the next loop : : : : : iteration to execute. : | **T1-08** | Resource Management via | High | Initializing proxy | : : Context Handlers for IPC : : connections or : : : : : multiprocessing managers : : : : : as raw variable : : : : : assignments without a : : : : : guaranteed teardown : : : : : phase. : | **T1-09** | Dual-Channel Error | High | Iterating over aggregated | : : Handling in Parallel : : data lists to decide if a : : : Processing : : parallel orchestration : : : : : should abort. : | **T1-10** | Minimum Process Pool Job | Critical | Directly using | : : Count Safeguard : : `min(target, len(items))` : : : : : to determine pool size, : : : : : which breaks if the item : : : : : list is unexpectedly : : : : : empty. : | **T1-11** | Safeguarding Object State | Critical | Truncating object | : : Across Parallel Execution : : payloads to scalar : : : Boundaries : : indices during IPC : : : : : context setup without : : : : : updating the receiver : : : : : logic to rehydrate the : : : : : objects. : | **T1-12** | Explicit Context | High | Assuming child workers | : : Initialization for : : inherit updated class : : : Multiprocessing Pools : : variables inherently : : : : : without explicit : : : : : initialization. : | **T1-13** | Dynamic Task Chunk Sizing | Medium | Passing a hardcoded batch | : : in Parallel Execution : : integer to the : : : : : `chunksize` parameter. : | **T1-14** | Deferred Worktree | High | Returning early from the | : : Operations in Sync Local : : sync phase without : : : Half : : applying required file : : : : : operations. : -------------------------------------------------------------------------------- ### Rules #### T1-01: Stateless Class Methods for Parallel Pool Workers > **Rule:** Always implement multiprocessing pool targets as stateless class > methods or standalone functions to bypass process serialization constraints. > > **What:** Worker execution targets within multiprocessing pools must be > constructed as fully decoupled class methods or standalone functions to bypass > process serialization constraints. > > **Applies To:** Multiprocessing process pools, concurrent task scheduling, and > `ExecuteInParallel` implementations. > > **Why:** Command execution objects were heavily bound to system state. > Attempting to pass instance methods (`self.method`) to a multiprocessing pool > frequently caused `PicklingError` crashes, as the underlying Python > serialization mechanism cannot cleanly isolate bound object graphs. Failing to > adhere to this typically results in **IPC Serialization Error**. **Trap 1: Passing bound instance methods to a parallel executor, dragging unnecessary state into the multiprocessing serialization pipeline.** **Don't:** ```python class InfoCommand: def _worker_logic(self, project): pass def run(self): self.ExecuteInParallel(jobs, self._worker_logic, projects) ``` **Do:** ```python class InfoCommand: @classmethod def _worker_logic(cls, project_idx): project = cls.get_parallel_context()["projects"][project_idx] pass def run(self): self.ExecuteInParallel(jobs, self._worker_logic, range(len(projects))) ``` **Exceptions:** Threading-based execution models where memory is shared and pickling is not strictly enforced. -------------------------------------------------------------------------------- #### T1-02: Buffered Serialization of Concurrent Standard Output > **Rule:** Must capture standard output generated by concurrently executing > tasks into isolated memory buffers for sequential display in the parent > process. > > **What:** Data emitted by concurrently executing tasks must be captured into > isolated memory buffers and returned to the parent process for sequential > display. > > **Applies To:** Parallel process execution layers generating human-readable > CLI output. > > **Why:** When command execution was parallelized, worker processes wrote > directly to standard output. This created severe race conditions resulting in > garbled, interleaved text output on the user's terminal. Failing to adhere to > this typically results in **Interleaved Terminal Output**. **Trap 1: Executing terminal print calls directly from within a parallelized worker routine.** **Don't:** ```python @classmethod def _DiffHelper(cls, project): print(f"Project: {project.name}") print(f"Revision: {project.rev}") ``` **Do:** ```python @classmethod def _DiffHelper(cls, project): buf = io.StringIO() buf.write(f"Project: {project.name}\n") buf.write(f"Revision: {project.rev}\n") return buf.getvalue() # In main process: for output in results: print(output, end="") ``` -------------------------------------------------------------------------------- #### T1-03: Interleaved Sync Path Validation > **Rule:** Always validate the presence of a worktree when executing > interleaved sync processes to prevent layout evaluation crashes. > > **What:** When executing interleaved sync processes (parallelized network > fetches and checkouts), the operation must explicitly validate the presence of > a worktree, as not all Git repository types (e.g., mirrors) maintain local > file checkouts. > > **Applies To:** Concurrency logic within `sync.py`. > > **Why:** Changing the default mode of `repo sync` to interleaved parallelized > checkout tasks indiscriminately. This immediately broke AOSP mirror syncing > because mirrors lack local checkout paths, resulting in a `TypeError` when > evaluating `NoneType` paths. Failing to adhere to this typically results in > **TypeError / Sync Failure**. **Trap 1: Dispatching parallel checkout jobs across all project variants without