source-analysis

# Source Analysis Methodology Extract behavioral intelligence from source code. Decompose the codebase into analyzable units, then systematically analyze each unit for behavioral claims with full provenance. ## When to Use This Mode Source analysis activates when: - The discovery inventory identifies source code files at the target path - Any source code needs behavioral analysis (any language, bundled or not) - Decompiled output exists at `workspace/raw/source/decompiled/` ## General Approach Source analysis takes three shapes depending on what the target looks like: - **Source tree** — a repository with a package manifest and conventional directory layout (Python, Rust, Go, Swift, Java, Node/TS, C++, etc.). Follow the **Source Tree Pipeline** below. - **Bundle** — a single-file minified or packed artifact (JavaScript bundle, Electron asar, webpack/esbuild output). Follow the **Bundle Pipeline** below. - **Decompiled binary** — a compiled artifact where source isn't directly available. Decompile first via the **Decompilation Path**, then analyze the decompiled output using whichever of the above shapes fits. Regardless of shape, the pipeline follows the same logical steps: 1. **Assess the source.** What language? How is it organized? How large is it? 2. **Decompose into analyzable units.** Use the language's natural boundaries (modules, packages, files) when they exist. Split bundled artifacts into chunks when they don't. 3. **Analyze each unit exhaustively.** Read every line. Identify every function, method, class. Understand what each does behaviorally. 4. **Extract behavioral specifications.** Write what the code DOES (observable behavior), not how it's structured (implementation details). Every claim gets a provenance citation. Phases 6-8 (per-unit analysis, per-function deep analysis, targeted extraction) are shape-agnostic and apply to all three paths. ## Decompilation Path When source code isn't directly available, decompile binaries into structured source before analysis. ### Language-Specific Decompilation Tools | Language/Platform | Tools | Notes | |-------------------|-------|-------| | JVM (Java, Kotlin) | cfr, procyon, fernflower | cfr is standalone JAR; fernflower is IntelliJ's built-in decompiler | | .NET (C#, F#) | ilspy, dotPeek CLI | ilspy has a command-line mode suitable for automation | | Python (.pyc, .pyo) | uncompyle6, decompyle3 | decompyle3 targets Python 3.7+; uncompyle6 covers older versions | | JavaScript (bundled/obfuscated) | js-beautify | Also covered by the Bundle Pipeline below | | Native (x86, ARM, etc.) | ghidra headless, radare2 | Produces pseudocode, not true source; useful for behavioral extraction but lower fidelity | ### Process 1. **Identify binary type.** Determine the platform and format (JAR, DLL, .pyc, ELF, Mach-O, etc.). 2. **Check tool availability.** Verify the appropriate decompiler is installed and accessible. If tools are unavailable, warn and fall back to binary analysis (strings, symbols, imports/exports). 3. **Decompile to workspace.** Output decompiled source to `workspace/raw/source/decompiled/`. Preserve directory structure from the binary where possible (e.g., Java package paths). 4. **Treat decompiled output as structured source.** Once decompiled, analyze using the same General Approach above — each decompiled file is an analyzable unit. ```bash mkdir -p workspace/raw/source/decompiled # Example: JVM with cfr java -jar cfr.jar target.jar --outputdir workspace/raw/source/decompiled/ # Example: Python with uncompyle6 uncompyle6 -o workspace/raw/source/decompiled/ target.pyc # Example: .NET with ilspy ilspycmd target.dll -o workspace/raw/source/decompiled/ # Example: Native with Ghidra headless analyzeHeadless /tmp/ghidra_project proj -import target.bin -postScript ExportDecompiled.java workspace/raw/source/decompiled/ ``` ### Quality Notes Decompiled code differs from original source in predictable ways: - **Variable and parameter names are lost.** Decompilers generate synthetic names (e.g., `var1`, `a0`). Focus on behavioral patterns, not identifier semantics. - **Comments are lost.** All inline documentation is gone. Rely on string constants, API calls, and control flow for behavioral understanding. - **Control flow may be less readable.** Optimized bytecode can decompile into awkward `goto`-style structures or deeply nested conditionals. These limitations are acceptable for behavioral extraction. The goal is to understand what the code does, not to recover the original source. ## Source Tree Pipeline Source tree analysis is the common path for Python, Rust, Go, Swift, Java, C++, and similar languages that ship as a repository rather than a single compiled artifact. The target is a directory with a package manifest and a conventional source layout. The Source Tree Pipeline defines Phases ST1-ST3 (survey, enumeration, and unit preparation). After ST3, control flows into the shape-neutral Phases 6-8 shared with the Bundle Pipeline. ```dot digraph source_tree_analysis { rankdir=TB; "Start (source tree)" [shape=doublecircle]; "Phase ST1: Manifest discovery" [shape=box]; "Phase ST2: Module enumeration" [shape=box]; "Phase ST3: Prepare for per-unit analysis" [shape=box]; "→ Phase 6 (shape-neutral)" [shape=box, style=filled, fillcolor="#f0f0f0"]; "Start (source tree)" -> "Phase ST1: Manifest discovery"; "Phase ST1: Manifest discovery" -> "Phase ST2: Module enumeration"; "Phase ST2: Module enumeration" -> "Phase ST3: Prepare for per-unit analysis"; "Phase ST3: Prepare for per-unit analysis" -> "→ Phase 6 (shape-neutral)"; } ``` ### Phase ST1: Manifest Discovery Identify the package manifest and read it. The manifest names the package, declares dependencies, and lists the source roots. | Language | Manifest | Source roots | |----------|----------|--------------| | Python | `pyproject.toml`, `setup.py`, `setup.cfg` | `src/`, `<package>/`, or top-level modules | | Rust | `Cargo.toml` (+ workspace `Cargo.toml`) | `src/`, each crate's `src/` | | Go | `go.mod` | Module root and every directory with `.go` files | | Swift | `Package.swift`, `*.xcodeproj/project.pbxproj` | `Sources/<target>/` per target | | Java/Kotlin | `pom.xml`, `build.gradle`, `build.gradle.kts` | `src/main/java/`, `src/main/kotlin/` | | C#/F# | `*.csproj`, `*.fsproj` | project-relative source directories | | C/C++ | `CMakeLists.txt`, `Makefile`, `meson.build` | varies; often `src/` and `include/` | | Node (non-bundled) | `package.json` | `src/`, `lib/`, `index.js` | Write the survey to `workspace/raw/source/analysis/survey.md`: ```markdown # Source Tree Survey **Language:** <primary> **Manifest:** <path> **Package name:** <from manifest> **Source roots:** <list> **Dependency count:** <count> **Line count by language:** <table> ``` ### Phase ST2: Module Enumeration Walk the source roots and produce a module list. A "module" in this pipeline is whatever the language's natural unit of organization is — a Python package, a Rust module/crate, a Go package, a Swift target, a Java package. Don't invent your own boundaries; use the language's. ```bash # Python: every directory with __init__.py plus every top-level .py file find <source-root> -name '__init__.py' -exec dirname {} \; find <source-root> -maxdepth 1 -name '*.py' -not -name '__init__.py' # Rust: every directory with a mod.rs or every file in src/ find <crate>/src -name '*.rs' # Go: every directory with at least one .go file find <source-root> -name '*.go' -exec dirname {} \; | sort -u # Swift: every directory under Sources/ find Sources -mindepth 2 -maxdepth 2 -type d # Java/Kotlin: every directory with .java or .kt files find src/main -name '*.java' -o -name '*.kt' -exec dirname {} \; | sort -u ``` Write the module inventory to `workspace/raw/source/manifests/module-manifest.json`: ```json { "modules": [ {"path": "src/parser", "files": 7, "lines": 1240, "language": "rust"}, {"path": "src/http", "files": 4, "lines": 680, "language": "rust"} ] } ``` ### Phase ST3: Prepare for Per-Unit Analysis Once Phases ST1-ST2 have produced the survey and module manifest, each module is ready to be treated as the analyzable unit for the shape-neutral Phase 6. The analysis unit differs by target shape: in a bundle, the unit is a chunk from the bundle-splitter; in a source tree, it's a module as the language defines it. Both produce the same per-unit output format. Phases 6-8 (Per-Unit Analysis, Per-Function Deep Analysis, Targeted Extraction), described later in this skill, apply from here on. --- ## Bundle Pipeline The Bundle Pipeline defines Phases B1-B5 (bundle-specific decomposition). After B5, control flows into the shape-neutral Phases 6-8 shared with the Source Tree Pipeline. ```dot digraph bundle_pipeline { rankdir=TB; "Start (bundle target)" [shape=doublecircle]; "Phase B1: Beautify bundle" [shape=box]; "Phase B2: Find module boundaries" [shape=box]; "Phase B3: Extract chunks" [shape=box]; "Phase B4: Extract functions" [shape=box]; "Phase B5: Extract strings" [shape=box]; "→ Phase 6 (shape-neutral)" [shape=box, style=filled, fillcolor="#f0f0f0"]; "Start (bundle target)" -> "Phase B1: Beautify bundle"; "Phase B1: Beautify bundle" -> "Phase B2: Find module boundaries"; "Phase B2: Find module boundaries" -> "Phase B3: Extract chunks"; "Phase B3: Extract chunks" -> "Phase B4: Extract functions"; "Phase B4: Extract functions" -> "Phase B5: Extract strings"; "Phase B5: Extract strings" -> "→ Phase 6 (shape-neutral)"; } ``` Phases B1-B5 are bundle decomposition infrastructure (bundle-splitter). Phases 6-7 produce behavioral analysis (chunk-analyzer, function-analyzer) and are shape-neutral. Phase 8 is optional targeted extraction (targeted-extractor). For non-bundled targets, run the Source Tree Pipeline's Phases ST1-ST3 instead of this section's B1-B5, then flow into the same shape-neutral Phases 6-8. ## Phase B1: Bundle Preparation ### 1.1 Beautify if Needed Check if the bundle is minified by comparing line count to byte count. A file with few lines but many bytes is minified. ```bash LINE_COUNT=$(wc -l < "$BUNDLE") BYTE_COUNT=$(wc -c < "$BUNDLE") if [ "$LINE_COUNT" -lt 100 ] && [ "$BYTE_COUNT" -gt 100000 ]; then echo "Minified - needs beautification" mkdir -p workspace/raw/source BEAUTIFIED="workspace/raw/source/target.js" if [ ! -f "$BEAUTIFIED" ]; then npx js-beautify --type js -f "$BUNDLE" -o "$BEAUTIFIED" fi TARGET="$BEAUTIFIED" else TARGET="$BUNDLE" fi ``` Always check for an existing beautified version before re-beautifying. Multiple agents may operate on the same bundle. ### 1.2 Initial Survey ```bash # Size and line count ls -lh "$TARGET" wc -l "$TARGET" # Detect bundler type head -100 "$TARGET" > workspace/raw/source/analysis/bundle-header.txt grep -m1 "webpack\|esbuild\|rollup\|parcel" "$TARGET" || echo "Unknown bundler" ``` Write the survey to `workspace/raw/source/analysis/survey.md`. ## Phase B2: Find Module Boundaries ### 2.1 Bundler-Specific Markers #### Webpack ```bash # Module ID markers grep -n "^/\*\*\*\*\*\*/ \"" "$TARGET" > manifests/webpack-modules.txt # Export declarations grep -n "__webpack_require__\.d\|__webpack_exports__" "$TARGET" | head -100 > manifests/webpack-exports.txt ``` #### esbuild ```bash # esbuild source comments grep -n "^// " "$TARGET" | grep -v "^// $" > manifests/esbuild-markers.txt ``` #### Generic IIFE Boundaries ```bash # Function expression boundaries grep -n "^(function\|^!function\|^var [a-z]* = function" "$TARGET" > manifests/iife-boundaries.txt ``` ### 2.2 Heuristic Boundaries When no bundler-specific markers are found, fall back to heuristics: ```bash # Long comment separators grep -n "^/\*\*\*\*\*\*\*\*" "$TARGET" > manifests/separator-comments.txt # Export clusters grep -n "^exports\.\|^module\.exports" "$TARGET" > manifests/export-points.txt # Large gaps (blank lines often separate modules) awk 'NF==0 {if(++blank>=3) print NR": --- blank section ---"} NF>0 {blank=0}' "$TARGET" | head -100 ``` ## Phase B3: Extract Chunks ### 3.1 Split by Module Markers ```bash mkdir -p workspace/raw/source/chunks # Webpack-style split csplit -f workspace/raw/source/chunks/chunk- -n 4 "$TARGET" '/^\/\*\*\*\*\*\*\//' '{*}' 2>/dev/null # Fall back to line-count splitting if no clear markers if [ $(ls workspace/raw/source/chunks/ | wc -l) -lt 5 ]; then rm workspace/raw/source/chunks/* split -l 2000 "$TARGET" workspace/raw/source/chunks/chunk- fi # Rename to .js for f in workspace/raw/source/chunks/chunk-*; do [ "${f%.js}" = "$f" ] && mv "$f" "${f}.js" done ``` ### 3.2 Generate Chunk Manifest ```bash