scalpel-tracing

name	scalpel-tracing
description	Use when working on scalpel-tracer crate, language probes in probes/ directory, coverage pipelines, event ingestion, resource attribution, or debugging missing trace events

Tracing & Instrumentation Expert Knowledge

Core Principle

Every coverage system has blind spots around generated code. Know your coverage system's limits per language, or you'll build a genome with invisible holes.

When to Use

• Modifying any file in crates/scalpel-tracer/
• Working on any probe in probes/ (Node.js, Python, Go, PHP, Ruby)
• Debugging missing or incorrect trace events
• Adding support for a new language's coverage mechanism
• Working on event ingestion or resource attribution
• Investigating <unknown> function names or low match rates

Domain Model

Language Probe → [coverage API] → TraceEvents → Ring Buffer → scalpel-tracer → Code Genome
     |                                              |
     Cold path: string interning                    Hot path: 64-byte events
     via Unix socket (~2-5us)                       via shared memory (<100ns)

Per-Language Coverage Mechanisms

Language	Mechanism	What It Sees	What It Misses
JS/Node	V8 Profiler.takePreciseCoverage()	All compiled functions w/ byte ranges	Lazy-parsed (never called), computed properties
TypeScript	V8 coverage + source maps	Same as JS after VLQ mapping	Multi-level bundler maps, non-ASCII offset mismatch
Python	sys.monitoring PY_START/PY_RETURN	All Python function calls	C-extension fast builtins, decorator phantom calls
Go	runtime/coverage (Go 1.20+)	AST-level instrumented functions	init() before coverage init, go:linkname
Rust	LLVM -C instrument-coverage	Region-based counters per function	Proc macros, derive impls, async state machines
Java	JaCoCo bytecode agent	All loaded class methods	Lambda synthetics, JIT-inlined, Kotlin inline
PHP	Xdebug/PCOV	Function-level calls	Dynamic includes
Ruby	TracePoint API	Method calls	method_missing, C extensions

Key Insight

No single coverage system gives complete function-level accuracy. The hybrid approach (runtime + static) is essential, not optional. Runtime is under-approximate (only exercised paths). Static is the soundness backstop.

Decision Trees

Event Processing

digraph event_processing {
    "Receive TraceEvent" [shape=box];
    "Event type?" [shape=diamond];
    "Push (node, cpu, wall) onto thread stack" [shape=box];
    "Thread stack empty?" [shape=diamond];
    "Log warning: missed ENTER. Skip." [shape=box style=filled fillcolor=lightsalmon];
    "Pop stack, compute self_time" [shape=box style=filled fillcolor=lightgreen];
    "Async flag set?" [shape=diamond];
    "Record pre-await segment only" [shape=box];
    "Normal attribution" [shape=box];

    "Receive TraceEvent" -> "Event type?";
    "Event type?" -> "Push (node, cpu, wall) onto thread stack" [label="ENTER"];
    "Event type?" -> "Thread stack empty?" [label="EXIT"];
    "Thread stack empty?" -> "Log warning: missed ENTER. Skip." [label="yes"];
    "Thread stack empty?" -> "Pop stack, compute self_time" [label="no"];
    "Pop stack, compute self_time" -> "Async flag set?";
    "Async flag set?" -> "Record pre-await segment only" [label="yes"];
    "Async flag set?" -> "Normal attribution" [label="no"];
}

self_time = cpu_delta - sum(children_cpu). This distinction is CRITICAL for identifying real hot spots vs functions that just call slow children.

Probe String Interning

Probe has string (function name, file path):
  Check local cache → found? → Use cached Spur ID (hot path, zero IPC)
  Not cached → Send INTERN request over Unix socket (cold path, ~2-5us)
    → Wait for INTERN_ACK with u32 ID
    → Cache locally for future hits

Known Traps

1. V8 best-effort vs precise mode

Best-effort misses lazy-parsed functions entirely (not count=0, just absent from output). Always use precise mode via Profiler.startPreciseCoverage({callCount: true, detailed: true}).

2. Python probe closure scoping

When loaded via exec(), _intern_sock = None in except block causes UnboundLocalError from closure scoping. Must use explicit variable binding: declare the variable outside the try block.

3. Python intern server timeout

Original protocol had 10s timeout. Real projects (HTTPie: 11K+ events) need 300s+. The _recv_exact() function must handle partial socket reads with a generous timeout.

4. Node.js real-time mode IDs

All IDs were 0 (unusable). Fixed with local ID counter + batch string table protocol (version byte 0x01). If you see all-zero IDs, the batch protocol isn't active.

5. Source map chaining

Bundlers produce multi-level maps (TS→JS + JS→bundle). Must chain both. Critical: V8 uses UTF-16 code units for offsets, source maps use byte offsets. This causes off-by-one errors for non-ASCII identifiers.

6. Go init() timing

May execute before coverage runtime initializes. These calls are silently dropped. Log a warning — don't silently ignore missing init() coverage.

Verification Protocol

After ANY change to tracing code:

1. cargo test -p scalpel-tracer
2. For probe changes: test against real projects, not just synthetics
   • Node.js: Express.js app (expect ~578 matched functions)
   • Python: HTTPie project (expect 11K+ events)
   • TypeScript: any TS app with source maps
3. Check event count is non-zero and reasonable for project size
4. Check function match rate: matched / total_runtime_functions (target: >90% for called functions)
5. Check for <unknown> function names — should be <5% after interning
6. Verify TraceEvent is exactly 64 bytes: assert_eq!(std::mem::size_of::<TraceEvent>(), 64)

Red Flags — STOP

• Assuming all coverage systems work the same way
• Testing probe changes with synthetic 5-function projects only
• Ignoring <unknown> function names (interning protocol failure)
• Changing TraceEvent fields without checking 64-byte alignment
• Setting probe timeouts under 60 seconds (real projects are slow)
• Using V8 best-effort mode instead of precise mode

Quick Reference

Operation	Rule
New language probe	Document coverage mechanism + blind spots first
<unknown> names >5%	Interning protocol bug — investigate
Event count = 0	Probe not connecting or coverage not enabled
Match rate <50%	Check: byte offsets, name reconciliation, source maps
Timeout errors	Increase to 300s minimum for real projects
Async functions	Record pre-await segment only, don't attribute suspension wall-time
TraceEvent change	Verify 64-byte compile-time assertion