| name | scalpel-tracing |
| description | Use when working on scalpel-tracer crate, language probes in probes/ directory, coverage pipelines, event ingestion, resource attribution, or debugging missing trace events |
Tracing & Instrumentation Expert Knowledge
Core Principle
Every coverage system has blind spots around generated code. Know your coverage system's limits per language, or you'll build a genome with invisible holes.
When to Use
- Modifying any file in
crates/scalpel-tracer/
- Working on any probe in
probes/ (Node.js, Python, Go, PHP, Ruby)
- Debugging missing or incorrect trace events
- Adding support for a new language's coverage mechanism
- Working on event ingestion or resource attribution
- Investigating
<unknown> function names or low match rates
Domain Model
Language Probe → [coverage API] → TraceEvents → Ring Buffer → scalpel-tracer → Code Genome
| |
Cold path: string interning Hot path: 64-byte events
via Unix socket (~2-5us) via shared memory (<100ns)
Per-Language Coverage Mechanisms
| Language | Mechanism | What It Sees | What It Misses |
|---|
| JS/Node | V8 Profiler.takePreciseCoverage() | All compiled functions w/ byte ranges | Lazy-parsed (never called), computed properties |
| TypeScript | V8 coverage + source maps | Same as JS after VLQ mapping | Multi-level bundler maps, non-ASCII offset mismatch |
| Python | sys.monitoring PY_START/PY_RETURN | All Python function calls | C-extension fast builtins, decorator phantom calls |
| Go | runtime/coverage (Go 1.20+) | AST-level instrumented functions | init() before coverage init, go:linkname |
| Rust | LLVM -C instrument-coverage | Region-based counters per function | Proc macros, derive impls, async state machines |
| Java | JaCoCo bytecode agent | All loaded class methods | Lambda synthetics, JIT-inlined, Kotlin inline |
| PHP | Xdebug/PCOV | Function-level calls | Dynamic includes |
| Ruby | TracePoint API | Method calls | method_missing, C extensions |
Key Insight
No single coverage system gives complete function-level accuracy. The hybrid approach (runtime + static) is essential, not optional. Runtime is under-approximate (only exercised paths). Static is the soundness backstop.
Decision Trees
Event Processing
digraph event_processing {
"Receive TraceEvent" [shape=box];
"Event type?" [shape=diamond];
"Push (node, cpu, wall) onto thread stack" [shape=box];
"Thread stack empty?" [shape=diamond];
"Log warning: missed ENTER. Skip." [shape=box style=filled fillcolor=lightsalmon];
"Pop stack, compute self_time" [shape=box style=filled fillcolor=lightgreen];
"Async flag set?" [shape=diamond];
"Record pre-await segment only" [shape=box];
"Normal attribution" [shape=box];
"Receive TraceEvent" -> "Event type?";
"Event type?" -> "Push (node, cpu, wall) onto thread stack" [label="ENTER"];
"Event type?" -> "Thread stack empty?" [label="EXIT"];
"Thread stack empty?" -> "Log warning: missed ENTER. Skip." [label="yes"];
"Thread stack empty?" -> "Pop stack, compute self_time" [label="no"];
"Pop stack, compute self_time" -> "Async flag set?";
"Async flag set?" -> "Record pre-await segment only" [label="yes"];
"Async flag set?" -> "Normal attribution" [label="no"];
}
self_time = cpu_delta - sum(children_cpu). This distinction is CRITICAL for identifying real hot spots vs functions that just call slow children.
Probe String Interning
Probe has string (function name, file path):
Check local cache → found? → Use cached Spur ID (hot path, zero IPC)
Not cached → Send INTERN request over Unix socket (cold path, ~2-5us)
→ Wait for INTERN_ACK with u32 ID
→ Cache locally for future hits
Known Traps
1. V8 best-effort vs precise mode
Best-effort misses lazy-parsed functions entirely (not count=0, just absent from output). Always use precise mode via Profiler.startPreciseCoverage({callCount: true, detailed: true}).
2. Python probe closure scoping
When loaded via exec(), _intern_sock = None in except block causes UnboundLocalError from closure scoping. Must use explicit variable binding: declare the variable outside the try block.
3. Python intern server timeout
Original protocol had 10s timeout. Real projects (HTTPie: 11K+ events) need 300s+. The _recv_exact() function must handle partial socket reads with a generous timeout.
4. Node.js real-time mode IDs
All IDs were 0 (unusable). Fixed with local ID counter + batch string table protocol (version byte 0x01). If you see all-zero IDs, the batch protocol isn't active.
5. Source map chaining
Bundlers produce multi-level maps (TS→JS + JS→bundle). Must chain both. Critical: V8 uses UTF-16 code units for offsets, source maps use byte offsets. This causes off-by-one errors for non-ASCII identifiers.
6. Go init() timing
May execute before coverage runtime initializes. These calls are silently dropped. Log a warning — don't silently ignore missing init() coverage.
Verification Protocol
After ANY change to tracing code:
cargo test -p scalpel-tracer- For probe changes: test against real projects, not just synthetics
- Node.js: Express.js app (expect ~578 matched functions)
- Python: HTTPie project (expect 11K+ events)
- TypeScript: any TS app with source maps
- Check event count is non-zero and reasonable for project size
- Check function match rate:
matched / total_runtime_functions (target: >90% for called functions)
- Check for
<unknown> function names — should be <5% after interning
- Verify TraceEvent is exactly 64 bytes:
assert_eq!(std::mem::size_of::<TraceEvent>(), 64)
Red Flags — STOP
- Assuming all coverage systems work the same way
- Testing probe changes with synthetic 5-function projects only
- Ignoring
<unknown> function names (interning protocol failure)
- Changing TraceEvent fields without checking 64-byte alignment
- Setting probe timeouts under 60 seconds (real projects are slow)
- Using V8 best-effort mode instead of precise mode
Quick Reference
| Operation | Rule |
|---|
| New language probe | Document coverage mechanism + blind spots first |
<unknown> names >5% | Interning protocol bug — investigate |
| Event count = 0 | Probe not connecting or coverage not enabled |
| Match rate <50% | Check: byte offsets, name reconciliation, source maps |
| Timeout errors | Increase to 300s minimum for real projects |
| Async functions | Record pre-await segment only, don't attribute suspension wall-time |
| TraceEvent change | Verify 64-byte compile-time assertion |
