seta-statistical-fault-attribution

name

seta-statistical-fault-attribution

description

Diagnose and attribute faults in compound AI systems (multi-model pipelines) using SETA's modular robustness testing framework. Applies perturbations, traces execution through each component, computes per-component metamorphic relation scores, and statistically attributes system failures to specific modules. Use when: 'find which model in my pipeline is failing', 'robustness test my multi-model system', 'trace error propagation in my AI pipeline', 'attribute faults in my compound AI system', 'which component is causing my pipeline to break under noise', 'test robustness of my ML pipeline'.

SETA: Statistical Fault Attribution for Compound AI Systems

This skill enables Claude to design, implement, and apply modular robustness testing for compound AI systems — pipelines composed of multiple interconnected neural networks or ML models. Using the SETA framework (Statistical Fault Attribution), Claude instruments a multi-model pipeline to record execution traces under clean and perturbed inputs, defines per-component metamorphic relations (correctness specifications), and computes Failure Contribution scores that statistically pinpoint which component is most responsible for system-level failures. This replaces opaque end-to-end metrics with fine-grained, per-module fault attribution.

When to Use

• When a user has a multi-model AI pipeline (e.g., detection -> classification -> OCR) and wants to know which component degrades most under perturbations
• When building robustness tests for a compound AI system and the user needs more than end-to-end accuracy
• When a user asks to trace how errors propagate from one model to the next in a pipeline
• When a user wants to compare component-level robustness across different perturbation types (noise, blur, weather)
• When debugging a production ML pipeline where system accuracy drops but it is unclear which model is at fault
• When a user needs to prioritize which model in a pipeline to retrain, harden, or replace
• When designing test harnesses for safety-critical multi-network systems (autonomous vehicles, medical imaging pipelines, inspection systems)

Key Technique

Modular testing via execution traces and metamorphic relations. SETA models a compound AI system as a state-transition system S = (Q, I, Phi, F, R, S) where Q is the set of computational modules, R contains routing functions that determine which module fires next, and F holds the model functions. For each test input x, the framework records an execution trace T(x) — a sequence of nodes v = (q, x_v, y_v, S_q) capturing which module q was activated, what it received, what it produced, and how to score it. A perturbation g(x) produces a perturbed trace T(g(x)). By aligning nodes across traces using structural identifiers, SETA compares each component's behavior under clean vs. perturbed conditions.

Per-component correctness via composite metamorphic relations. Instead of a single end-to-end metric, each module i gets a composite score S_i(x, x_tilde) = product of M_ij(x, x_tilde) over all its metamorphic relations. These relations encode domain-specific invariants: "detections should be a subset of originals" (for detectors), "class label should not change" (for classifiers), "IoU must exceed 0.9 for persisting detections," or "Levenshtein distance below threshold" (for OCR). The system-level score is S(x, x_tilde) = product of all S_i. A system failure occurs when S = 0.

Statistical fault attribution via Failure Contribution scores. The key metric is FC_i = E[Z_i * I(S(x, x_tilde) = 0)] where Z_i = 1 - S_i measures component i's deviation and I(S=0) indicates system failure. Empirically: FC_i = sum over all (input, perturbation) pairs of Z_i * I(S=0), divided by the total number of system failures. The normalized attribution weight alpha_i = FC_i / sum(FC_j) tells you each component's share of blame. A high alpha_i means component i is the primary bottleneck.

Step-by-Step Workflow

1. Map the pipeline topology. Enumerate every model/module in the compound system. Identify the routing logic: which modules are always invoked, which are conditionally invoked (e.g., a detector triggers specialized classifiers). Document input/output formats for each module.

2. Instrument execution trace recording. Wrap each module to capture (module_id, input, output) tuples during inference. Store traces as ordered lists of node records. Ensure the wrapper is architecture-agnostic — treat each module as a black box with known I/O types.

   @dataclass
   class TraceNode:
       module_id: str
       input_data: Any
       output_data: Any
       score_fn: Callable  # metamorphic relation scorer
   
   def record_trace(pipeline, x) -> list[TraceNode]:
       trace = []
       # Run pipeline, intercepting each module call
       for module_id, module_fn in pipeline.modules_in_order(x):
           inp = pipeline.get_input(module_id, x, trace)
           out = module_fn(inp)
           trace.append(TraceNode(module_id, inp, out, pipeline.scorer(module_id)))
       return trace
   

3. Define metamorphic relations for each component. Write scoring functions S_q(y, y_tilde) -> {0, 1} that encode what "correct under perturbation" means for each module. Common patterns:

   • Detectors: (a) detections are subset of originals, (b) IoU > threshold for persisting boxes, (c) class labels unchanged, (d) downstream invocations unchanged
   • Classifiers: output label matches reference label
   • OCR/text models: Levenshtein distance below threshold tau
   • Regressors: absolute difference below epsilon
   • Embeddings: cosine similarity above threshold

4. Select and configure perturbations. Choose perturbation functions g(x) appropriate to the input modality. For images: use the imagecorruptions library (gaussian_noise, fog, frost, motion_blur, etc.) at multiple severity levels (1-5). For text: typo injection, synonym replacement. For audio: noise overlay, speed changes. Apply each perturbation to the full test set.

Concrete Examples

Example 1: Autonomous Rail Inspection Pipeline

User: "I have a rail inspection system with a YOLOv4-tiny detector that feeds into 5 specialized classifiers (speed signs, signals, crossings, etc.). Accuracy drops under fog. Which component should I fix?"

Approach:

1. Map the pipeline: f0 (YOLO detector) -> conditional routing -> f1-f5 (classifiers)
2. Instrument each module to record traces
3. Define MRs: for f0, detections subset + IoU > 0.9 + labels unchanged + routing unchanged; for f1-f5, label equality
4. Apply fog perturbation at severity 1-5 from imagecorruptions
5. Run paired inference on 500 test images x 5 fog levels = 2500 pairs
6. Compute FC scores

Output:

SETA Fault Attribution Report — Fog Perturbation
=================================================
Component         | FC_i   | alpha_i | Failures (of 2500)
------------------|--------|---------|-------------------
f0 (YOLO det.)    | 0.847  | 0.621   | 1,843
f1 (speed_sign)   | 0.198  | 0.145   | 312
f3 (crossing_sig) | 0.214  | 0.157   | 287
f2 (main_signal)  | 0.072  | 0.053   | 104
f4 (catenary)     | 0.033  | 0.024   | 48

Recommendation: f0 (detector) accounts for 62.1% of system failures
under fog. Retrain with fog-augmented data or add defogging
preprocessing before detection.

Example 2: Document Processing Pipeline

User: "My pipeline has OCR -> NER -> relation extraction. It works on clean scans but fails on noisy faxes. Help me find the bottleneck."

Approach:

1. Map pipeline: f0 (OCR engine) -> f1 (NER model) -> f2 (relation extractor)
2. Define MRs: f0 uses Levenshtein distance <= 2, f1 uses entity set equality, f2 uses relation tuple matching
3. Apply perturbations: Gaussian noise, JPEG compression, brightness variation at multiple severity levels
4. Run on 200 test documents x 3 perturbation types x 3 severities = 1800 pairs

Output:

SETA Fault Attribution Report — Noisy Document Input
=====================================================
Component        | FC_i   | alpha_i | By Perturbation (alpha)
-----------------|--------|---------|------------------------
f0 (OCR)         | 0.912  | 0.704   | noise:0.82 jpeg:0.54 bright:0.71
f1 (NER)         | 0.247  | 0.191   | noise:0.12 jpeg:0.31 bright:0.18
f2 (RelExtract)  | 0.136  | 0.105   | noise:0.06 jpeg:0.15 bright:0.11

Recommendation: OCR dominates failures (70.4%), especially under
Gaussian noise (alpha=0.82). Consider a denoising preprocessor or
switch to a noise-robust OCR model.

Example 3: Implementing SETA Harness in Python

User: "I want to add robustness testing to my image pipeline. Can you write the testing harness?"

Approach: Generate a reusable Python module implementing the full SETA loop.

Output (key structure):

# seta_harness.py
from dataclasses import dataclass, field
from collections import defaultdict
from typing import Any, Callable, Protocol

class MetamorphicRelation(Protocol):
    def __call__(self, y_clean: Any, y_perturbed: Any) -> int: ...

@dataclass
class ModuleSpec:
    module_id: str
    fn: Callable
    relations: list[MetamorphicRelation]

@dataclass
class PipelineSpec:
    modules: list[ModuleSpec]
    router: Callable  # (input, prev_outputs) -> next_module_id

class SETAHarness:
    def __init__(self, pipeline: PipelineSpec, perturbations: list[Callable]):
        self.pipeline = pipeline
        self.perturbations = perturbations

    def run_trace(self, x):
        """Execute pipeline, return trace as dict of module_id -> output."""
        ...

    def score_component(self, module_id, y_clean, y_pert) -> int:
        
        spec = ._get_spec(module_id)
         ((mr(y_clean, y_pert)  mr  spec.relations))

     () -> :
        
        fc = defaultdict()
        n_failures = 
         x  test_data:
            trace_clean = .run_trace(x)
             g  .perturbations:
                trace_pert = .run_trace(g(x))
                scores = {}
                system_ok = 
                 mid  trace_clean:
                    s = .score_component(mid, trace_clean[mid], trace_pert.get(mid))
                    scores[mid] = s
                     s == :
                        system_ok = 
                  system_ok:
                    n_failures += 
                     mid, s  scores.items():
                        fc[mid] += ( - s)
        alpha = {m: fc[m]/(fc.values())  m  fc}  fc  {}
         {: (fc), : alpha, : n_failures}

Best Practices

• Do: Define metamorphic relations that are tight but realistic. An IoU threshold of 0.9 for detector persistence is meaningful; 0.5 is too loose to catch real degradation.
• Do: Test across multiple perturbation types and severities. A component robust to noise may be fragile to blur — SETA's per-perturbation breakdown reveals this.
• Do: Record routing divergences (when perturbation causes a different subgraph to execute). These are first-class failures in SETA, not just metric differences.
• Do: Normalize FC scores to alpha weights for cross-component comparison. Raw FC values depend on how many test cases invoke each component.
• Avoid: Relying solely on end-to-end accuracy. A pipeline with 90% accuracy may hide a detector that fails 40% of the time but is compensated by downstream filtering.
• Avoid: Applying white-box perturbations (adversarial gradients) across module boundaries. SETA is designed for black-box, input-level perturbations that propagate naturally through the pipeline.

Error Handling

• Trace alignment failure: If the perturbed input causes a completely different execution path (e.g., detector finds zero objects so no classifier runs), handle gracefully by scoring all skipped downstream modules as S_i = 0 and recording the routing divergence.
• Empty test set for a component: Some conditional modules may be invoked on very few test inputs. Flag components with fewer than 30 activations as having low-confidence FC scores. Consider stratified sampling to ensure coverage.
• Non-deterministic models: If a module produces different outputs on repeated runs of the same input, run multiple reference traces and use majority-vote outputs as the reference. Alternatively, set random seeds and disable dropout at test time.
• Perturbation incompatibility: If a perturbation produces out-of-domain inputs (e.g., negative pixel values), clip or normalize before feeding to the pipeline. Document which perturbations are valid for which input domains.

Limitations

• SETA requires black-box access to each component's inputs and outputs. Fully opaque API-only services where you cannot intercept intermediate representations need proxy wrappers to capture I/O.
• The framework assumes you can define meaningful metamorphic relations for each component. For generative or open-ended models (LLMs, diffusion models), defining binary pass/fail MRs is non-trivial and may require embedding-based similarity thresholds.
• FC scores identify which component fails, not why it fails. Root-cause analysis within a component still requires component-specific debugging.
• Statistical significance requires sufficient test data. With fewer than ~100 system failures, FC estimates have high variance. Use confidence intervals when reporting.
• SETA doubles inference cost (clean + perturbed runs per input). For large test sets or expensive models, budget computation accordingly or use sampling.

Reference

Paper: SETA: Statistical Fault Attribution for Compound AI Systems — Chowdhury & D'Souza, CAIN 2026. Look for: the formal state-transition model (Section 3), the FC_i formula and Algorithm 3 (Section 4), and the rail inspection case study with per-perturbation robustness breakdowns (Section 5).