artifact-detection

name	artifact-detection
description	Detect annotation artifacts and shortcuts in benchmarks
execution	tactic

Artifact Detection Tactic

Systematically probe benchmarks for annotation artifacts, dataset shortcuts, and spurious correlations that allow models to achieve high scores without the intended capability.

Stages

Stage 1: Hypothesis-Only Baseline Test

Search literature for evidence that partial-input baselines achieve unexpectedly high performance:

• Hypothesis-only baselines (NLI without premise)
• Question-only baselines (QA without context)
• Label-word frequency baselines
• Majority-class and surface-pattern baselines

Search queries: "[benchmark] annotation artifacts", "[benchmark] hypothesis only", "[benchmark] spurious correlations", "[benchmark] dataset bias"

If published partial-input results exist, record performance gap between partial and full input. Gap < 10 points above random indicates severe artifacts.

Stage 2: Contrast Set Construction

Identify whether contrast sets or adversarial evaluations exist:

• Search for "[benchmark] contrast sets", "[benchmark] adversarial examples"
• Check if CheckList-style behavioral tests have been applied
• Look for counterfactual data augmentation studies

Record performance drops on contrast sets. Drops > 20 points indicate reliance on surface patterns.

Stage 3: Format Manipulation Probes

Search for evidence of format sensitivity:

• Prompt template sensitivity studies
• Label name/ordering effects
• Verbalization effects in classification
• Input length correlations with labels

Record whether minor format changes cause disproportionate score changes.

Stage 4: Conclusion Synthesis

Aggregate evidence into artifact severity assessment:

Severity	Criteria
Critical	Partial-input baseline within 5 points of full model
High	Contrast set drop >20 points OR format sensitivity >10 points
Medium	Known artifacts documented but partial mitigations exist
Low	Minor artifacts, full-input still required for high performance
None	No evidence of artifacts (may indicate insufficient probing)

Output

artifact_report:
  benchmark: string
  overall_severity: critical|high|medium|low|none
  partial_input_baselines:
    - input_type: string  # e.g., "hypothesis only"
      performance: float
      full_model_performance: float
      gap: float
      source: string
  contrast_set_results:
    - contrast_set: string
      original_performance: float
      contrast_performance: float
      drop: float
      source: string
  format_sensitivity:
    - manipulation: string
      score_range: string
      source: string
  shortcuts_identified:
    - shortcut: string
      mechanism: string
      exploitability: high|medium|low
  evidence_completeness: thorough|partial|minimal

Yield Report

Metric	Minimum
Literature sources checked	5
Artifact categories probed	3
Evidence items collected	4
Severity classification produced	1