interobserver-agreement

name	interobserver-agreement
description	Use when assessing the reliability of behavioral measurement through interobserver agreement methods, calculating IOA, and troubleshooting low agreement.

Interobserver Agreement (IOA)

Interobserver agreement quantifies the extent to which two or more independent observers agree on the occurrence of a target behavior when simultaneously observing the same events. IOA is a necessary condition—though not a sufficient one—for establishing measurement reliability.

Purpose

• Demonstrates that the behavioral definition is clear and the measurement system is sound.
• Required for publication in behavior analytic journals and for rigorous clinical practice.
• Low IOA signals a problem with the definition, the measurement system, observer training, or all three.
• IOA is not a measure of accuracy (validity)—two observers can agree and both be wrong.

IOA Methods

Total Count IOA

The simplest method. Compare the total counts from both observers for the entire session.

Formula: (Smaller count / Larger count) × 100

When to use: Quick check when the behavior occurs at a low-to-moderate rate and the session is brief.

Limitation: Can inflate agreement when behavior rates are high—observers may record the same total but disagree on when specific instances occurred.

Exact Count per Interval

Divide the session into intervals. Compare the counts within each interval.

Formula: Calculate agreement for each interval (smaller/larger × 100), then average across all intervals.

When to use: When the behavior occurs at moderate-to-high rates and you need more precision than total count IOA.

Trial-by-Trial (Point-by-Point) IOA

Compare observer records on each discrete trial. Score each trial as "agree" or "disagree."

Formula: (Agreements / [Agreements + Disagreements]) × 100

When to use: Discrete trial training, structured tasks with clear trial boundaries.

Interval-by-Interval IOA

For interval recording data. Compare each interval and score as agree or disagree.

Formula: (Intervals with agreement / Total intervals) × 100

When to use: Whole-interval, partial-interval, or momentary time sampling data.

Limitation: Can inflate agreement when the behavior occurs at very high or very low rates (base rate problem).

Scored-Interval IOA

Only consider intervals where at least one observer scored the behavior as occurring.

Formula: (Intervals both scored occurrence / Intervals at least one scored occurrence) × 100

When to use: When the behavior is infrequent (low base rate). Standard interval IOA inflates agreement by counting all the "no-no" agreements.

Unscored-Interval IOA

Only consider intervals where at least one observer scored the behavior as not occurring.

Formula: (Intervals both scored non-occurrence / Intervals at least one scored non-occurrence) × 100

When to use: When the behavior is very frequent (high base rate). Standard interval IOA inflates agreement by counting all the "yes-yes" agreements.

Occurrence IOA

Synonym for scored-interval IOA in some texts. The more conservative measure when behavior is infrequent.

Nonoccurrence IOA

Synonym for unscored-interval IOA in some texts. The more conservative measure when behavior is frequent.

Selecting the Appropriate IOA Method

Behavior Characteristic	Recommended IOA Method
Discrete trials with clear boundaries	Trial-by-trial
Low-rate behavior with interval recording	Scored-interval
High-rate behavior with interval recording	Unscored-interval
Moderate-rate behavior with interval recording	Interval-by-interval
Event recording, low-to-moderate rate	Total count or exact count per interval
Duration recording	Total duration or mean duration per occurrence

Standards

• Minimum acceptable agreement: 80% (some journals and standards require 90%).
• Frequency of assessment: Collect IOA data on a minimum of 20–33% of all sessions.
• Across conditions: IOA must be assessed in all experimental conditions or treatment phases, not just baseline.
• Report: State the IOA method used, the percentage of sessions assessed, the range, and the mean.

Duration-Based IOA

When using duration recording, IOA can be calculated by:

• Total duration: (Shorter total / Longer total) × 100
• Mean duration per occurrence: Compare average episode durations.
• Interval-based duration: Compare duration within intervals.

Training Observers to Criterion

Procedure

1. Provide the operational definition with examples and non-examples.
2. Practice together on training videos—discuss each scoring decision.
3. Practice independently on the same training videos.
4. Calculate IOA after each independent practice session.
5. Criterion: ≥80% IOA across 3 consecutive independent practice sessions before collecting study or clinical data.
6. If criterion is not met, retrain on steps 2–4 with new exemplars.

Ongoing Calibration

• Conduct periodic reliability checks (unannounced when possible).
• Review data together after IOA checks to resolve disagreements.
• Recalibrate if IOA drops below 80% on any check.

Addressing Low IOA

Diagnostic Steps

1. Review the operational definition: Is it sufficiently specific? Are the onset/offset criteria clear?
2. Check the measurement system: Is the interval length appropriate? Are data sheets well-designed?
3. Assess observer training: Did observers meet criterion? Has drift occurred?
4. Consider behavior characteristics: Very brief behaviors, high-rate behaviors, and behaviors with ambiguous boundaries are inherently harder to score reliably.
5. Evaluate environmental factors: Can observers see/hear the behavior adequately from their positions?

Corrective Actions

• Refine the operational definition with clearer boundaries.
• Simplify the measurement system (e.g., longer intervals, fewer simultaneous targets).
• Retrain observers with new exemplar videos.
• Change observer positioning for better visual access.
• Add onset/offset rules (e.g., "A new instance begins after a 3-second pause").

Reporting IOA in Publications and Clinical Records

Minimum reporting elements:

• IOA method used and rationale for selection.
• Percentage of sessions with IOA assessment.
• Mean IOA across all assessed sessions.
• Range (lowest and highest IOA values).
• Whether IOA was assessed across all conditions/phases.

Example: "Interobserver agreement was assessed on 30% of sessions across all phases using trial-by-trial IOA. Mean agreement was 94% (range: 87%–100%)."

Key References

• Cooper, J. O., Heron, T. E., & Heward, W. L. (2020). Applied Behavior Analysis (3rd ed.). Pearson.
• Kazdin, A. E. (2010). Single-Case Research Designs (2nd ed.). Oxford University Press.
• Ledford, J. R., Lane, J. D., & Severini, K. E. (2018). Systematic use of visual analysis for assessing outcomes in single case design studies. Brain Impairment, 19, 4–17.