interobserver-agreement

# Interobserver Agreement (IOA) Interobserver agreement quantifies the extent to which two or more independent observers agree on the occurrence of a target behavior when simultaneously observing the same events. IOA is a necessary condition—though not a sufficient one—for establishing measurement reliability. ## Purpose - Demonstrates that the behavioral definition is clear and the measurement system is sound. - Required for publication in behavior analytic journals and for rigorous clinical practice. - Low IOA signals a problem with the definition, the measurement system, observer training, or all three. - IOA is not a measure of accuracy (validity)—two observers can agree and both be wrong. ## IOA Methods ### Total Count IOA The simplest method. Compare the total counts from both observers for the entire session. **Formula**: (Smaller count / Larger count) × 100 **When to use**: Quick check when the behavior occurs at a low-to-moderate rate and the session is brief. **Limitation**: Can inflate agreement when behavior rates are high—observers may record the same total but disagree on when specific instances occurred. ### Exact Count per Interval Divide the session into intervals. Compare the counts within each interval. **Formula**: Calculate agreement for each interval (smaller/larger × 100), then average across all intervals. **When to use**: When the behavior occurs at moderate-to-high rates and you need more precision than total count IOA. ### Trial-by-Trial (Point-by-Point) IOA Compare observer records on each discrete trial. Score each trial as "agree" or "disagree." **Formula**: (Agreements / [Agreements + Disagreements]) × 100 **When to use**: Discrete trial training, structured tasks with clear trial boundaries. ### Interval-by-Interval IOA For interval recording data. Compare each interval and score as agree or disagree. **Formula**: (Intervals with agreement / Total intervals) × 100 **When to use**: Whole-interval, partial-interval, or momentary time sampling data. **Limitation**: Can inflate agreement when the behavior occurs at very high or very low rates (base rate problem). ### Scored-Interval IOA Only consider intervals where at least one observer scored the behavior as occurring. **Formula**: (Intervals both scored occurrence / Intervals at least one scored occurrence) × 100 **When to use**: When the behavior is infrequent (low base rate). Standard interval IOA inflates agreement by counting all the "no-no" agreements. ### Unscored-Interval IOA Only consider intervals where at least one observer scored the behavior as not occurring. **Formula**: (Intervals both scored non-occurrence / Intervals at least one scored non-occurrence) × 100 **When to use**: When the behavior is very frequent (high base rate). Standard interval IOA inflates agreement by counting all the "yes-yes" agreements. ### Occurrence IOA Synonym for scored-interval IOA in some texts. The more conservative measure when behavior is infrequent. ### Nonoccurrence IOA Synonym for unscored-interval IOA in some texts. The more conservative measure when behavior is frequent. ## Selecting the Appropriate IOA Method | Behavior Characteristic | Recommended IOA Method | |------------------------|----------------------| | Discrete trials with clear boundaries | Trial-by-trial | | Low-rate behavior with interval recording | Scored-interval | | High-rate behavior with interval recording | Unscored-interval | | Moderate-rate behavior with interval recording | Interval-by-interval | | Event recording, low-to-moderate rate | Total count or exact count per interval | | Duration recording | Total duration or mean duration per occurrence | ## Standards - **Minimum acceptable agreement**: 80% (some journals and standards require 90%). - **Frequency of assessment**: Collect IOA data on a minimum of 20–33% of all sessions. - **Across conditions**: IOA must be assessed in all experimental conditions or treatment phases, not just baseline. - **Report**: State the IOA method used, the percentage of sessions assessed, the range, and the mean. ## Duration-Based IOA When using duration recording, IOA can be calculated by: - **Total duration**: (Shorter total / Longer total) × 100 - **Mean duration per occurrence**: Compare average episode durations. - **Interval-based duration**: Compare duration within intervals. ## Training Observers to Criterion ### Procedure 1. Provide the operational definition with examples and non-examples. 2. Practice together on training videos—discuss each scoring decision. 3. Practice independently on the same training videos. 4. Calculate IOA after each independent practice session. 5. Criterion: ≥80% IOA across 3 consecutive independent practice sessions before collecting study or clinical data. 6. If criterion is not met, retrain on steps 2–4 with new exemplars. ### Ongoing Calibration - Conduct periodic reliability checks (unannounced when possible). - Review data together after IOA checks to resolve disagreements. - Recalibrate if IOA drops below 80% on any check. ## Addressing Low IOA ### Diagnostic Steps 1. **Review the operational definition**: Is it sufficiently specific? Are the onset/offset criteria clear? 2. **Check the measurement system**: Is the interval length appropriate? Are data sheets well-designed? 3. **Assess observer training**: Did observers meet criterion? Has drift occurred? 4. **Consider behavior characteristics**: Very brief behaviors, high-rate behaviors, and behaviors with ambiguous boundaries are inherently harder to score reliably. 5. **Evaluate environmental factors**: Can observers see/hear the behavior adequately from their positions? ### Corrective Actions - Refine the operational definition with clearer boundaries. - Simplify the measurement system (e.g., longer intervals, fewer simultaneous targets). - Retrain observers with new exemplar videos. - Change observer positioning for better visual access. - Add onset/offset rules (e.g., "A new instance begins after a 3-second pause"). ## Reporting IOA in Publications and Clinical Records **Minimum reporting elements**: - IOA method used and rationale for selection. - Percentage of sessions with IOA assessment. - Mean IOA across all assessed sessions. - Range (lowest and highest IOA values). - Whether IOA was assessed across all conditions/phases. **Example**: "Interobserver agreement was assessed on 30% of sessions across all phases using trial-by-trial IOA. Mean agreement was 94% (range: 87%–100%)." ## Key References - Cooper, J. O., Heron, T. E., & Heward, W. L. (2020). *Applied Behavior Analysis* (3rd ed.). Pearson. - Kazdin, A. E. (2010). *Single-Case Research Designs* (2nd ed.). Oxford University Press. - Ledford, J. R., Lane, J. D., & Severini, K. E. (2018). Systematic use of visual analysis for assessing outcomes in single case design studies. *Brain Impairment*, 19, 4–17.