Use this Skill to design, validate, and deploy research surveys: Likert scale construction, attention checks, pilot testing, ICC reliability, and Qualtrics/LimeSurvey export.
Use this Skill to design, validate, and deploy research surveys: Likert scale construction, attention checks, pilot testing, ICC reliability, and Qualtrics/LimeSurvey export.
Survey Design — Likert Scales, Reliability, and Quality Control
TL;DR — Build psychometrically sound research surveys: construct and
balance Likert scales, embed attention checks, run cognitive-interview pilot
testing, compute ICC and Cronbach's alpha with pingouin, detect straightliners
and speedy responders, and export clean data for Qualtrics or LimeSurvey.
When to Use This Skill
Use this Skill whenever you need to:
Design a new Likert-scale questionnaire or adapt an existing validated scale
Choose between 5-point, 7-point, forced-choice, and visual analogue formats
Write and embed instructed-response items (IRIs) and bogus attention checks
Plan a pilot study (cognitive interviews + quantitative pilot) before main data collection
Calculate intraclass correlation coefficients (ICC) for inter-rater or test-retest reliability
Compute Cronbach's alpha and item-total correlations to assess internal consistency
Filter low-quality responses (straight-lining, implausibly fast completion) before analysis
Export survey data from Qualtrics CSV format and apply recoding schemas
Task
When to apply
Scale construction
Designing a new multi-item measure
Reliability analysis
Validating a new instrument or adapted translation
Attention check filtering
Any online survey with monetary incentive
Pilot testing plan
Before main data collection begins
ICC calculation
Inter-rater agreement or test-retest studies
Qualtrics export processing
After data collection on Qualtrics platform
Background & Key Concepts
Likert Scale Construction
A Likert scale measures attitudes or perceptions through a set of items
sharing a common response format. Key design choices:
Choice
5-point
7-point
Reliability
Adequate for most purposes (α ≈ 0.70+)
Slightly higher discrimination
Respondent burden
Lower fatigue
Higher cognitive load
Use case
Short surveys, general populations
Nuanced attitude measurement
Fully labeled response options (labels on every point) produce more
reliable data than endpoint labeled options (labels only at extremes),
particularly in non-Western samples.
Forced choice (even number of points, no neutral midpoint) removes
acquiescence bias but frustrates respondents who are genuinely neutral.
Include a neutral midpoint unless you have a specific theoretical reason to
omit it.
Acquiescence Bias and Reverse Scoring
Acquiescence bias: the tendency to agree regardless of item content.
Mitigate by including 30–50 % reverse-scored items (negative wording).
Reverse-score during data cleaning: if the scale runs 1–5, reverse score =
(max + 1) − raw score = 6 − raw score.
Attention Check Item Types
Type
Description
Example
Instructed response item (IRI)
Explicit instruction to select a specific option
"To show you are paying attention, select 'Strongly agree'."
Bogus item
Asks about a fictional construct
"I often use a Markovian relaxation device when stressed."
Consistency check
Repeated item with minor rewording
High correlation expected between duplicates
Exclude respondents who fail ≥ 1 IRI or select the bogus item. Report the
exclusion rate in the Methods section.
Pilot Testing Workflow
Phase 1 — Cognitive interviews (n = 5–10)
Recruit participants similar to your target population
Use think-aloud protocol: ask participants to verbalize thoughts while answering
Probe: "What does this question mean to you?" "How did you decide on that answer?"
Revise items with comprehension or ambiguity issues
Phase 2 — Quantitative pilot (n = 30–50)
Administer full survey including attention checks
Calculate Cronbach's alpha (target ≥ 0.70 for exploratory, ≥ 0.80 for confirmatory)
Remove items with item-total correlation < 0.30
Check response distributions for floor/ceiling effects
Estimate completion time for main survey planning
ICC — Intraclass Correlation Coefficient
For inter-rater reliability or test-retest, use ICC(2,1) (two-way mixed,
absolute agreement, single measures):
ICC range
Interpretation
< 0.50
Poor
0.50–0.74
Moderate
0.75–0.89
Good
≥ 0.90
Excellent
Cite Koo & Mae (2016) when reporting ICC values.
Response Quality Indicators
Straight-lining: all responses within a block are identical. Flag if
standard deviation across block items = 0.
Speedy responding: completion time < 1/3 of the median time across
respondents signals insufficient attention.
For Qualtrics API access (optional automated export):
# Register at https://api.qualtrics.com/ to obtain a Data Center ID and API token# export QUALTRICS_API_TOKEN="<paste-your-key>"# export QUALTRICS_DATA_CENTER="<your-datacenter-id>"
python -c "import os; print(os.getenv('QUALTRICS_API_TOKEN', 'NOT SET'))"
Core Workflow
Step 1 — ICC Calculation with pingouin
import pandas as pd
import numpy as np
import pingouin as pg
from typing importLiteraldefcompute_icc(
data: pd.DataFrame,
raters_col: str,
targets_col: str,
ratings_col: str,
icc_type: str = "ICC2",
model_desc: str = "two-way mixed, absolute agreement",
) -> dict:
"""
Compute intraclass correlation coefficient using pingouin.
Args:
data: Long-format DataFrame with columns for rater, target, rating.
raters_col: Column name identifying the rater.
targets_col: Column name identifying the rated subject/item.
ratings_col: Column name containing the numeric rating.
icc_type: ICC type per Shrout & Fleiss (1979). Options:
'ICC1', 'ICC2', 'ICC3', 'ICC1k', 'ICC2k', 'ICC3k'.
model_desc: Descriptive label for the model.
Returns:
Dictionary with ICC estimate, 95 % CI, F-statistic, and p-value.
Example:
>>> icc_result = compute_icc(df_long, 'rater', 'participant', 'score')
>>> print(icc_result)
"""
icc_results = pg.intraclass_corr(
data=data,
raters=raters_col,
targets=targets_col,
ratings=ratings_col,
)
row = icc_results[icc_results["Type"] == icc_type].iloc[0]
result = {
"icc_type": icc_type,
"model": model_desc,
"icc": round(row["ICC"], 4),
"ci_lower": round(row["CI95%"][0], 4),
"ci_upper": round(row["CI95%"][1], 4),
"f_value": round(row["F"], 4),
"df1": int(row["df1"]),
"df2": int(row["df2"]),
"p_value": row["pval"],
"interpretation": _interpret_icc(row["ICC"]),
}
return result
def_interpret_icc(icc_val: float) -> str:
"""Map ICC value to qualitative interpretation (Koo & Mae, 2016)."""if icc_val < 0.50:
return"Poor"elif icc_val < 0.75:
return"Moderate"elif icc_val < 0.90:
return"Good"else:
return"Excellent"defsimulate_icc_dataset(
n_subjects: int = 30,
n_raters: int = 2,
true_icc: float = 0.85,
seed: int = 42,
) -> pd.DataFrame:
"""
Simulate a long-format rating dataset for ICC demonstration.
Args:
n_subjects: Number of rated subjects.
n_raters: Number of raters.
true_icc: Approximate ICC to target (controls signal-to-noise ratio).
seed: Random seed for reproducibility.
Returns:
Long-format DataFrame with columns: rater, participant, score.
"""
rng = np.random.default_rng(seed)
subject_effects = rng.normal(loc=50, scale=10 * np.sqrt(true_icc), size=n_subjects)
rows = []
for r inrange(1, n_raters + 1):
noise_sd = 10 * np.sqrt(1 - true_icc)
for s_idx, subject_mean inenumerate(subject_effects):
score = subject_mean + rng.normal(0, noise_sd)
score = float(np.clip(round(score), 1, 100))
rows.append({"rater": f"Rater_{r}", "participant": f"P{s_idx+1:03d}", "score": score})
return pd.DataFrame(rows)
if __name__ == "__main__":
df_long = simulate_icc_dataset(n_subjects=30, n_raters=2, true_icc=0.85)
result = compute_icc(df_long, "rater", "participant", "score")
print("ICC Results:")
for k, v in result.items():
print(f" {k}: {v}")
# Report line: ICC(2,1) = 0.87, 95% CI [0.75, 0.93], p < .001 (Good reliability)print(
f"\nReport: ICC(2,1) = {result['icc']:.2f}, "f"95% CI [{result['ci_lower']:.2f}, {result['ci_upper']:.2f}], "f"p = {result['p_value']:.3f} ({result['interpretation']} reliability)"
)
Step 2 — Straight-Lining and Attention Check Filtering Pipeline
import pandas as pd
import numpy as np
from typing importOptionaldefflag_straightliners(
df: pd.DataFrame,
item_cols: list[str],
id_col: str = "response_id",
sd_threshold: float = 0.0,
) -> pd.DataFrame:
"""
Flag respondents who gave identical responses across all scale items.
Args:
df: Wide-format response DataFrame.
item_cols: List of column names for scale items (all same Likert range).
id_col: Respondent identifier column.
sd_threshold: Responses with SD ≤ this threshold are flagged.
Use 0.0 for exact straight-lining; raise to 0.3 for near-SL.
Returns:
DataFrame with added column 'is_straightliner' (bool).
"""
df = df.copy()
row_sds = df[item_cols].apply(pd.to_numeric, errors="coerce").std(axis=1)
df["item_sd"] = row_sds
df["is_straightliner"] = row_sds <= sd_threshold
n_flagged = df["is_straightliner"].sum()
print(f"Straight-liners flagged: {n_flagged}/{len(df)} "f"({100*n_flagged/len(df):.1f}%)")
return df
defflag_speedy_responders(
df: pd.DataFrame,
duration_col: str,
threshold_fraction: float = 1 / 3,
) -> pd.DataFrame:
"""
Flag respondents whose survey duration is below a fraction of the median.
Args:
df: Response DataFrame with a duration column (seconds).
duration_col: Column name for total survey duration in seconds.
threshold_fraction: Fraction of median; default 1/3 per standard practice.
Returns:
DataFrame with added column 'is_speedy' (bool).
"""
df = df.copy()
median_duration = df[duration_col].median()
threshold = threshold_fraction * median_duration
df["is_speedy"] = df[duration_col] < threshold
n_flagged = df["is_speedy"].sum()
print(f"Speedy responders flagged: {n_flagged}/{len(df)} "f"(threshold: {threshold:.0f}s; median: {median_duration:.0f}s)")
return df
defapply_attention_check_filter(
df: pd.DataFrame,
iri_checks: list[dict],
bogus_cols: Optional[list[str]] = None,
) -> pd.DataFrame:
"""
Filter respondents who fail instructed-response items or select bogus items.
Args:
df: Response DataFrame.
iri_checks: List of dicts, each with keys:
'col' (column name), 'expected' (correct response value).
bogus_cols: List of column names for bogus items; any non-null/non-zero
selection counts as a failure.
Returns:
DataFrame with added column 'failed_attention_check' (bool).
"""
df = df.copy()
fail_mask = pd.Series(False, index=df.index)
for check in iri_checks:
col, expected = check["col"], check["expected"]
if col in df.columns:
fail_mask |= df[col] != expected
if bogus_cols:
for col in bogus_cols:
if col in df.columns:
fail_mask |= df[col].notna() & (df[col] != 0) & (df[col] != "")
df["failed_attention_check"] = fail_mask
n_failed = fail_mask.sum()
print(f"Attention check failures: {n_failed}/{len(df)} ({100*n_failed/len(df):.1f}%)")
return df
defrun_full_qc_pipeline(
df: pd.DataFrame,
item_cols: list[str],
duration_col: str,
iri_checks: list[dict],
bogus_cols: Optional[list[str]] = None,
reverse_score_cols: Optional[list[str]] = None,
scale_max: int = 5,
) -> tuple[pd.DataFrame, pd.DataFrame]:
"""
Run full data-quality pipeline: reverse scoring, straight-line detection,
speed detection, and attention check filtering.
Returns:
Tuple of (cleaned DataFrame, excluded DataFrame) with exclusion reasons.
"""
df = df.copy()
# Reverse scoringif reverse_score_cols:
for col in reverse_score_cols:
if col in df.columns:
df[col] = (scale_max + 1) - pd.to_numeric(df[col], errors="coerce")
df = flag_straightliners(df, item_cols)
df = flag_speedy_responders(df, duration_col)
df = apply_attention_check_filter(df, iri_checks, bogus_cols)
exclude_mask = (
df["is_straightliner"] | df["is_speedy"] | df["failed_attention_check"]
)
excluded = df[exclude_mask].copy()
cleaned = df[~exclude_mask].copy()
print(f"\nQC Summary: {len(cleaned)} retained, {len(excluded)} excluded "f"({100*len(excluded)/len(df):.1f}% exclusion rate)")
return cleaned, excluded
# ── Demo ──────────────────────────────────────────────────────────────────────if __name__ == "__main__":
rng = np.random.default_rng(123)
n = 200
item_names = [f"item_{i:02d}"for i inrange(1, 11)]
demo_data = {col: rng.integers(1, 6, size=n) for col in item_names}
demo_data["response_id"] = [f"R{i:04d}"for i inrange(n)]
demo_data["duration_sec"] = rng.integers(60, 900, size=n)
demo_data["iri_check"] = rng.choice([3, 1, 2, 4, 5], size=n, p=[0.85, 0.05, 0.04, 0.03, 0.03])
demo_data["bogus_markov"] = rng.choice([None, 1, 2], size=n, p=[0.93, 0.04, 0.03])
df_raw = pd.DataFrame(demo_data)
# Inject 5 straight-linersfor i inrange(5):
df_raw.loc[i, item_names] = 3
cleaned, excluded = run_full_qc_pipeline(
df=df_raw,
item_cols=item_names,
duration_col="duration_sec",
iri_checks=[{"col": "iri_check", "expected": 3}],
bogus_cols=["bogus_markov"],
reverse_score_cols=["item_03", "item_07"],
scale_max=5,
)
print(f"\nCleaned dataset shape: {cleaned.shape}")
Step 3 — Cronbach's Alpha and Item-Total Correlations
import pandas as pd
import numpy as np
from scipy.stats import pearsonr
import pingouin as pg
import matplotlib.pyplot as plt
defcompute_cronbach_alpha(
df: pd.DataFrame,
item_cols: list[str],
ci_level: float = 0.95,
) -> dict:
"""
Compute Cronbach's alpha with 95 % confidence interval using pingouin.
Args:
df: DataFrame containing item responses (numeric).
item_cols: List of item column names to include.
ci_level: Confidence interval level (default 0.95).
Returns:
Dictionary with alpha, CI, and interpretation.
"""
items = df[item_cols].apply(pd.to_numeric, errors="coerce").dropna()
result = pg.cronbach_alpha(data=items, ci=ci_level)
alpha_val = result[0]
ci = result[1]
interpretation = (
"Unacceptable"if alpha_val < 0.60else"Questionable"if alpha_val < 0.70else"Acceptable"if alpha_val < 0.80else"Good"if alpha_val < 0.90else"Excellent"
)
return {
"cronbach_alpha": round(alpha_val, 4),
"ci_lower": round(ci[0], 4),
"ci_upper": round(ci[1], 4),
"n_items": len(item_cols),
"n_respondents": len(items),
"interpretation": interpretation,
}
defcompute_item_total_correlations(
df: pd.DataFrame,
item_cols: list[str],
) -> pd.DataFrame:
"""
Compute corrected item-total correlations (CITC) for all scale items.
CITC = Pearson correlation between the item and the sum of all OTHER items.
Items with CITC < 0.30 are flagged for removal.
Args:
df: DataFrame with numeric item responses.
item_cols: List of item column names.
Returns:
DataFrame with columns: item, citc, alpha_if_deleted, flag.
"""
items_df = df[item_cols].apply(pd.to_numeric, errors="coerce").dropna()
rows = []
for col in item_cols:
rest_cols = [c for c in item_cols if c != col]
rest_sum = items_df[rest_cols].sum(axis=1)
r, p = pearsonr(items_df[col], rest_sum)
# Alpha if deleted
alpha_if_del = pg.cronbach_alpha(data=items_df[rest_cols])[0]
rows.append({
"item": col,
"citc": round(r, 4),
"alpha_if_deleted": round(alpha_if_del, 4),
"flag": "REMOVE"if r < 0.30else"",
})
return pd.DataFrame(rows).sort_values("citc", ascending=False)
defplot_item_total_correlations(
citc_df: pd.DataFrame,
output_path: str = "item_total_correlations.png",
) -> None:
"""Plot CITC bar chart with the 0.30 threshold line."""
fig, ax = plt.subplots(figsize=(10, 5))
colors = ["#D32F2F"if r < 0.30else"#1976D2"for r in citc_df["citc"]]
ax.barh(citc_df["item"], citc_df["citc"], color=colors)
ax.axvline(0.30, color="red", linestyle="--", linewidth=1.5, label="Threshold (0.30)")
ax.set_xlabel("Corrected Item-Total Correlation")
ax.set_title("Item-Total Correlations (red = below threshold)")
ax.legend()
plt.tight_layout()
plt.savefig(output_path, dpi=150)
print(f"Item-total correlation chart saved to {output_path}")
if __name__ == "__main__":
rng = np.random.default_rng(42)
n_resp, n_items = 150, 10
item_cols = [f"item_{i:02d}"for i inrange(1, n_items + 1)]
# Create correlated item data (factor structure)
latent = rng.normal(0, 1, size=n_resp)
demo = {col: np.clip((latent + rng.normal(0, 0.8, n_resp) * 2 + 3).round(), 1, 5)
for col in item_cols}
df_items = pd.DataFrame(demo)
alpha_result = compute_cronbach_alpha(df_items, item_cols)
print("Cronbach's Alpha Results:")
for k, v in alpha_result.items():
print(f" {k}: {v}")
citc_df = compute_item_total_correlations(df_items, item_cols)
print("\nItem-Total Correlations:")
print(citc_df.to_string(index=False))
items_to_remove = citc_df[citc_df["flag"] == "REMOVE"]["item"].tolist()
if items_to_remove:
print(f"\nRecommended for removal: {items_to_remove}")
else:
print("\nAll items meet the CITC ≥ 0.30 threshold.")
plot_item_total_correlations(citc_df)
Advanced Usage
Qualtrics CSV Export Column Conventions
When downloading Qualtrics data as CSV, the default export has two header rows:
the first contains column labels (question text), the second contains encoded
question IDs (e.g., Q1, Q1_1). Use skiprows=[1] after loading the second
header row to parse cleanly:
import pandas as pd
defload_qualtrics_csv(
path: str,
skip_preview_row: bool = True,
) -> pd.DataFrame:
"""
Load a Qualtrics CSV export with proper header handling.
The Qualtrics default export has three header rows:
row 0: column labels (human-readable)
row 1: question text (verbose)
row 2+: data
Args:
path: File path to the Qualtrics CSV export.
skip_preview_row: If True, skip row 1 (question text row).
Returns:
DataFrame with row-0 column labels and data rows only.
"""
headers = pd.read_csv(path, nrows=0).columns.tolist()
df = pd.read_csv(path, skiprows=[1, 2] if skip_preview_row else [1], header=0)
df.columns = headers
# Drop Qualtrics metadata columns
meta_cols = ["StartDate", "EndDate", "Status", "IPAddress", "Progress",
"Duration (in seconds)", "Finished", "RecordedDate",
"ResponseId", "RecipientLastName", "RecipientFirstName",
"RecipientEmail", "ExternalReference", "LocationLatitude",
"LocationLongitude", "DistributionChannel", "UserLanguage"]
data_cols = [c for c in df.columns if c notin meta_cols]
return df[["ResponseId", "Duration (in seconds)"] + data_cols]
Test-Retest Reliability
For scales administered twice (Time 1 and Time 2):
import pingouin as pg
import pandas as pd
deftest_retest_reliability(
df_t1: pd.DataFrame,
df_t2: pd.DataFrame,
item_cols: list[str],
id_col: str = "participant_id",
) -> dict:
"""
Compute test-retest ICC(2,1) for composite scale scores.
Args:
df_t1: Time-1 DataFrame with participant ID and item columns.
df_t2: Time-2 DataFrame with participant ID and item columns.
item_cols: Scale item columns to sum into composite score.
id_col: Participant identifier column.
Returns:
Dictionary with ICC results and Pearson r.
"""
scores_t1 = df_t1.set_index(id_col)[item_cols].sum(axis=1).rename("T1")
scores_t2 = df_t2.set_index(id_col)[item_cols].sum(axis=1).rename("T2")
merged = pd.concat([scores_t1, scores_t2], axis=1).dropna().reset_index()
long_df = merged.melt(id_vars=id_col, value_vars=["T1", "T2"],
var_name="time", value_name="score")
icc = pg.intraclass_corr(data=long_df, raters="time",
targets=id_col, ratings="score")
icc2_row = icc[icc["Type"] == "ICC2"].iloc[0]
r = merged[["T1", "T2"]].corr().iloc[0, 1]
return {
"icc2_1": round(icc2_row["ICC"], 4),
"ci_lower": round(icc2_row["CI95%"][0], 4),
"ci_upper": round(icc2_row["CI95%"][1], 4),
"pearson_r": round(r, 4),
"n_pairs": len(merged),
}
Troubleshooting
Problem
Likely cause
Fix
pingouin.intraclass_corr returns NaN
Missing values in ratings column
Drop rows with missing data before calling the function
Cronbach's alpha is negative
Reverse-coded items not reversed
Run reverse-scoring step before computing alpha
ICC lower bound is negative
Very small sample (n < 20)
Increase pilot sample; report wide CI transparently
High exclusion rate (> 30 %)
Overly strict attention checks
Review IRI wording; consider one IRI instead of two
Floor/ceiling effects on Likert items
Response scale too narrow or extreme population
Switch from 5-point to 7-point; check sample frame
