research-data-labeling

name	research-data-labeling
description	Method for labeling/classifying rows of a CSV using EDSL survey objects with iterative sampling, user approval, and analysis. Use when the user wants to classify, tag, or extract structured labels from tabular data using LLMs.
tags	["methods","data-collection","quantitative"]

Data Labeling with EDSL

Use this workflow when a user has a CSV and wants to classify, label, or extract information from each row using LLMs. Each CSV row becomes an EDSL scenario; a survey asks labeling questions about each row; results are exported as a labeled CSV with an analysis report.

Overview

The data-labeling workflow has seven steps:

1. Ingest the CSV as a ScenarioList
2. Design a labeling survey (with skip logic if needed)
3. Sample a small subset and run the survey on it
4. Iterate on the survey design with user feedback
5. Run the full dataset
6. Export labeled results as a CSV
7. Analyze and write a report

This follows the standard EDSL study structure. The study directory uses the normal scaffold from skills/workflow-file-layout/scripts/create_study_project.py, with the user's CSV placed in data/uploaded/.

---

Step 1: Ingest the CSV

Place the user's CSV in data/uploaded/ and load it as a ScenarioList.

from edsl import ScenarioList

scenario_list = ScenarioList.from_csv("data/uploaded/job_posts.csv")

Alternative using FileStore:

from edsl import FileStore

scenario_list = FileStore(path="data/uploaded/job_posts.csv").to_scenario_list()

Inspect the data

Before designing questions, understand what you have:

print(f"Rows: {len(scenario_list)}")
print(f"Columns: {list(scenario_list[0].keys())}")

# Preview a few rows
for s in scenario_list[:3]:
    print(s)

Select or rename columns

If the CSV has many columns, you may only need a subset. Filter to the columns your questions will reference — every scenario field must appear in at least one question_text.

# Keep only the columns referenced in questions
scenario_list = ScenarioList(
    [s.select("title", "description") for s in scenario_list]
)

IMPORTANT: Every field in the ScenarioList must be referenced in at least one question's question_text using {{ scenario.field_name }} syntax, or EDSL raises a JobsCompatibilityError.

---

Step 2: Design the labeling survey

Choose question types based on the labeling task. Common patterns:

Binary classification

from edsl import Survey, QuestionYesNo

q_mentions_ai = QuestionYesNo(
    question_name="mentions_ai",
    question_text=(
        "Does the following job post mention the use of AI tools?\n\n"
        "Title: {{ scenario.title }}\n"
        "Description: {{ scenario.description }}"
    )
)

survey = Survey([q_mentions_ai])

Multi-class classification

from edsl import QuestionMultipleChoice

q_category = QuestionMultipleChoice(
    question_name="category",
    question_text=(
        "Classify the following product review into one category.\n\n"
        "Review: {{ scenario.review_text }}"
    ),
    question_options=["Positive", "Negative", "Neutral", "Mixed"]
)

Confidence scoring: For any classification question, you can extract per-option probabilities by adding use_code=True, include_comment=False to the question and using an OpenAI model with logprobs=True, temperature=1. See skills/methods-logprob-confidence/SKILL.md for the full pattern.

Extraction

from edsl import QuestionList

q_tools = QuestionList(
    question_name="ai_tools_mentioned",
    question_text=(
        "List all AI tools mentioned in this job post. "
        "Return an empty list if none.\n\n"
        "Title: {{ scenario.title }}\n"
        "Description: {{ scenario.description }}"
    )
)

Combined: screening + conditional follow-up (skip logic)

This is the most common data-labeling pattern. Ask a screening question first, then only ask the follow-up if the answer warrants it.

from edsl import Survey, QuestionYesNo, QuestionList, QuestionFreeText

q_mentions_ai = QuestionYesNo(
    question_name="mentions_ai",
    question_text=(
        "Does the following job post mention the use of AI tools?\n\n"
        "Title: {{ scenario.title }}\n"
        "Description: {{ scenario.description }}"
    )
)

q_which_tools = QuestionList(
    question_name="which_tools",
    question_text=(
        "Which specific AI tools are mentioned in this job post?\n\n"
        "Title: {{ scenario.title }}\n"
        "Description: {{ scenario.description }}"
    )
)

q_how_used = QuestionFreeText(
    question_name="how_used",
    question_text=(
        "Briefly describe how AI tools are used or expected to be used "
        "according to this job post.\n\n"
        "Title: {{ scenario.title }}\n"
        "Description: {{ scenario.description }}"
    )
)

survey = (
    Survey([q_mentions_ai, q_which_tools, q_how_used])
    .add_skip_rule("which_tools", "{{ mentions_ai.answer }} == 'No'")
    .add_skip_rule("how_used", "{{ mentions_ai.answer }} == 'No'")
)

Multi-stage labeling with navigation

For more complex labeling with branching paths:

from edsl import QuestionMultipleChoice, QuestionFreeText, Survey

q_type = QuestionMultipleChoice(
    question_name="post_type",
    question_text="What type of posting is this?\n\n{{ scenario.text }}",
    question_options=["Job posting", "Advertisement", "News article", "Other"]
)

q_job_details = QuestionFreeText(
    question_name="job_details",
    question_text="Summarize the key requirements of this job posting.\n\n{{ scenario.text }}"
)

q_other_details = QuestionFreeText(
    question_name="other_details",
    question_text="Briefly describe what this posting is about.\n\n{{ scenario.text }}"
)

survey = (
    Survey([q_type, q_job_details, q_other_details])
    .add_skip_rule("job_details", "{{ post_type.answer }} != 'Job posting'")
    .add_skip_rule("other_details", "{{ post_type.answer }} == 'Job posting'")
)

---

Step 3: Sample and preview

Before running the full dataset, test on a small random sample.

labeling sample --n 5
labeling plan sample --latest
# After reviewing the plan, run the returned command:
ep run edsl_jobs/job_a/sample_jobs.ep --output data/results.sample.ep
labeling record run --kind sample --results data/results.sample.ep
labeling preview latest --human

What to check in the preview

• Do the labels make sense for the given rows?
• Are skip rules firing correctly (skipped answers show as None)?
• Are free-text extractions capturing the right information?
• Are the question options comprehensive enough?

Present the sample results to the user via AskUserQuestion and ask:

• Do the labels look correct?
• Should any question wording or options change?
• Is the skip logic behaving as expected?

---

Step 4: Iterate

Based on user feedback, adjust the survey and build a fresh sample job:

labeling validate
labeling generate --replace-generated
labeling sample --n 5
labeling plan sample --latest

Common adjustments:

• Add or remove answer options
• Refine question wording for clarity
• Add skip rules for edge cases
• Add an "Other" or "N/A" option
• Split a question into multiple more specific questions

Repeat until the user approves the labeling quality.

---

Step 5: Run the full dataset

After user approval, ask Labeling for the full-run plan. Labeling builds the Jobs artifact; ep exclusively owns execution.

Cost check: Before running, confirm with the user. State the number of rows and estimated cost if known. This is required by the "no silent spending" constraint.

labeling plan full
# After explicit approval, run the returned command:
ep run edsl_jobs/job_a/jobs.ep --output data/results.ep
labeling record run --kind full --results data/results.ep

Note: For data-labeling jobs there is typically no agent_list — the default agent is used. If the user wants labels from a specific persona, add agents as usual.

---

Step 6: Export labeled CSV

Create an export script in analysis/ that merges scenario data with labels.

# analysis/export_labeled_csv.py
from pathlib import Path
from edsl import Results

STUDY_ROOT = Path(__file__).resolve().parent.parent
results = Results.load(str(STUDY_ROOT / "data" / "results.ep"))

# Select scenario fields and answer fields
df = results.select("scenario.*", "answer.*").to_pandas()

# Clean column names: remove "scenario." and "answer." prefixes
df.columns = [c.replace("scenario.", "").replace("answer.", "label_") for c in df.columns]

output_path = STUDY_ROOT / "data" / "cooked" / "labeled_results.csv"
output_path.parent.mkdir(parents=True, exist_ok=True)
df.to_csv(output_path, index=False)
print(f"Exported {len(df)} labeled rows to {output_path}")

Makefile target:

data/cooked/labeled_results.csv: analysis/export_labeled_csv.py data/results.ep
	python analysis/export_labeled_csv.py

export: data/cooked/labeled_results.csv

---

Step 7: Analyze and report

Follow the standard study conventions. Create analysis scripts in analysis/, with outputs going to writeup/plots/ and writeup/tables/.

Typical analyses for labeled data

Label distribution — How many rows got each label?

# analysis/plot_label_distribution.py
from pathlib import Path
from edsl import Results
import matplotlib.pyplot as plt

STUDY_ROOT = Path(__file__).resolve().parent.parent
results = Results.load(str(STUDY_ROOT / "data" / "results.ep"))

labels = results.select("answer.mentions_ai").to_list()
counts = {}
for label in labels:
    counts[label] = counts.get(label, 0) + 1

plt.figure(figsize=(8, 5))
plt.bar(counts.keys(), counts.values())
plt.title("Label Distribution: Mentions AI Tools")
plt.ylabel("Count")
plt.tight_layout()

output = STUDY_ROOT / "writeup" / "plots" / "label_distribution.png"
output.parent.mkdir(parents=True, exist_ok=True)
plt.savefig(output, dpi=150)
print(f"Saved {output}")

Cross-tabulation — How do labels relate to other columns?

# analysis/table_cross_tab.py
import pandas as pd
from pathlib import Path
from edsl import Results

STUDY_ROOT = Path(__file__).resolve().parent.parent
results = Results.load(str(STUDY_ROOT / "data" / "results.ep"))

df = results.select("scenario.department", "answer.mentions_ai").to_pandas()
cross_tab = pd.crosstab(df["scenario.department"], df["answer.mentions_ai"])

output = STUDY_ROOT / "writeup" / "tables" / "cross_tab_department.csv"
output.parent.mkdir(parents=True, exist_ok=True)
cross_tab.to_csv(output)
print(f"Saved {output}")

Example rows per label — Show representative examples for each label category.

Model agreement — If multiple models were used, compare label consistency across models.

Makefile integration

.PHONY: data export PLOTS TABLES report all

data:
	ep run edsl_jobs/job_a/jobs.ep --output data/results.ep

export: data/cooked/labeled_results.csv

data/cooked/labeled_results.csv: analysis/export_labeled_csv.py data/results.ep
	python analysis/export_labeled_csv.py

PLOTS: writeup/plots/label_distribution.png

writeup/plots/label_distribution.png: analysis/plot_label_distribution.py data/results.ep
	python analysis/plot_label_distribution.py

TABLES: writeup/tables/cross_tab_department.csv

writeup/tables/cross_tab_department.csv: analysis/table_cross_tab.py data/results.ep
	python analysis/table_cross_tab.py

writeup/report.html: writeup/report.md PLOTS TABLES
	cd writeup && pandoc report.md -o report.html --standalone --embed-resources --css=report.css

writeup/report.pdf: writeup/report.md writeup/report_header.tex writeup/fix-table-widths.lua PLOTS TABLES
	cd writeup && pandoc report.md -o report.pdf --pdf-engine=xelatex --include-in-header=report_header.tex --lua-filter=fix-table-widths.lua

report: writeup/report.html writeup/report.pdf

all: data export PLOTS TABLES report

Report structure

Before writing the final report, run labeling report context and optionally labeling report template. These are bounded analytical inputs. The calling report agent—not Labeling—owns writeup/report.md, branding, and final audience framing.

The report in writeup/report.md should follow the standard format:

---
title: "Data Labeling: [Topic]"
date: YYYY-MM-DD
---

## Summary

Brief overview of the labeling task, dataset, and key findings.

## Dataset

- Source: [description of the CSV and where it came from]
- Rows: N
- Columns used for labeling: [list]

## Labeling Design

- Survey questions and their purpose
- Skip logic rules
- Models used

## Results

### Label Distribution

[Table and/or chart showing how labels are distributed]

### Cross-Tabulations

[How labels relate to other variables in the data]

### Example Rows

[Representative examples for each label category]

## Key Findings

[2-3 paragraphs synthesizing what the labeling revealed about the data]

## Files Generated

| File | Description |
|------|-------------|
| `data/cooked/labeled_results.csv` | Full labeled dataset |
| `writeup/plots/label_distribution.png` | Label distribution chart |
| `writeup/tables/cross_tab_department.csv` | Cross-tabulation table |

---

Quick Reference

Task	Code
Load CSV as scenarios	ScenarioList.from_csv("data/uploaded/file.csv")
Alternative CSV load	FileStore(path="file.csv").to_scenario_list()
Inspect columns	list(scenario_list[0].keys())
Select columns	ScenarioList([s.select("col1", "col2") for s in scenario_list])
Random sample	scenario_list.sample(5)
Binary label	QuestionYesNo(question_name=..., question_text=...)
Multi-class label	QuestionMultipleChoice(question_name=..., question_text=..., question_options=...)
Extraction	QuestionList(question_name=..., question_text=...)
Skip logic	survey.add_skip_rule("q2", "{{ q1.answer }} == 'No'")
Build sample job	labeling sample --n 5
Execute job	ep run <jobs.ep> --output <results.ep>
Preview results	results.select("scenario.*", "answer.*").print(max_rows=10)
Export to CSV	results.select("scenario.*", "answer.*").to_pandas().to_csv(...)
Load results	Results.load("data/results.ep")