菜单
Method for labeling/classifying rows of a CSV using EDSL survey objects with iterative sampling, user approval, and analysis. Use when the user wants to classify, tag, or extract structured labels from tabular data using LLMs.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/expectedparrot/labeling --skill research-data-labeling命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
| name | research-data-labeling |
| description | Method for labeling/classifying rows of a CSV using EDSL survey objects with iterative sampling, user approval, and analysis. Use when the user wants to classify, tag, or extract structured labels from tabular data using LLMs. |
| tags | ["methods","data-collection","quantitative"] |
Use this workflow when a user has a CSV and wants to classify, label, or extract information from each row using LLMs. Each CSV row becomes an EDSL scenario; a survey asks labeling questions about each row; results are exported as a labeled CSV with an analysis report.
The data-labeling workflow has seven steps:
This follows the standard EDSL study structure. The study directory uses the normal scaffold from skills/workflow-file-layout/scripts/create_study_project.py, with the user's CSV placed in data/uploaded/.
Place the user's CSV in data/uploaded/ and load it as a ScenarioList.
from edsl import ScenarioList
scenario_list = ScenarioList.from_csv()
Alternative using FileStore:
from edsl import FileStore
scenario_list = FileStore(path="data/uploaded/job_posts.csv").to_scenario_list()
Before designing questions, understand what you have:
print(f"Rows: {len(scenario_list)}")
print(f"Columns: {list(scenario_list[0].keys())}")
# Preview a few rows
for s in scenario_list[:3]:
print(s)
If the CSV has many columns, you may only need a subset. Filter to the columns your questions will reference — every scenario field must appear in at least one question_text.
# Keep only the columns referenced in questions
scenario_list = ScenarioList(
[s.select("title", "description") for s in scenario_list]
)
IMPORTANT: Every field in the ScenarioList must be referenced in at least one question's
question_textusing{{ scenario.field_name }}syntax, or EDSL raises aJobsCompatibilityError.
Choose question types based on the labeling task. Common patterns:
from edsl import Survey, QuestionYesNo
q_mentions_ai = QuestionYesNo(
question_name="mentions_ai",
question_text=(
"Does the following job post mention the use of AI tools?\n\n"
"Title: {{ scenario.title }}\n"
"Description: {{ scenario.description }}"
)
)
survey = Survey([q_mentions_ai])
from edsl import QuestionMultipleChoice
q_category = QuestionMultipleChoice(
question_name="category",
question_text=(
"Classify the following product review into one category.\n\n"
"Review: {{ scenario.review_text }}"
),
question_options=["Positive", "Negative", "Neutral", "Mixed"]
)
Confidence scoring: For any classification question, you can extract per-option probabilities by adding
use_code=True, include_comment=Falseto the question and using an OpenAI model withlogprobs=True, temperature=1. Seeskills/methods-logprob-confidence/SKILL.mdfor the full pattern.
from edsl import QuestionList
q_tools = QuestionList(
question_name="ai_tools_mentioned",
question_text=(
"List all AI tools mentioned in this job post. "
"Return an empty list if none.\n\n"
"Title: {{ scenario.title }}\n"
"Description: {{ scenario.description }}"
)
)
This is the most common data-labeling pattern. Ask a screening question first, then only ask the follow-up if the answer warrants it.
from edsl import Survey, QuestionYesNo, QuestionList, QuestionFreeText
q_mentions_ai = QuestionYesNo(
question_name="mentions_ai",
question_text=(
"Does the following job post mention the use of AI tools?\n\n"
"Title: {{ scenario.title }}\n"
"Description: {{ scenario.description }}"
)
)
q_which_tools = QuestionList(
question_name="which_tools",
question_text=(
"Which specific AI tools are mentioned in this job post?\n\n"
"Title: {{ scenario.title }}\n"
"Description: {{ scenario.description }}"
)
)
q_how_used = QuestionFreeText(
question_name="how_used",
question_text=(
"Briefly describe how AI tools are used or expected to be used "
"according to this job post.\n\n"
"Title: {{ scenario.title }}\n"
"Description: {{ scenario.description }}"
)
)
survey = (
Survey([q_mentions_ai, q_which_tools, q_how_used])
.add_skip_rule("which_tools", "{{ mentions_ai.answer }} == 'No'")
.add_skip_rule("how_used", "{{ mentions_ai.answer }} == 'No'")
)
For more complex labeling with branching paths:
from edsl import QuestionMultipleChoice, QuestionFreeText, Survey
q_type = QuestionMultipleChoice(
question_name="post_type",
question_text="What type of posting is this?\n\n{{ scenario.text }}",
question_options=["Job posting", "Advertisement", "News article", "Other"]
)
q_job_details = QuestionFreeText(
question_name="job_details",
question_text="Summarize the key requirements of this job posting.\n\n{{ scenario.text }}"
)
q_other_details = QuestionFreeText(
question_name="other_details",
question_text="Briefly describe what this posting is about.\n\n{{ scenario.text }}"
)
survey = (
Survey([q_type, q_job_details, q_other_details])
.add_skip_rule("job_details", "{{ post_type.answer }} != 'Job posting'")
.add_skip_rule("other_details", "{{ post_type.answer }} == 'Job posting'")
)
Before running the full dataset, test on a small random sample.
labeling sample --n 5
labeling plan sample --latest
# After reviewing the plan, run the returned command:
ep run edsl_jobs/job_a/sample_jobs.ep --output data/results.sample.ep
labeling record run --kind sample --results data/results.sample.ep
labeling preview latest --human
None)?Present the sample results to the user via AskUserQuestion and ask:
Based on user feedback, adjust the survey and build a fresh sample job:
labeling validate
labeling generate --replace-generated
labeling sample --n 5
labeling plan sample --latest
Common adjustments:
Repeat until the user approves the labeling quality.
After user approval, ask Labeling for the full-run plan. Labeling builds the
Jobs artifact; ep exclusively owns execution.
Cost check: Before running, confirm with the user. State the number of rows and estimated cost if known. This is required by the "no silent spending" constraint.
labeling plan full
# After explicit approval, run the returned command:
ep run edsl_jobs/job_a/jobs.ep --output data/results.ep
labeling record run --kind full --results data/results.ep
Note: For data-labeling jobs there is typically no agent_list — the default agent is used. If the user wants labels from a specific persona, add agents as usual.
Create an export script in analysis/ that merges scenario data with labels.
# analysis/export_labeled_csv.py
from pathlib import Path
from edsl import Results
STUDY_ROOT = Path(__file__).resolve().parent.parent
results = Results.load(str(STUDY_ROOT / "data" / "results.ep"))
# Select scenario fields and answer fields
df = results.select("scenario.*", "answer.*").to_pandas()
# Clean column names: remove "scenario." and "answer." prefixes
df.columns = [c.replace("scenario.", "").replace("answer.", "label_") for c in df.columns]
output_path = STUDY_ROOT / "data" / "cooked" / "labeled_results.csv"
output_path.parent.mkdir(parents=True, exist_ok=True)
df.to_csv(output_path, index=False)
print(f"Exported {len(df)} labeled rows to {output_path}")
Makefile target:
data/cooked/labeled_results.csv: analysis/export_labeled_csv.py data/results.ep
python analysis/export_labeled_csv.py
export: data/cooked/labeled_results.csv
Follow the standard study conventions. Create analysis scripts in analysis/, with outputs going to writeup/plots/ and writeup/tables/.
Label distribution — How many rows got each label?
# analysis/plot_label_distribution.py
from pathlib import Path
from edsl import Results
import matplotlib.pyplot as plt
STUDY_ROOT = Path(__file__).resolve().parent.parent
results = Results.load(str(STUDY_ROOT / "data" / "results.ep"))
labels = results.select("answer.mentions_ai").to_list()
counts = {}
for label in labels:
counts[label] = counts.get(label, 0) + 1
plt.figure(figsize=(8, 5))
plt.bar(counts.keys(), counts.values())
plt.title("Label Distribution: Mentions AI Tools")
plt.ylabel("Count")
plt.tight_layout()
output = STUDY_ROOT / "writeup" / "plots" / "label_distribution.png"
output.parent.mkdir(parents=True, exist_ok=True)
plt.savefig(output, dpi=150)
print(f"Saved {output}")
Cross-tabulation — How do labels relate to other columns?
# analysis/table_cross_tab.py
import pandas as pd
from pathlib import Path
from edsl import Results
STUDY_ROOT = Path(__file__).resolve().parent.parent
results = Results.load(str(STUDY_ROOT / "data" / "results.ep"))
df = results.select("scenario.department", "answer.mentions_ai").to_pandas()
cross_tab = pd.crosstab(df["scenario.department"], df["answer.mentions_ai"])
output = STUDY_ROOT / "writeup" / "tables" / "cross_tab_department.csv"
output.parent.mkdir(parents=True, exist_ok=True)
cross_tab.to_csv(output)
print(f"Saved {output}")
Example rows per label — Show representative examples for each label category.
Model agreement — If multiple models were used, compare label consistency across models.
.PHONY: data export PLOTS TABLES report all
data:
ep run edsl_jobs/job_a/jobs.ep --output data/results.ep
export: data/cooked/labeled_results.csv
data/cooked/labeled_results.csv: analysis/export_labeled_csv.py data/results.ep
python analysis/export_labeled_csv.py
PLOTS: writeup/plots/label_distribution.png
writeup/plots/label_distribution.png: analysis/plot_label_distribution.py data/results.ep
python analysis/plot_label_distribution.py
TABLES: writeup/tables/cross_tab_department.csv
writeup/tables/cross_tab_department.csv: analysis/table_cross_tab.py data/results.ep
python analysis/table_cross_tab.py
writeup/report.html: writeup/report.md PLOTS TABLES
cd writeup && pandoc report.md -o report.html --standalone --embed-resources --css=report.css
writeup/report.pdf: writeup/report.md writeup/report_header.tex writeup/fix-table-widths.lua PLOTS TABLES
cd writeup && pandoc report.md -o report.pdf --pdf-engine=xelatex --include-in-header=report_header.tex --lua-filter=fix-table-widths.lua
report: writeup/report.html writeup/report.pdf
all: data export PLOTS TABLES report
Before writing the final report, run labeling report context and optionally
labeling report template. These are bounded analytical inputs. The calling
report agent—not Labeling—owns writeup/report.md, branding, and final audience
framing.
The report in writeup/report.md should follow the standard format:
---
title: "Data Labeling: [Topic]"
date: YYYY-MM-DD
---
## Summary
Brief overview of the labeling task, dataset, and key findings.
## Dataset
- Source: [description of the CSV and where it came from]
- Rows: N
- Columns used for labeling: [list]
## Labeling Design
- Survey questions and their purpose
- Skip logic rules
- Models used
## Results
### Label Distribution
[Table and/or chart showing how labels are distributed]
### Cross-Tabulations
[How labels relate to other variables in the data]
### Example Rows
[Representative examples for each label category]
## Key Findings
[2-3 paragraphs synthesizing what the labeling revealed about the data]
## Files Generated
| File | Description |
|------|-------------|
| `data/cooked/labeled_results.csv` | Full labeled dataset |
| `writeup/plots/label_distribution.png` | Label distribution chart |
| `writeup/tables/cross_tab_department.csv` | Cross-tabulation table |
| Task | Code |
|---|---|
| Load CSV as scenarios | ScenarioList.from_csv("data/uploaded/file.csv") |
| Alternative CSV load | FileStore(path="file.csv").to_scenario_list() |
| Inspect columns | list(scenario_list[0].keys()) |
| Select columns | ScenarioList([s.select("col1", "col2") for s in scenario_list]) |
| Random sample | scenario_list.sample(5) |
| Binary label | QuestionYesNo(question_name=..., question_text=...) |
| Multi-class label | QuestionMultipleChoice(question_name=..., question_text=..., question_options=...) |
| Extraction | QuestionList(question_name=..., question_text=...) |
| Skip logic | survey.add_skip_rule("q2", "{{ q1.answer }} == 'No'") |
| Build sample job | labeling sample --n 5 |
| Execute job | ep run <jobs.ep> --output <results.ep> |
| Preview results | results.select("scenario.*", "answer.*").print(max_rows=10) |
| Export to CSV | results.select("scenario.*", "answer.*").to_pandas().to_csv(...) |
| Load results | Results.load("data/results.ep") |