paper-data-processing

name	paper-data-processing
description	Use when building or rerunning bibliometric data preprocessing pipelines: WoS txt parsing, institution-country extraction, author disambiguation, schema validation, and UT-key dataset preparation. Portable — no hardcoded paths. Keywords: preprocess, clean, parse WoS, data pipeline, UT linkage, country extraction, author extraction, 数据处理, 清洗, 解析WOS, 作者消歧, 国家机构提取.

Paper Data Processing Skill

Purpose

Build a reproducible preprocessing layer that transforms raw WoS exports into canonical, UT-keyed datasets for downstream analysis. This skill is fully portable: it uses workspace-relative paths only, so it works on any machine with zero path editing.

Portable Directory Convention

All paths are relative to the workspace root (the folder that contains this .github/ directory). When the project is copied to a new machine, everything works as-is.

{workspace}/
├── data/
│   ├── raw/               # ← USER DROPS WoS .txt FILES HERE (the ONLY manual input)
│   ├── processed/         # ← All processing outputs land here
│   └── external/          # Optional external lookups
├── src/                   # Processing and analysis scripts
├── reports/               # Downstream report artifacts (populated by analysis skill)
├── models/                # Optional model artifacts (e.g. BERTopic)
├── figures/               # Optional standalone figure outputs
├── requirements.txt       # Python dependencies
└── .github/skills/        # Skill definitions (this file)

Key portability rules (enforced by the agent before every script run):

1. Scripts MUST read inputs from data/raw/ or data/processed/ and write outputs to data/processed/ or reports/.
2. Scripts MUST NOT contain absolute paths. Use Path(__file__).resolve().parent.parent to derive the workspace root, or accept paths via CLI arguments.
3. If the agent detects hardcoded absolute paths in a script, it MUST patch them to workspace-relative paths before execution.
4. The user provides exactly one thing: WoS .txt export files. Everything else is generated.

Pre-flight Interactive Inquiry (MANDATORY before first script run): The agent MUST ask the user:

1. “您的项目根目录在哪里？” — 用户确认后设置 PROJECT_ROOT 环境变量
2. “数据文件在哪里？格式是什么？ (.txt / .csv / .xlsx)”
   • .txt → 复制到 data/raw/，从 Stage A 开始
   • .csv / .xlsx → agent 读取前 5 行检测字段，根据字段完整度决定入口阶段：
     • 有 UT + 基础列 (TI/AB/PY) → 复制为 wos_merged.csv，跳过 Stage A，从 Stage B 开始
     • 有 UT + 国家/机构列 → 复制为 wos_cleaned.csv，跳过 A+B，从 Stage C 开始
     • 有 UT + 国家 + 作者列 → 复制到对应位置，跳过 A+B+C，直接 Stage D 质量检查
     • 无 UT 列 → 提示用户可能非 WoS 数据，尝试字段映射或要求用户确认 All scripts derive paths from ROOT = Path(os.environ.get("PROJECT_ROOT", Path(__file__).resolve().parent.parent)).

Scope

• In scope:
  • WoS txt parsing and canonical paper table generation.
  • Country and institution extraction/normalization.
  • Region extraction/normalization (EU27 merged to EU for regional analysis).
  • Author extraction and disambiguation.
  • Schema and quality checks before analysis handoff.
  • Patching scripts to use portable paths when hardcoded paths are detected.
• Out of scope:
  • BERTopic or any other thematic modeling.
  • Mobility trend interpretation and report writing.
  • Visualization-driven analysis conclusions.

Stage Workflow

1. Stage A — Parse raw WoS txt files from data/raw/ into a paper-level canonical CSV at data/processed/wos_merged.csv. SKIP if user provided CSV/Excel that already has UT + base columns.
2. Stage B — Enrich with country/institution/region fields → data/processed/wos_cleaned.csv. SKIP if input already has country/institution fields.
3. Stage C — Build author-level outputs → data/processed/paper_authors.csv, data/processed/authors_summary.csv. SKIP if input already has author fields.
4. Stage D — Run quality gates and emit handoff summary. Always runs.

Core Scripts

Stage	Script	Input	Output
A	src/wos_txt_to_csv.py	data/raw/*.txt	data/processed/wos_merged.csv
B	src/extract_country_institution.py	data/processed/wos_merged.csv	data/processed/wos_cleaned.csv
C	src/extract_authors.py	data/processed/wos_cleaned.csv	data/processed/paper_authors.csv, data/processed/authors_summary.csv

Before running any script, the agent MUST:

1. Read the script and check for hardcoded absolute paths.
2. If found, replace them with workspace-relative paths.
3. Confirm the script's input file exists.

Contract Outputs

All outputs in data/processed/:

Artifact	File	Stage
Canonical paper table	data/processed/wos_merged.csv	A
Enriched paper table	data/processed/wos_cleaned.csv	B
Paper-author pairs	data/processed/paper_authors.csv	C
Author summary	data/processed/authors_summary.csv	C

Required Quality Gates

• Gate 1: data/processed/wos_merged.csv exists and is readable.
• Gate 2: UT column exists, non-empty, unique at paper level.
• Gate 3: Required columns in data/processed/wos_cleaned.csv including country/region collaboration fields.
• Gate 4: data/processed/paper_authors.csv exists and contains UT linkage.

Handoff Rule

Analysis can start only when all quality gates pass. The paper-processing-analysis-handoff skill is the mandatory next step.

Usage Pattern

• Full rerun: stages A → D.
• Partial rerun: changed stage + dependent downstream stages.
• Schema change in upstream stage forces downstream rerun.

New Machine Setup

1. Clone or copy the workspace folder.
2. python -m venv .venv && .venv/Scripts/activate (Windows) or source .venv/bin/activate (Linux/Mac).
3. pip install -r requirements.txt.
4. Place data files into data/raw/ (supports .txt, .csv, .xlsx).
5. Run stages — agent auto-detects format and skips completed stages. No path editing required.

References

• references/runbook.md
• references/contracts.md
• references/quality-gates.md
• references/troubleshooting.md
• checklists/release-checklist.md

Done Criteria

• All processing stages complete without unhandled errors.
• All required artifacts exist in data/processed/.
• Quality gates pass and handoff is explicitly marked ready.