data-collection

name	data-collection
description	Use when collecting data for a research project, downloading time series, building a dataset, accessing economic or social data APIs, or scraping data from a non-API source. Handles source discovery, respectful collection, local caching, and manifest documentation.

Data Collection

Overview

This skill guides data collection from the research question to a versionable artifact in data/raw/. It is field-agnostic and open-ended about sources — the references/common-sources.md file is a starting point, not a boundary. For any research question, the skill uses web search to find appropriate sources beyond the common list.

When to Use

• "I need data on X"
• "Download the unemployment series"
• "Build a dataset of country-level GDP"
• "Scrape this website for paper data"
• "Where can I get data about Y?"
• Any new project's collection phase

Mandatory Steps

1. Identify data needs from the research question. Variables, units (country, firm, individual, pixel), frequency, period, geography, and any necessary keys for merging across sources.

2. Find appropriate sources. Start with references/common-sources.md. If the user's needs are not covered there, search the web for the relevant source. Never invent a URL or API endpoint from memory.

3. Prefer APIs over scraping. APIs are versioned, documented, and legal. Scraping is the last resort when no API is available.

4. When scraping is necessary, be respectful:

   • Check and honor robots.txt
   • Rate limit — minimum one request per second, often slower
   • Exponential backoff on errors (e.g., 1s, 2s, 4s, 8s, cap at 60s)