crawlers-and-wikidata-pipelines

# Crawlers and Wikidata Pipelines Use this sub-skill when the task is to inspect, run, repair, or validate the repository's crawler-driven data acquisition and intermediate CSV generation workflows for Hudong/Baike pages, Wikidata entity/relation alignment, weather relations, attributes, and tree source lists. ## Load this when - The user asks how to run a Scrapy spider, choose the correct working directory, or understand crawler outputs. - The task involves `hudong_pedia`, `entityRelation`, `wikidata_relation`, `new_node`, `weather_plant`, `city_weather`, `attributes`, `staticResult`, or tree-list artifacts before Neo4j import. - A pipeline failed because expected JSON/CSV files are missing, a Scrapy project cannot find settings, or CSV rows are malformed. - You need a no-network validation check for relation/weather CSV headers and row invariants. ## Route elsewhere when - The next action is final Neo4j `LOAD CSV`, Cypher import order, graph constraints, graph query semantics, or tree/vector query APIs; route to the graph-query-and-data-management sub-skill. - The task is PCNN relation-extraction dataset generation, deduplication, or TensorFlow training; route to the relation-extraction-pipeline sub-skill. - The task is Django route operation, Mongo-backed tagging UI, THULAC NER behavior, or KNN/fastText entity labeling; use the corresponding sibling sub-skill. ## Operating references 1. Start with [crawler-workflows.md](references/crawler-workflows.md) for Scrapy projects, DFS tree crawlers, legacy data-processing scripts, working directories, output names, and network constraints. 2. Use [wikidata-processing.md](references/wikidata-processing.md) for Wikidata relation conversion, `new_node.csv`, `wikidata_relation*.csv`, relation distribution analysis, and safe CSV validation. 3. Use [weather-and-attribute-pipelines.md](references/weather-and-attribute-pipelines.md) for city/weather, weather-to-plant, climate-list, and attribute-extraction artifacts. 4. Use [troubleshooting.md](references/troubleshooting.md) for crawler-specific recovery steps, path-sensitive failures, service prerequisites, and malformed CSV symptoms. ## Bundled safe script - [validate_relation_csvs.py](scripts/validate_relation_csvs.py) checks required CSV headers, non-empty fields, duplicate/suspicious rows, and cross-file consistency for `wikidata_relation.csv`, `wikidata_relation2.csv`, `new_node.csv`, `weather_plant.csv`, and `city_weather.csv`. It performs no network or database calls. Example safe checks: ```bash python scripts/validate_relation_csvs.py --help python scripts/validate_relation_csvs.py --self-test python scripts/validate_relation_csvs.py --root path/to/csv-export-directory ``` ## Safety defaults - Treat all live crawls as network-expensive and site-policy-sensitive. Do not start a broad crawl, disable robots compliance, or raise concurrency without explicit user approval. - Prefer inspecting existing generated artifacts and running the bundled CSV validator before any live crawl or service-backed conversion. - Run path-sensitive source scripts from their documented project directories. Many scripts write outputs relative to the current working directory and some execute work at module import time. - Do not claim Wikidata/weather conversion is verified unless the required local files and, where applicable, Neo4j/THULAC dependencies were actually checked in the active environment.