| name | ecological-data-foundation |
| description | Cleans, validates, and standardizes ecological occurrence records and downloads biodiversity data from global repositories. Use this skill when the user needs data cleaning, coordinate validation, duplicate removal, outlier detection, taxonomic harmonization, or downloads from GBIF, iNaturalist, eBird, OBIS, or IUCN Red List. Also triggers for Darwin Core formatting, data quality reports, flagged records, and biodiversity data preparation. |
| skill_version | 1.0.0 |
Skill: ecological-data-foundation
Domain: Data ingestion · QA · Schema · Metadata
Phase: 1 — Foundation
Used by: All workflows
Purpose
This skill guides the agent through the first mandatory step of any quantitative ecology project: getting raw data into a clean, well-documented, analysis-ready state. It covers ingestion of heterogeneous sources, structural validation, duplicate and outlier detection, schema standardisation, and metadata generation.
When to Invoke
- A new dataset arrives (CSV, XLSX, GDB, Shapefile, database export, API pull)
- Before any spatial operation, statistical test, or model fitting
- When the user asks to "clean", "validate", "check", or "prepare" ecological data
- When merging datasets from different sources or institutions
Inputs
| Input | Format | Required |
|---|
| Raw occurrence or survey data | CSV, XLSX, TSV, GDB | Yes |
| Data dictionary or field protocol | PDF, DOCX, TXT | Recommended |
| Environmental layers (if applicable) | GeoTIFF, NetCDF | Optional |
| Existing metadata record | EML, JSON, YAML | Optional |
Outputs
| Output | Description |
|---|
data_clean.csv | Validated, deduplicated, standardised dataset |
qa_report.md | Summary of issues found and actions taken |
schema.yaml | Field definitions, types, valid ranges, units |
metadata.xml | EML or Dublin Core metadata record |
flagged_records.csv | Records removed or flagged, with reason codes |
Steps
1. Ingest and Inspect
- Load all source files; report file sizes, row counts, column names, and data types
- Identify encoding issues, BOM characters, delimiter inconsistencies
- Document source provenance (institution, date, version, license)
2. Schema Standardisation
- Map field names to a canonical schema (Darwin Core preferred for biodiversity)
- Enforce consistent data types (dates as ISO-8601, coordinates as decimal degrees WGS84)
- Flag or convert non-standard units
3. Duplicate Detection
- Identify exact duplicates (all fields identical)
- Identify spatial-temporal near-duplicates (same species, same coordinates, same date ± N days)
- Document resolution strategy (keep first, keep most complete, remove all)
4. Coordinate and Spatial QA
- Check coordinate ranges (latitude: −90 to 90; longitude: −180 to 180)
- Flag records with coordinates at country centroids, capital cities, or institution headquarters (likely georeferencing errors)
- Flag records with zero coordinates (0,0)
- Validate records fall within the stated country/region polygon
5. Taxonomic QA
- Check species names against a reference taxonomy (GBIF Backbone, Catalogue of Life)
- Flag synonyms, misspellings, and higher-rank-only identifications
- Resolve to accepted name + authorship
6. Temporal QA
- Check for dates in the future or before plausible survey era
- Flag records with only year-level precision when finer precision is needed
7. Attribute QA
- Check numeric fields for out-of-range values (e.g., abundance < 0, DBH > 500 cm)
- Check categorical fields for invalid entries
- Assess missing value rates per field; flag fields above threshold (default 20%)
8. Generate Outputs
- Write
data_clean.csv with action codes in an appended QA_status column
- Write
qa_report.md summarising each issue category, count, and resolution
- Write
schema.yaml with field definitions
- Write
metadata.xml in EML format
Decision Points
| Condition | Diagnosis | Recommended Action |
|---|
| n_records_clean < 30 | Insufficient records for SDM or occupancy modelling | Warn user; suggest removing country/taxon filters; proceed only with explicit user confirmation |
| Duplicate records > 20% of raw dataset | Strong collection bias or data ingestion error | Investigate source before proceeding; document deduplication method in qa_report.md |
| Coordinate uncertainty > threshold (e.g., > 10 km) | Spatial precision too low for fine-scale analysis | Flag records; exclude from spatial modelling; keep for range-map validation only |
| Taxonomic name match fails (0 results from backbone) | Synonym or spelling error in species name | Resolve synonym via GBIF backbone; halt if species identity cannot be confirmed |
Key Decisions to Document
- Duplicate resolution strategy
- Coordinate uncertainty threshold for exclusion
- Temporal precision requirement
- Missing value threshold per field
- Taxonomy backbone version used
Tools and Libraries
R: dplyr, readr, janitor, CoordinateCleaner, taxize, EML
Python: pandas, pyjanitor, geopy, pycountry, dwca-reader
CLI: csvkit, miller (mlr)
Resources
resources/darwin-core-glossary.md — Darwin Core field referenceresources/qa-checklist.md — printable QA checklistresources/coordinate-cleaning-flags.md — flag code referenceexamples/ — example prompt invocations
Notes
- Always preserve the original raw file; never overwrite it
- Record the exact software versions used for reproducibility
- Large raster datasets should be handled by the
geoprocessing-for-ecology skill after this step
