تشغيل أي مهارة في Manus بنقرة واحدة

data-catalog

النجوم٣٨

التفرعات١٥

آخر تحديث٦ أبريل ٢٠٢٦ في ١٨:٢٤

Activates when querying Danish agricultural data from R2/cloud storage. Use this skill for: data discovery, finding datasets, understanding schemas, querying parquet files, joining datasets on CVR/CHR/BFE identifiers. Keywords: data, catalog, datasets, R2, GCS, parquet, schema, query, DuckDB, rclone

التثبيت

التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.

تشغيل في Manus

المصدر

Klimabevaegelsen

Klimabevaegelsen/landbruget.dk

فتح مستودع GitHub عرض مستودعات المنشئ

تنزيل

تشغيل في Manus

المهن ذات الصلةSOC

استنادا إلى تصنيف SOC المهني

علماء البياناتمهن الحاسوب والرياضيات·SOC 15-2051

مستكشف الملفات

6 ملفات

SKILL.md

readonly

المزيد من هذا المستودع

نفس المستودع

data-pipeline-operations

Klimabevaegelsen/landbruget.dk

Activates when working with Python data pipelines, GCS operations, or medallion architecture (Bronze/Silver/Gold). Use this skill for: running pipelines, debugging data transformations, GCS uploads/downloads, data quality validation, CVR/CHR/BFE identifier handling, GeoPandas/PostGIS operations, and DuckDB queries for large files. Keywords: pipeline, bronze, silver, gold, GCS, parquet, CVR, CHR, BFE, transform, ingest, ETL, DuckDB, large files

2026-04-1238

r2-husdyr-data

Klimabevaegelsen/landbruget.dk

Activates when querying livestock and animal data from R2. Use this skill for: CHR registry, pig movements, animal welfare, antibiotics, animal density, mortality rates, herd tracking, svineflytning. Keywords: husdyr, livestock, animal, dyr, CHR, svin, pig, ko, cattle, antibiotika, dyrevelfærd, flytning, movement

2026-04-0638

r2-landbrugsareal-data

Klimabevaegelsen/landbruget.dk

Activates when querying agricultural land and field data from R2. Use this skill for: field boundaries, crop data, land use, organic farming, production estimates, cadastral data, agricultural blocks, building data. Keywords: landbrugsareal, field, mark, marker, crop, afgrøde, organic, økologisk, production, areal, cadastral, matrikel

2026-04-0638

r2-medarbejdere-data

Klimabevaegelsen/landbruget.dk

Activates when querying employee and workplace safety data from R2. Use this skill for: Arbejdstilsynet inspections, work permits, safety violations, workplace accidents, compliance rates, foreign workers, incident tracking. Keywords: medarbejdere, employees, worker, arbejdstilsynet, inspection, tilsyn, safety, arbejdsmiljø, accident, ulykke, compliance

2026-04-0638

r2-miljo-data

Klimabevaegelsen/landbruget.dk

Activates when querying environmental data from R2. Use this skill for: pesticides, nitrogen leaching, BNBO drinking water protection, wetlands, soil types, environmental compliance, biodiversity. Keywords: miljø, environment, pesticide, pesticid, nitrogen, kvælstof, BNBO, wetlands, vådomr, soil, jord, biodiversity

2026-04-0638

r2-okonomi-data

Klimabevaegelsen/landbruget.dk

Activates when querying financial/economic data from R2. Use this skill for: subsidies, farm financials, property values, CVR enrichment, ownership data, støtte per hektar, samlet støtte, grundværdi. Keywords: økonomi, finance, subsidies, støtte, tilskud, CVR, property, ejendom, grundværdi

2026-04-0638

name	data-catalog
description	Activates when querying Danish agricultural data from R2/cloud storage. Use this skill for: data discovery, finding datasets, understanding schemas, querying parquet files, joining datasets on CVR/CHR/BFE identifiers. Keywords: data, catalog, datasets, R2, GCS, parquet, schema, query, DuckDB, rclone

Data Catalog - R2 Storage (landbruget-data)

Data lake with 130+ datasets across bronze/silver/gold medallion layers in Cloudflare R2.

Discovering Data

Use rclone to browse R2 — never use gsutil or google.cloud.storage.

# List top-level directories
rclone lsd r2:landbruget-data/

# List datasets in a layer
rclone lsd r2:landbruget-data/bronze/
rclone lsd r2:landbruget-data/silver/
rclone lsd r2:landbruget-data/gold/

# List snapshots (timestamped subdirectories) for a dataset
rclone lsd r2:landbruget-data/silver/subsidies/

# List files in a snapshot
rclone ls r2:landbruget-data/silver/subsidies/

# Find latest snapshot for a dataset
rclone lsd r2:landbruget-data/gold/field_production_2024/ | tail -1

Important: Some folder names contain spaces (e.g., silver/animal welfare/, silver/work permits/). Always quote paths when scripting.

Reading Data with DuckDB

DuckDB with R2 auth is the primary way to query data. Use StorageAccess from backend/common/storage/core.py:

from common.storage.core import StorageAccess

storage = StorageAccess()

# Read a parquet file into DuckDB
storage.create_table_from_storage_parquet("my_table", "landbruget-data/silver/subsidies/20260401_020000/data.parquet")

# Query it
result = storage.execute_query("SELECT cvr_number, COUNT(*) FROM my_table GROUP BY cvr_number")

Or use DuckDB directly after auth setup:

import duckdb
from common.storage.filesystem import setup_duckdb_cloud_auth

conn = duckdb.connect()
setup_duckdb_cloud_auth(conn)

# Query directly from R2
result = conn.execute("""
    SELECT cvr_number, SUM(area_ha) as total_area
    FROM read_parquet('r2://landbruget-data/gold/field_production_2024/*/data.parquet')
    GROUP BY cvr_number
""").fetchdf()

Environment Variables

# R2 credentials (required)
R2_ACCESS_KEY_ID=<access-key>
R2_SECRET_ACCESS_KEY=<secret-key>
R2_ACCOUNT_ID=<account-id>

# Bucket name (defaults to "landbruget-data")
R2_BUCKET=landbruget-data
# Or: STORAGE_BUCKET=landbruget-data

Medallion Architecture

bronze/ — Raw data exactly as received (133 datasets)
silver/ — Cleaned, validated, standardized (126 datasets)
gold/ — Analysis-ready, joined datasets (86 datasets)

Full Dataset Inventory

Bronze (133 datasets)

Category	Datasets
Fields	`agricultural_blocks_{2020-2024}`, `agricultural_fields_{2020-2025}`, `fields`
FVM Marker	`fvm_marker_{2008-2025}`, `fvm_marker_smaabiotoper_{2023-2025}`
FVM Markblokke	`fvm_markblokke_{2005-2026}`
Organic	`fvm_organic_areas_{2012-2024}`, `fvm_organic_subsidies_{2019-2024}`
Subsidies	`subsidies`, `fvm_environmental_subsidies_{2019-2023}`, `fvm_grassland_subsidies_{2019-2024}`
Jordbrugsanalyser	`jordbrugsanalyser_markers_{2012-2024}`
Cadastral/Geo	`cadastral`, `dagi_kommuner`, `dagi_landsdele`, `dagi_postnumre`, `dagi_regioner`, `bbr_buildings`
Environment	`bnbo_status`, `wetlands`, `soil_types`, `water_projects`, `water_typology_` (3 datasets), `grukos_` (2 datasets), `fertiliser`
Pesticides	`pesticides`, `bmd`, `geus_dataverse_pesticides`, `kemidata_surface_water_pesticides`
Livestock	`chr`, `animal_welfare`, `animal_mortality`, `animal_international_movements`, `pig_tail_cutting`, `slurry_leaks`, `stable_fires`, `transportation_accidents`
Companies	`cvr_raw_companies`, `dst`, `dmi`
Workers	`arbejdstilsynet_inspections`, `work_permits`, `worker_safety`

Silver (126 datasets)

Category	Datasets
Fields	`fvm_marker_{2008-2025}`, `fvm_markblokke_{2005-2026}`, `fvm_smaabiotoper_{2023-2025}`, `fields`
Organic	`fvm_organic_areas_{2012-2023}`, `fvm_organic_subsidies_{2019-2024}`
Subsidies	`subsidies`, `fvm_environmental_subsidies_{2019-2023}`, `fvm_grassland_subsidies_{2019-2024}`
Cadastral/Geo	`cadastral`, `dagi_kommuner`, `dagi_landsdele`, `dagi_postnumre`, `dagi_regioner`, `bbr_buildings`, `dst_zone_mapping`, `dst_zone_mapping_reference`
Environment	`bnbo_status`, `bnbo_status_dissolved`, `wetlands` (implied), `grukos`, `grukos_*_dissolved` (2), `water_projects`, `water_projects_dissolved`, `fertiliser`
Pesticides	`pesticides`, `bmd`, `geus_dataverse_pesticides`, `geus_dataverse_pesticides_pfas`
Livestock	`chr`, `svineflytning`, `animal welfare`, `animal mortality`, `animal international movements`, `pig tail cutting`, `slurry leaks`, `stable fires`, `transportation accidents`
Companies	`cvr_companies`, `cvr_employment`, `cvr_persons`, `property_owners`
Workers	`arbejdstilsynet_inspections`, `work permits`, `worker safety`
Legacy/Other	`2016_*` (4 datasets), `gr {2015-2023}` (8 datasets), `fro_processed`, `gartn1_processed`, `halm1_processed`, `hst77_processed`

Gold (86 datasets)

Category	Datasets
Field Production	`field_production_{2008-2025}` (18 years)
Field Analysis	`field_analysis_field_bnbo_intersections_{2024,2025}`, `field_analysis_field_bnbo_water_intersections_{2024,2025}`, `field_analysis_field_grukos_intersections_{2024,2025}`, `field_analysis_field_wetland_intersections_{2024,2025}`, `field_analysis_field_wetland_water_intersections_{2024,2025}`, `field_analysis_soil_intersections_{2024,2025}`, `field_analysis_water_projects_bnbo_intersections_{2024,2025}`, `field_analysis_water_projects_wetlands_intersections_{2024,2025}`, `field_analysis_wetland_water_coverage`
Environmental Analysis	`field_environmental_analysis_fields_{2024,2025}`, `field_environmental_analysis_properties_{2024,2025}`
Pesticides	`pesticide_disaggregation_{2010_2011 through 2023_2024}` (13 year-pairs), `pesticide_proximity_{2010_2011 through 2023_2024}` (13 year-pairs)
Pre-computed stages	`stage0_bnbo_filtered_{2024,2025}`, `stage0_grukos_filtered_{2024,2025}`, `stage0_properties_filtered_{2024,2025}`, `stage0_soil_types_filtered_{2024,2025}`, `stage0_water_projects_filtered_{2024,2025}`, `stage0_wetlands_filtered_{2024,2025}`
Livestock	`chr_timeline_summary`, `chr_veterinary_timeline`
Companies	`cvr_enrichment`, `cvr_enrichment_collection`, `cvr_enrichment_companies`, `cvr_enrichment_financial`, `cvr_enrichment_financial_statements`, `cvr_enrichment_pnumbers`
Cadastral	`property_cadastral_merged`

Other Top-Level

api/ — API-related data
cvr_collections/ — CVR collection data

Key Identifiers

Identifier	Format	Description	Validation
CVR	8 digits	Company registration number	`^\d{8}$`
CHR	6 digits	Central Husbandry Register (herd ID)	`^\d{6}$`
BFE	Variable	Cadastral parcel number	varies
field_id	String	Field identifier from FVM	varies
field_uuid	UUID	Unique field identifier	UUID format

Dataset Quick Reference

Okonomi (Finance)

Dataset	Path	Rows	Key Columns
Subsidies	`silver/subsidies/`	554K	cvr_number, tilskudsberetigt
CVR Enrichment	`gold/cvr_enrichment/`	varies	cvr_number, company data
Property Owners	`silver/property_owners/`	8.2M	CVRNummer, owner info

Landbrugsareal (Agricultural Land)

Dataset	Path	Rows	Key Columns
FVM Marker (fields)	`silver/fvm_marker_{year}/`	617K/year	field_id, cvr_number, crop_code, geometry
Field Production	`gold/field_production_{year}/`	617K/year	field_id, yield_estimate, crop_type
Cadastral	`silver/cadastral/`	2.16M	bfe_number, geometry

Miljo (Environment)

Dataset	Path	Rows	Key Columns
Pesticide Disaggregation	`gold/pesticide_disaggregation_{year}/`	1.52M	cvr_number, PesticideName, DosageQuantity
BNBO Status	`silver/bnbo_status/`	5.4K	geometry, status_bnbo
Wetlands	`silver/wetlands/` (in bronze)	1.7M	geometry, toerv_pct

Husdyr (Livestock)

Dataset	Path	Rows	Key Columns
Svineflytning	`silver/svineflytning/`	1.27M	sender_chr_number, receiver_chr_number, total_animals
CHR Movements	`bronze/chr/`	124K	reporting_herd_number, animal_count
Animal Welfare	`silver/animal welfare/`	varies	chr_number

Medarbejdere (Employees)

Dataset	Path	Rows	Key Columns
Arbejdstilsynet	`gold/arbejdstilsynet_inspections/`	536	cvr_number, decision, severity_score
Work Permits	`silver/work permits/`	varies	cvr_number
Worker Safety	`silver/worker safety/`	varies	cvr_number

Cross-Dataset Joins

CVR-based joins (most common)

# Join subsidies with pesticides on CVR using DuckDB
conn.execute("""
    SELECT s.cvr_number, s.tilskudsberetigt, p.PesticideName
    FROM read_parquet('r2://landbruget-data/silver/subsidies/*/data.parquet') s
    JOIN read_parquet('r2://landbruget-data/gold/pesticide_disaggregation_2023_2024/*/data.parquet') p
    ON s.cvr_number = p.cvr_number
""")

Field-based joins

# Join field production with environmental analysis
conn.execute("""
    SELECT fp.field_id, fp.yield_estimate, fe.bnbo_overlap_pct
    FROM read_parquet('r2://landbruget-data/gold/field_production_2024/*/data.parquet') fp
    JOIN read_parquet('r2://landbruget-data/gold/field_environmental_analysis_fields_2024/*/data.parquet') fe
    ON fp.field_id = fe.field_id
""")

Data Update Schedule

Layer	Frequency	Notes
Bronze	Weekly (Mondays 2AM UTC)	Immutable, timestamped
Silver	After bronze update	Cleaned, validated
Gold	After silver update	Analysis-ready

Related Skills

okonomi/ - Financial data: subsidies, property values
landbrugsareal/ - Field and crop data: FVM marker, production
miljo/ - Environmental data: pesticides, nitrogen, BNBO
husdyr/ - Livestock data: CHR, movements, welfare
medarbejdere/ - Employee data: inspections, safety

Troubleshooting

Check R2 access

rclone lsd r2:landbruget-data/

If rclone fails

Check ~/.config/rclone/rclone.conf has an [r2] section with:

type = s3
provider = Cloudflare
access_key_id and secret_access_key
endpoint pointing to your R2 account

Large Files

Use DuckDB — never load large parquet files into Pandas:

conn.execute("""
    SELECT cvr_number, SUM(area_ha) as total_area
    FROM read_parquet('r2://landbruget-data/gold/field_production_2024/*/data.parquet')
    GROUP BY cvr_number
""").fetchdf()

CRS

All geometry stored in EPSG:4326 (WGS84) in Supabase. Bronze/Silver/Gold processing uses EPSG:25832 (UTM 32N).