unified-ameriflux-extractor

name	unified-ameriflux-extractor
description	Run the combined AmeriFlux to EcoSIM extraction workflow for site metadata, NADP or tDEP atmospheric chemistry, gSSURGO soil data, and optional climate or grid NetCDF forcing. Use when building a complete site-specific EcoSIM input package.

Unified AmeriFlux Extractor

Use When

• You need a complete EcoSIM input package for an AmeriFlux site.
• You want one workflow to combine site metadata, atmospheric chemistry, tDEP deposition, gSSURGO soil profile data, and optional NetCDF forcing files.
• You need derivation reports for climate or grid variables left at fill/default values.

Constraints

• Do not extract climate data beyond site metadata (the script only extracts site metadata, NADP, tDEP, and gSSURGO soil data).
• Site metadata extraction must use the structured ameriflux-site-info workflow; it should not require a local vision model.
• EcoSIM grid NetCDF files generated by this workflow do not need the optional atmospheric gas scalar variables OXYEG (or typo OXYGE), Z2GEG, CO2EIG, CH4EG, Z2OEG, or ZNH3EG. Do not report their absence as a grid derivation gap.

Workflow

1. Resolve site metadata with ameriflux-site-info.
2. Extract NADP and tDEP chemistry with ameriflux-atmchem-info.
3. Extract soil profile data with ameriflux-surgo-grid-extract when gSSURGO inputs are provided.
4. Optionally invoke Tools/create_ecosim_climate_forcing.py or Tools/create_ecosim_grid_forcing.py.
5. When grid, PFT-management, or soil-management NetCDF files are created or found, also create editable Excel sidecars under result/<SITE_ID>/:
   • result/<SITE_ID>/<SITE_ID>_ecosim_grid.xlsx
   • result/<SITE_ID>/<SITE_ID>_pft_mgmt.xlsx
   • result/<SITE_ID>/<SITE_ID>_soil_mgmt.xlsx
6. When climate, grid, plant-management, and soil-management files are available, create a site namelist with ameriflux-namelist-generator.
7. Write merged JSON and site-specific outputs under result/<SITE_ID>/.

Purpose

Combine multiple AmeriFlux data extraction capabilities into a single workflow:

1. Retrieve site metadata (latitude, longitude, elevation, mean annual temperature, Koppen climate code, IGBP vegetation type) from cached EcoSIM site JSON or structured AmeriFlux site page text.
2. Extract atmospheric chemistry (NADP) data for a range of years.
3. Extract atmospheric deposition (tDEP) data for a range of years.
4. Extract a dominant-component soil profile from a gSSURGO geodatabase using the ameriflux-surgo-grid-extract skill.

The result is a single JSON file containing all extracted information.

Implementation Details

• Uses the structured extraction logic from .agents/skills/ameriflux-site-info.
• Reuses NADP and tDEP extraction code from .agents/skills/ameriflux-atmchem-info.
• Calls the gSSURGO extraction script (extract_gssurgo_profile.py) via subprocess to obtain soil profile data.
• Coordinates are transformed as needed for each data source.
• All data are merged under top‑level keys: site_id, site_metadata, nadp, tdep, and gssurgo.

Prerequisites

• Python 3.8+.
• Install required packages:

  pip install playwright requests rasterio pyproj pyogrio geopandas numpy pandas netCDF4
  playwright install chromium
  

• Site metadata extraction uses cached JSON, static AmeriFlux HTML, or rendered DOM text. No local vision service is required.
• Access to the gSSURGO geodatabase (gSSURGO_CONUS.gdb).

User-Downloaded Data

Before running the full workflow, confirm the user has downloaded or can access the source data needed for the requested outputs:

• AmeriFlux BASE/FULLSET meteorological files and BADM/BIF metadata for the target site when using curated AmeriFlux forcing or management records. Store local copies under data/ or pass explicit paths.
• NADP precipitation-chemistry rasters and EPA tDEP deposition rasters for US atmospheric chemistry and deposition inputs.
• gSSURGO_CONUS.gdb for US soil/grid extraction, plus any EcoSIM grid template NetCDF requested by downstream grid tools.
• ERA5 source forcing files when --climate-output is requested through the local climate-forcing tool. For AmeriFlux downloads, pay attention to whether the source is half-hourly ERA5_HH or hourly ERA5_HR.
• EcoSIM assets that this workflow references but does not download: PFT parameter NetCDF, atmospheric GHG NetCDF, plant-management NetCDF for existing cases, and microbial parameter files when required by the namelist.

Usage

python .agents/skills/unified-ameriflux-extractor/unified_ameriflux_extractor.py \
    --site-id US-Ha1 \
    --nadp-input /path/to/nadp_data \
    --tdep-input /path/to/tdep_data \
    --year1 2010 --year2 2020 \
    --gssurgo-gdb /path/to/gSSURGO_CONUS.gdb \
    --gssurgo-template /path/to/template.nc \
    --gssurgo-out result/US-Ha1/gssurgo_US-Ha1.json \
    --gssurgo-extend-last \
    --output result/US-Ha1/unified_output.json \
    --climate-output result/${site_id}/${site_id}_ecosim_climate.nc \
    --grid-output result/${site_id}/${site_id}_ecosim_grid.nc \
    --create-namelist \
    --pft-file /path/to/ecosim_pftpar_YYYYMMDD.nc \
    --atm-ghg-file /path/to/fatm_hist_GHGs_1750-2025.nc \
    --climate-data-dir data \
    --result-dir result

• --site-id is optional if latitude/longitude are provided directly, but required when generating EcoSIM climate or grid forcing files.
• If --gssurgo-gdb, --gssurgo-template, and --gssurgo-out are omitted, the gSSURGO extraction step is skipped.
• --climate-output (optional) specifies where to write the EcoSIM climate forcing NetCDF file. The script will invoke the climate forcing script located at Tools/create_ecosim_climate_forcing.py.
• --grid-output (optional) specifies where to write the EcoSIM grid forcing NetCDF file. The script will invoke the grid forcing script located at Tools/create_ecosim_grid_forcing.py.
• --climate-data-dir points to the directory containing ERA5 files (default: data).
• --result-dir is the directory for intermediate results (default: result).
• --create-namelist creates result/<SITE_ID>/<SITE_ID>.namelist after site forcing files are available.
• --namelist-output can override the namelist destination; otherwise --create-namelist uses the standard site result directory.
• --namelist-run-dir writes namelist input paths relative to an EcoSIM run directory.
• --pft-file, --atm-ghg-file, --pft-mgmt-file, and --micpar-file are passed through to ameriflux-namelist-generator. If --pft-mgmt-file is omitted, the generator searches for result/<SITE_ID>/<SITE_ID>_pft_mgmt.nc.
• The workflow should keep editable Excel copies for review. Use ameriflux-surgo-grid-extract/scripts/grid_netcdf_excel_bridge.py for grid NetCDF, ecosim-natural-plant-mgmt/scripts/plant_mgmt_excel_bridge.py for PFT management, and ecosim-soil-mgmt/scripts/soil_mgmt_excel_bridge.py for soil management.

The script will create the unified JSON output, generate any requested NetCDF forcing files and namelist, and print confirmation messages. Site-specific outputs should be placed under result/<SITE_ID>/.

When climate or grid NetCDF files are generated, the workflow should also emit derivation report JSON files that list template variables which could not be derived from source data and were left as fill/default values. Do not include the optional atmospheric gas scalar variables OXYEG/OXYGE, Z2GEG, CO2EIG, CH4EG, Z2OEG, or ZNH3EG in those gap lists.

Namelist generation must fail loudly if grid, climate, plant-management, PFT-parameter, or atmospheric GHG inputs are unavailable. Do not set missing grid, climate, or plant-management files to NO; that creates an EcoSIM setup with invalid forcing or no active vegetation management. Validate the chosen PFT parameter catalog against the plant-management PFT code before using the namelist for a run.

Climate NetCDF generation must apply range-aware gap filling:

• ERA5 meteorology values outside legitimate physical bounds are masked and filled by time interpolation before hourly aggregation.
• NADP chemistry values must be finite and non-negative, with PHRG constrained to pH 0-14.
• Annual chemistry gaps are filled by linear interpolation with nearest-edge filling.
• If PHRG cannot be derived from NADP data, write PHRG=7 and record the policy fallback in the derivation report.

Note: create_ecosim_climate_forcing.py and create_ecosim_grid_forcing.py have been moved to the Tools/ directory. All internal calls have been updated accordingly.