Run any Skill in Manus with one click

e2sa-add-adapter

Stars0

Forks0

UpdatedJuly 8, 2026 at 01:04

Implement a new E2SA data-source adapter (the code that fetches a source and parses it into Observation records). Use when writing the adapter for a source that has already been onboarded/inspected — "build the adapter for X", "write the X adapter", "implement the fetch/parse for X", or the coding step after e2sa-add-data-source. Walks the BaseAdapter contract (list_available / fetch / parse_to_schema), the FetchResult single-file-vs-whole-package shape, the Observation + Provenance schema, canonical units + depth + serves, ADAPTER_REGISTRY registration, the fixture + test, and the parse gotchas already learned (sentinels, CRS tiers, obs_id uniqueness, schema drift, idempotency).

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

jingtao-lbl

jingtao-lbl/E2SA-SPADE

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

More from this repository

same repository

e2sa-add-skill

jingtao-lbl/E2SA-SPADE

Scaffold and register a new E2SA skill so the steps are never half-done. Use when the human or agent wants to create a new capability — "add a skill", "create a new skill", "scaffold a skill", "make this reusable as a skill", "distill X into a skill". Creates the SKILL.md with correct frontmatter, registers it in the AGENTS.md capability catalog (the manual registry), seeds a Changelog, and stops for human review before commit. Human-gated.

2026-07-090

e2sa-refine-skill

jingtao-lbl/E2SA-SPADE

Improve an existing E2SA skill from accumulated evidence, human-gated. Use when a skill missed a step or needs updating, when a lesson/feedback in a skill's domain has piled up, or during a periodic skill review — "refine the X skill", "improve this skill", "the X skill should have caught Y", "review the skills". Gathers the signal (lessons, run-journal outcomes, feedback), proposes a concrete SKILL.md diff, and STOPS for human approval; it never self-applies. After approval it edits, appends a Changelog line, and commits.

2026-07-090

e2sa-add-connector

jingtao-lbl/E2SA-SPADE

Implement a new E2SA data-center connector (the per-data-center access layer that owns auth + search + whole-package fetch under Option C). Use when wrapping a data center not yet connected — "add a connector for X", "wrap the PANGAEA/Earthdata/Zenodo API", or when migrating an existing self-fetching adapter onto a connector. Walks the BaseConnector contract, the probe-the-API-first step, the two search patterns (server-side filter vs enrich + bbox coverage-ratio), the two fetch shapes (BagIt-zip vs file-by-file), the reusable conventions (urllib + User-Agent, raw layout, token-free reads, on-disk fast-path, manual fallback), CONNECTOR_REGISTRY registration, and the mocked-urlopen + E2E_LIVE test split. Distilled from the arctic_data_center and ess_dive connectors.

2026-07-080

e2sa-add-data-source

jingtao-lbl/E2SA-SPADE

Add a new data source (data center, archive, or dataset) into an E2SA project. Use when the human or agent has found a dataset to bring in, is inspecting a new archive, or says "add a data source", "add this dataset", "I found data at X", or after downloading data from a center not yet documented. Walks the full procedure: inventory check, download whole package, inspect (format/metadata-standard/CRS/license/citation), write the project-named source card, update the sources index and data-center registry, capture a finding, and credential setup. Always uses the latest dataset version and records a full citation.

2026-07-080

e2sa-audit-adapter

jingtao-lbl/E2SA-SPADE

Audit an E2SA adapter's data output against the REAL source and propose source-grounded corrections (applied only on human approval). Use after building or changing an adapter, after a parse, when staged data looks off, or for a periodic adapter review — "audit/validate the X adapter", "check the data X produces", "is above_stdm correct", "review the adapters for data-quality". Runs the e2sa/qc checks (serves-subset, value-range, subsurface-depth, citation-not-synthesized, self-describing) on a real parse, surfaces the value distributions, cross-checks variables/units/citation against the source, and PROPOSES a source-grounded fix for each finding (never invents) that is applied only on human approval (propose-first, like e2sa-refine-skill). NOT for run-level shippability (run skeleton / notebooks / secrets / a run_id) — that is `e2sa-validate`. Distilled from the 2026-06-23 above_stdm fabrications (reflection 20260623s).

2026-07-080

e2sa-file-task

jingtao-lbl/E2SA-SPADE

File a data-preparation task into projects/<project>/tasks/ when the current work needs site data that isn't already prepared. Use whenever the agent (not just the human) recognizes a need for forcing data, surface dataset assembly, observations, or QC/analysis that warrants its own request with provenance. Always sets status=requested and surfaces to the human; never auto-executes the filed task in the same fire.

2026-07-080

name	e2sa-add-adapter
description	Implement a new E2SA data-source adapter (the code that fetches a source and parses it into Observation records). Use when writing the adapter for a source that has already been onboarded/inspected — "build the adapter for X", "write the X adapter", "implement the fetch/parse for X", or the coding step after e2sa-add-data-source. Walks the BaseAdapter contract (list_available / fetch / parse_to_schema), the FetchResult single-file-vs-whole-package shape, the Observation + Provenance schema, canonical units + depth + serves, ADAPTER_REGISTRY registration, the fixture + test, and the parse gotchas already learned (sentinels, CRS tiers, obs_id uniqueness, schema drift, idempotency).
allowed-tools	["Read","Glob","Grep","Write","Edit","Bash"]

e2sa-add-adapter

The coding step for bringing a source into SPADE: implement a BaseAdapter subclass that emits Observation records, and wire it in. This is the downstream half of onboarding — e2sa-add-data-source does the look-first inspection + source card; this skill writes the adapter. Distilled from the six existing adapters (calm, gtnp, alaska_thaw_db, above, plus the connector-backed sloan_2014_barrow_soil and kanevskiy_2024_cryostratigraphy) and the lessons they cost.

Option C default (read this first). A dataset on a data center that has a connector (e2sa/data/connectors/) is a connector-backed adapter: it sets data_center, delegates fetch to the connector, and owns only parse_to_schema + serves. Its source_id IS the dataset_id slug (<owner>[_<year>]_<descriptor>, docs/design/15 §9). If the center has no connector yet, write it first with e2sa-add-connector (and see "Migrating an existing adapter" below). Only legacy/standalone sources still do their own fetch.

Read first: e2sa/data/base.py (the contract), e2sa/schema.py (Observation, Variable, CANONICAL_UNITS), one existing adapter closest to your source shape, and the source card under projects/<project>/data/sources/.

When to fire

A source is onboarded (source card exists) and now needs its adapter.
The human says "build/write the X adapter" or "implement fetch/parse for X".
NOT for the inspection/source-card step — that is e2sa-add-data-source.

The interfaces (the contract you implement)

BaseAdapter (e2sa/data/base.py) — set two class attrs and implement three methods:

source_id: str — matches the source card. (Under Option C the raw folder keys on the data center, raw/<data_center>/<dataset_id>/, not on source_id; connector-backed adapters also set data_center and delegate fetch.)
adapter_version: str.
serves: ClassVar[frozenset[Variable]] — the Variables this adapter emits (the capability index routes on it; MUST be a subset of what parse_to_schema actually emits).
data_center: ClassVar[str | None] — set it for a connector-backed adapter (the connector key); leave None only for a legacy standalone source.
list_available() -> list[DatasetInfo] — what datasets/variables the source offers.
fetch(dataset_id) -> FetchResult — connector-backed (the default): do NOT write a fetch method. BaseAdapter.fetch already delegates to your data_center connector (which writes raw_dir/<data_center>/<dataset_id>/). Legacy standalone (no connector): override fetch to download to self.raw_dir.
parse_to_schema(fetch_result) -> list[Observation] — raw files → Observation records (the adapter's real job).

DatasetInfo (returned by list_available): dataset_id, name, description, variables: list[str], spatial_coverage, temporal_coverage, format, url, license, extra.

FetchResult (returned by fetch): dataset_id, local_path, bytes_downloaded, access_timestamp (UTC), content_checksum (sha256), source_url, files: list[Path]. Two shapes: single-file (CALM/GTN-P/Alaska-Thaw/ABoVE) leaves files empty and local_path is the file; whole-package (ESS-DIVE, BagIt) sets local_path to the package root and files to every file. The indexer walks files if non-empty, else local_path.

Observation (emitted by parse_to_schema): obs_id, obs_type (ObservationType: POINT/PROFILE/GRID_CELL/EVENT), variable (a Variable), value: float, unit: str, latitude [-90,90], longitude [-180,180], depth_m (positive downward, None if surface), time_start, time_end, qc_flags, provenance (a Provenance), extra.

Provenance (on every Observation): source_id, source_url, access_timestamp, content_checksum, license, adapter_version, schema_version.

Procedure

Pick the closest existing adapter as a template. Connector-backed (the default): sloan_2014_barrow_soil (ESS-DIVE) or kanevskiy_2024_cryostratigraphy (Arctic Data Center / BagIt). Legacy standalone single-file: calm/gtnp. Match the project's style; do not invent a new shape.
list_available — return DatasetInfo with variables, spatial_coverage, temporal_coverage, license, url and citation + references + keywords populated. These feed the self-describing metadata bundle acquire() writes into the staged folder (PROVENANCE.json/CITATION.cff/README.md, doc 18) — a bare data file is not an acceptable staged artifact. Take the citation from the source card / the source's native metadata; do not fabricate authors (leave citation=None and rely on the native record if unsure). (Discovery across a center is the connector's search; list_available returns this adapter's known dataset(s).)
fetch — connector-backed: just set data_center; do not write a fetch method (BaseAdapter's default delegates to the connector, which owns download + idempotency + the raw_dir/<data_center>/<dataset_id>/ layout). If the center has no connector, write it first with e2sa-add-connector. Legacy standalone only: override fetch to download into self.raw_dir, set the right FetchResult shape, sha256 content_checksum, UTC access_timestamp, on-disk fast-path, idempotent.
parse_to_schema — map source columns/variables to Variable members and emit Observations. Apply the gotchas below. Set obs_type correctly (PROFILE for depth series, POINT for surface, GRID_CELL for raster, EVENT for episodic). Attach full Provenance.
Declare serves = the Variables actually emitted (derive from a var-map where possible, like above, so it cannot drift).
Register — add "<source_id>": <AdapterClass> to ADAPTER_REGISTRY in e2sa/data/registry.py. Non-negotiable: without this acquire()/get_adapter() cannot reach it.
Fixture + test — commit a tiny real-shaped fixture to tests/fixtures/; add a test that parses it and asserts schema fields, the gotchas, serves ⊆ emitted, and unit == CANONICAL_UNITS[variable]. Also assert validate_observations(adapter.serves, observations) returns no error-severity findings (the QC layer acquire(parse=True) runs; e2sa/qc/checks.py), so a new adapter is QC-clean by construction. The fixture MUST mirror the source's real units/conventions, not the code's assumption — a fixture built to match the adapter can hide a real bug (the above_stdm VWC fixture stored fraction-style values and masked a percent-vs-fraction bug; 20260629e).
Index — the package is indexable by index_package (e2sa/data/indexing.py); no per-adapter indexing code.

Migrating an existing self-fetching adapter to a connector (Option C refactor)

The recipe for moving a legacy adapter (calm/gtnp/above/alaska_thaw_db) onto a connector — proven on the ess_dive/sloan_2014_barrow_soil split (dev log 20260623e):

Connector first. If the data center has no connector, write it with e2sa-add-connector, moving the adapter's existing fetch/auth body into the connector's fetch (move, don't rewrite — the download already works). One connector can back several adapters (PANGAEA → both CALM and GTN-P).
Point the adapter at it. Set data_center; delete the adapter's fetch method (it inherits BaseAdapter's connector-delegating default); keep parse_to_schema + serves.
Rename to the slug. source_id = dataset_id = the docs/design/15 §9 slug (calm → calm_alt, gtnp → gtnp_magt). Update ADAPTER_REGISTRY, the source card, the raw-folder note, and any test/label references.
Split the tests. Connector-level (fetch/auth, mocked urlopen) → tests/test_<center>_connector.py; adapter-level (parse/serves/delegation) → tests/test_<slug>_adapter.py.
Update docs. docs/design/13 inventory row + the README data-center registry. Keep the suite green at each step.

Parse gotchas (each one cost a real bug)

Missing-value sentinels — read the per-file sentinel (-9999, blank, not_determined); never treat it as data. 0 is often meaningful (EIC=0 ≠ missing).
Units → canonical — emit Observation.unit as the canonical unit for the variable (CANONICAL_UNITS, e2sa/schema.py); convert at parse (e.g. EIC % → fraction /100, ALT cm → m). Do not store the raw source unit.
Depth — for subsurface variables (soil/ground temperature, ice content, moisture) populate depth_m (convert to metres; parse from a column or a filename like ..._{N}cm_...). A subsurface reading without depth is half a measurement.
obs_id uniqueness — must be unique per (station/site, depth, time). An event/label alone collides across stations (GTN-P: an event-only id collapsed 4,088 rows to 715). Include station name + lat/lon + depth + the full date.
CRS tiered fallback — machine-readable metadata → PDF user-file → assume WGS84 with a low-confidence flag; record which tier. Reproject to WGS84.
Column-name drift / aliasing — map by concept with OR-chained fallbacks, not one exact string (2025 PANGAEA renamed Event/MAGT/Date/Time; ABoVE has _VAR_MAP). Check the live schema, not just the fixture.
Filename vs metadata mismatch — for BagIt/EML, match files by manifest-md5.txt checksum, not the declared <entityName>.
Encoding — try UTF-8, fall back to latin-1 on legend/mojibake rows.
Blank rows as separators — do not treat them as EOF; they may delimit boreholes within one file.

Guardrails

Register in ADAPTER_REGISTRY or the adapter is unreachable.
serves ⊆ what parse_to_schema emits (tested); declare what you emit, not what you wish you served.
Canonical units + depth_m for subsurface variables; full Provenance on every Observation; raw is immutable; no secrets in the repo.
Fixture + test required; idempotent fetch; on-disk fast-path where downloads are large.
One adapter module per dataset under e2sa/data/adapters/<dataset_id>.py — the file name matches the slug (source_id = dataset_id = the ADAPTER_REGISTRY key = the file name), so the directory is self-documenting. Per-data-center connectors live in e2sa/data/connectors/. The interface is fixed — add functionality inside it, not in the caller.

Changelog

2026-06-22: Initial version. Split out of e2sa-add-data-source step 8 (which is now a handoff). Distilled the BaseAdapter/FetchResult/Observation interfaces and the parse gotchas (sentinels, units, depth, obs_id uniqueness, CRS tiers, schema drift, filename-vs-md5, encoding) from the five existing adapters + captured lessons.
2026-06-23: list_available step now requires populating DatasetInfo.citation + references + keywords (they feed the self-describing metadata bundle acquire() writes, doc 18); don't fabricate authors.
2026-06-23: Updated for the Option C connector era. Connector-backed is now the default (set data_center, delegate fetch to get_connector, source_id == dataset_id slug); added the data_center contract attr, the delegate one-liner, the "Migrating an existing self-fetching adapter to a connector" recipe (the Phase 2 refactor checklist), and a handoff to the new e2sa-add-connector. Refreshed stale references (six adapters; the built ESS-DIVE/ADC connector-backed templates).
2026-06-23: Phase 3 cleanup — BaseAdapter.fetch now PROVIDES the connector-delegating default, so a connector-backed adapter writes NO fetch method (just data_center + list_available + parse_to_schema). Adapter file name now matches the slug (e2sa/data/adapters/<dataset_id>.py).
2026-06-30: Step 7 now requires the fixture test assert validate_observations returns no error-severity findings (QC-clean by construction; the layer acquire(parse=True) runs), and that the fixture mirror the source's real units, not the code's assumption (Josue handoff #5, proposed in 20260626a; motivated by the above_stdm VWC fixture that masked a bug, 20260629e). Dropped the now-stale "once units land" qualifier (canonical units shipped 2026-06-29).