airtable-register

name	airtable-register
description	Register zarr positions into the Computational Imaging Database on Airtable, write channels_metadata/experiment_metadata to zarr .zattrs, or bulk-update Airtable records via MCP. Use when the user asks to "register a dataset", "register zarr positions", "update airtable from zarr", "write metadata to zarr", "run register on", "sync airtable", "populate channel markers", "update airtable records", "backfill fields", or "fill in missing fields". Also use for Marker Registry questions.
version	3.0.0
author	ai-x-imaging
tags	["Airtable","OME-Zarr","Metadata","Registration","DynaCLR","VisCy"]

Airtable Registration & Update Skill

Manages bidirectional metadata sync between OME-Zarr datasets and the Computational Imaging Database on Airtable, and supports bulk field updates via MCP.

Airtable Configuration

• Base ID: app8vqaoWyOwa0sB5 (Computational Imaging Database)
• Datasets table ID: tblaFzrDMlVZHPZIj
• Collections table ID: tblu0Rbj9OnLl7vJf
• Models table ID: tblVZhRA48tDMWj8U
• Marker Registry table: tblmP8l2GmpCeERyD
• Script: applications/airtable/scripts/write_experiment_metadata.py
• Core logic: applications/airtable/src/airtable_utils/registration.py
• Schemas: applications/airtable/src/airtable_utils/schemas.py
• Database interface: applications/airtable/src/airtable_utils/database.py
• Channel parsing: packages/viscy-data/src/viscy_data/channel_utils.py
• MAX_CHANNELS = 8 (defined in schemas.py)
• AIRTABLE_API_KEY and AIRTABLE_BASE_ID must be set in environment

Operations

1. Register (zarr -> Airtable)

Reads zarr metadata and writes per-FOV records to Airtable.

Fields written by register:

• data_path — full path to zarr position
• channel_{i}_name — zarr channel names (up to 8)
• channel_{i}_marker — protein marker, derived from Marker Registry
• t_shape, c_shape, z_shape, y_shape, x_shape — array dimensions
• pixel_size_xy_um, pixel_size_z_um — from zarr coordinate transforms

Marker derivation rules:

• labelfree channels -> marker = channel name (e.g. "Phase3D", "BF", "DIC")
• virtual_stain channels -> marker = channel name (e.g. "nuclei_prediction")
• fluorescence channels -> substring-match aliases against channel name -> protein marker from registry (e.g. "TOMM20", "SEC61B")

Commands

# Dry run first (always recommended)
uv run --package airtable-utils \
    applications/airtable/scripts/write_experiment_metadata.py \
    register --dry-run /path/to/dataset.zarr/*/*/*

# Register all positions
uv run --package airtable-utils \
    applications/airtable/scripts/write_experiment_metadata.py \
    register /path/to/dataset.zarr/*/*/*

# Single position
uv run --package airtable-utils \
    applications/airtable/scripts/write_experiment_metadata.py \
    register /path/to/dataset.zarr/A/1/000000

# Override dataset name (when zarr stem doesn't match Airtable)
uv run --package airtable-utils \
    applications/airtable/scripts/write_experiment_metadata.py \
    register --dataset my_dataset /path/to/dataset.zarr/*/*/*

Parquet Readiness Report

After registration, the CLI prints a Parquet Readiness report that flags any fields still needed before a flat parquet cell index can be built. Fields are split by source:

• zarr fields (auto-filled by register): data_path, channel_N_name, channel_N_marker, pixel_size_xy_um, pixel_size_z_um
• platemap fields (biologist fills in Airtable): tracks_path, perturbation, time_interval_min, hours_post_perturbation, cell_type

If any platemap fields are missing, the report shows what to fill in and how (Airtable UI or MCP bulk update).

2. Write (Airtable -> zarr)

Writes channels_metadata and experiment_metadata to zarr .zattrs.

uv run --package airtable-utils \
    applications/airtable/scripts/write_experiment_metadata.py \
    write /path/to/dataset.zarr/*/*/*

channels_metadata schema

{
  "Phase3D": {
    "channel_type": "labelfree",
    "biological_annotation": {"marker": "Phase3D"}
  },
  "raw GFP EX488 EM525-45": {
    "channel_type": "fluorescence",
    "biological_annotation": {
      "marker": "TOMM20",
      "marker_type": "protein_tag",
      "fluorophore": null
    }
  }
}

experiment_metadata schema

{
  "perturbations": [
    {"name": "ZIKV", "type": "unknown", "hours_post": 48.0, "moi": 5.0}
  ],
  "time_sampling_minutes": 15.0
}

3. Bulk Update (via Airtable MCP)

For updating fields that don't come from zarr (e.g. tracks_path, organelle, manually-curated fields).

Process:

1. Fetch target records with mcp__airtable__list_records using filterByFormula
2. Compute new values (python/jq)
3. Batch update with mcp__airtable__update_records (max 10 per call, send all batches in parallel)
4. Verify with the same filter query (must return zero remaining records)

Pagination warning: mcp__airtable__list_records returns ~100 records max per call. If count equals ~100, re-query with tighter filters.

Datasets Table Fields

Field	Description	Written by register?
dataset	Experiment name	on create
well_id	Well path (e.g. B/2)	on create
fov	FOV identifier	on create
data_path	Path to HCS OME-Zarr position	yes
tracks_path	Path to tracking zarr	no (manual/MCP)
channel_0_name .. channel_7_name	Zarr channel names	yes
channel_0_marker .. channel_7_marker	Protein marker per channel	yes
t_shape .. x_shape	Array dimensions	yes
pixel_size_xy_um	Physical XY pixel size (um)	yes
pixel_size_z_um	Physical Z pixel size (um)	yes
marker	Well-level primary marker	template copy
organelle	Target organelle	template copy
perturbation	Perturbation applied	template copy
cell_type	Cell type (e.g. A549)	template copy
cell_state	Condition label	template copy
time_interval_min	Minutes between frames	template copy
hours_post_perturbation	HPI at imaging start	template copy
moi	Multiplicity of infection	template copy
fluorescence_modality	Imaging modality	template copy

Marker Registry

Table tblmP8l2GmpCeERyD — maps constructs to protein markers.

Field	Type	Example
marker-fluorophore	text (primary)	TOMM20-GFP
channel_name_aliases	text	GFP, FITC
marker	text	TOMM20

Matching is substring-based: "GFP" in "raw GFP EX488 EM525-45" -> match.

Flat Parquet Alignment

The register command writes all fields needed to build a flat parquet cell index:

• channel_{i}_name -> parquet channel_name
• channel_{i}_marker -> parquet marker
• pixel_size_xy_um, pixel_size_z_um -> parquet pixel size columns
• data_path -> parquet store_path

Dataset Directory Conventions

For organelle_dynamics datasets:

data_path:    /hpc/projects/intracellular_dashboard/organelle_dynamics/{EXP}/2-assemble/{EXP}.zarr
tracks_path:  /hpc/projects/intracellular_dashboard/organelle_dynamics/{EXP}/1-preprocess/label-free/3-track/{EXP}_cropped.zarr

Other families (organelle_box, viral-sensor) have non-standard structures — check filesystem.

Example Invocations

• "register this dataset /path/to/dataset.zarr///*"
• "write metadata to zarr for dataset X"
• "update tracks_path for all organelle_dynamics datasets"
• "fill in pixel_size_xy_um for all records where it's missing"
• "set organelle = 'mitochondria' for all 2024_11_21 records"