| name | scientific-data-parser |
| description | Parse specialized scientific data formats (e.g., HDF5, NetCDF, FITS, mzML, CIF, PDB) or unfamiliar raw data files. Use when the user wants to process complex scientific files into AI-Ready formats. |
Scientific Data Parsing SOP
Scientific data often contains a mix of structured numerical arrays and unstructured metadata/remarks. Follow this workflow to ensure ZERO data loss.
Phase 1: Exploration & Dependency Management
- Never use
read_file on raw data.
- Text vs Binary:
- For text-based formats (CIF, PDB, XML, JSON, CSV, unfamiliar txt): Use the
read_file_lines tool (e.g., lines 1-100) to understand the schema and locate metadata blocks.
- For binary formats (HDF5
.h5, NetCDF .nc, FITS .fits), do NOT read them as text.
- Install Dependencies: If the format requires a specific library, use
python_exec to install it first.
- HDF5 ->
!pip install h5py
- NetCDF ->
!pip install netCDF4
- FITS/Astronomy ->
!pip install astropy
- Crystallography (CIF) ->
!pip install pymatgen
Phase 2: Parser Script Generation with Strict Feedback Loop
When writing the Python parsing script (via python_exec), you must adhere to the Zero Data Loss Rule:
- Read
skills/scientific-data-parser/scripts/template_science.py for the standard architecture.
- Your script must proactively catch any unparsed items, unmatched regex groups, or trailing metadata/remarks.
- Raise Warnings: If the script finds extra columns, floating text, or undocumented metadata, it MUST print a warning to
stderr or stdout in this format:
[UNPARSED_WARNING] File: <path>, Line/Key: <location>, Content: <content>
Phase 3: Evaluate & Iterate
- After running the script, carefully read the output from
python_exec.
- If you see
[UNPARSED_WARNING], it means your script missed valuable scientific context.
- Self-Correction: Rewrite the script to explicitly extract those missed items, map them to appropriate JSON fields (like
metadata or operator_notes), and run it again until no warnings are generated.
- Output the final data using the standard dual-output format (.md and .jsonl).
