Reusable template for REANA serial workflows that run a Python analysis script on remote data, cache processed results locally as Parquet, and produce PNG outputs. Designed for SHBoost-like analyses where only the script, selected columns, and output names change.
Reusable template for REANA serial workflows that run a Python analysis script on remote data, cache processed results locally as Parquet, and produce PNG outputs. Designed for SHBoost-like analyses where only the script, selected columns, and output names change.
author
Hermi (sorgenfresser)
Overview
This skill contains a reusable operational workflow. Follow the existing task-specific steps and examples in the sections below.
Canonical Routing
This is a specialized or legacy example skill. For new work, start with astro-data-access-umbrella and route through:
reana-operator
s3-parquet-astro-access
Keep this skill for dataset-specific examples, but prefer the canonical skills for new implementations, live probes, REANA execution, and plotting/cache conventions.
Pitfalls
Do not hardcode credentials, tokens, or personal secrets.
Verify external service URLs, paths, and permissions before making changes.
Keep generated outputs reproducible and record input assumptions.
Verification
Confirm required inputs and credentials are available.
Run the smallest safe command or example before scaling up.
Check produced files, API responses, or plots before reporting success.
Purpose
Use this template whenever the analysis pattern is the same:
one Python script does the analysis
data are read remotely at runtime (for example from S3)
only selected columns or filtering logic change
processed/sample data are cached locally as Parquet
final outputs are PNG and optional JSON summaries
execution happens in a REANA serial workflow
This template is intentionally generic so it can be reused for CMDs, alternative column projections, density plots, selections, and similar analyses.
Key rules
Use reana-client run -w <workflow-name> whenever possible.
If using the Dockerized REANA client, always mount the local workflow directory and set the container working directory to /workspace.
Keep all edits in the same project folder and rerun from there.
Use an approved REANA environment only.
Default memory is 32 GB.
Cache processed data locally as Parquet.
Prefer PNG outputs.
Project layout
Example local project directory:
my_analysis/
reana.yaml
analysis.py
Generic Python script template
Replace placeholders such as S3 path, selected columns, filters, and plot logic:
If you forget -w /workspace on the Docker container, reana-client run may upload reana.yaml but fail to upload or resolve the analysis script correctly.
SHBoost-specific note
For the public SHBoost 2024 dataset, use the notebook-backed access pattern:
