reana-operator

name	reana-operator
description	Use when operating REANA from natural language: check job status, list available backends, show recent jobs by status, scaffold reana.yaml projects, run code as REANA workflows, inspect logs, validate YAML, and download outputs using REANA_SERVER_URL and REANA_ACCESS_TOKEN from the environment.
version	1.0.0
author	AstroAgent / AIP
license	MIT
metadata	{"hermes":{"tags":["reana","workflow","operator","jobs","status","reproducibility"],"category":"reana-workflows","related_skills":["reana-client-config","reana-client-docker","reana-serial-python","reana-workflow-best-practices","reana-run-script-with-workspace"]}}

REANA Operator

Overview

This is the front-door skill for day-to-day REANA operations. Use it to translate user requests into safe, concrete REANA actions: check workflow status, list recent jobs, inspect failed logs, show the active backend, scaffold a new project, generate reana.yaml, submit a workflow, validate inputs, and download outputs.

The implementation assumes credentials are already present in the shell environment:

export REANA_SERVER_URL="https://..."
export REANA_ACCESS_TOKEN="..."

Do not ask the user to paste tokens into files. Do not write tokens into reana.yaml, logs, README files, or commits.

When to Use

Use this skill when the user asks any of the following:

• “What is the status of my REANA job?”
• “Show recent REANA jobs.”
• “List failed / running / pending / successful workflows.”
• “Which REANA backend am I using?”
• “Can REANA connect?”
• “Run this Python code on REANA.”
• “Create a REANA job for this project.”
• “Generate an appropriate reana.yaml.”
• “I have a task; make sure it runs on REANA.”
• “Choose the REANA environment based on available libraries.”
• “Show logs for the failed job.”
• “Download outputs from workflow X.”

When the user gives an executable analysis/science task, prefer the REANA path by default: create a small task project, generate task-specific reana.yaml, validate it, and submit or give the exact run command. Use local execution only for quick inspection, formatting, or repository maintenance that is not intended as the user’s scientific/computational result.

Helper CLI

This skill ships a helper:

python reana-workflows/reana-operator/scripts/reana_operator.py --help

It prefers a native reana-client if installed. Otherwise it uses Docker with the Docker Hub image documented at https://hub.docker.com/r/reanahub/reana-client:

REANA_CLIENT_MODE=${REANA_CLIENT_MODE:-auto}     # auto | native | docker
REANA_CLIENT_IMAGE=${REANA_CLIENT_IMAGE:-reanahub/reana-client:0.95.0-alpha.3}

Use REANA_CLIENT_MODE=docker to force the Dockerized client for testing or reproducibility.

The helper never prints REANA_ACCESS_TOKEN.

Quick Commands

Check client/backend/credentials

python reana-workflows/reana-operator/scripts/reana_operator.py client
python reana-workflows/reana-operator/scripts/reana_operator.py backends
python reana-workflows/reana-operator/scripts/reana_operator.py envs
python reana-workflows/reana-operator/scripts/reana_operator.py ping

Recent jobs

python reana-workflows/reana-operator/scripts/reana_operator.py recent --limit 10
python reana-workflows/reana-operator/scripts/reana_operator.py recent --status failed --limit 10
python reana-workflows/reana-operator/scripts/reana_operator.py recent --status running --limit 10
python reana-workflows/reana-operator/scripts/reana_operator.py recent --status success --limit 10
python reana-workflows/reana-operator/scripts/reana_operator.py recent --status pending --limit 10

Status aliases are normalized:

User wording	REANA statuses checked
success, successful, finished	finished, succeeded, success
failed, error	failed, failure, error
pending, queued	created, queued, pending
running, active	running, active
stopped, cancelled	stopped, cancelled, canceled, deleted

Job status and logs

python reana-workflows/reana-operator/scripts/reana_operator.py status <workflow-name-or-id>
python reana-workflows/reana-operator/scripts/reana_operator.py logs <workflow-name-or-id> --tail 100

If a workflow failed, fetch logs immediately and summarize the likely failure class: missing input, YAML validation, missing package, environment mismatch, timeout, memory, or remote-data access.

Download outputs

python reana-workflows/reana-operator/scripts/reana_operator.py download <workflow-name-or-id> --out outputs/<workflow>

Creating a REANA Job from Code

Task-first REANA mode

For user tasks that produce a computational result, use task as the friendly front door. It creates a clean project, detects Python imports, chooses a modeled REANA environment profile, writes reana-env-report.md, generates reana.yaml, validates the project, and can submit immediately.

python reana-workflows/reana-operator/scripts/reana_operator.py task \
  --project /tmp/my-reana-task \
  --task "short description of the user task" \
  --code 'from pathlib import Path; Path("output.txt").write_text("hello\n")' \
  --output output.txt \
  --environment-profile astro-ml \
  --run --timestamp

If --script already exists and neither --code nor --command is supplied, task preserves that script and only generates reana.yaml plus the environment report. If the script does not exist, the helper writes a safe analysis.py placeholder; replace it before submission.

Environment-aware YAML generation

The helper has modeled AIP REANA environment profiles:

Profile	Image	Use for
astro-ml	gitlab-p4n.aip.de:5005/p4nreana/reana-env:py311-astro-ml.2891a60c	default scientific Python, astronomy, data analysis, ML-adjacent tasks
astro	gitlab-p4n.aip.de:5005/p4nreana/reana-env:py311-astro	astronomy tasks without heavy ML dependencies
python	python:3.11-slim	minimal public Python image; install explicit packages

Run:

python reana-workflows/reana-operator/scripts/reana_operator.py envs

Generation logic:

1. Parse Python imports from the selected script.
2. Read requirements.txt if present.
3. Add repeated --package <name> values from the user/task.
4. Compare requested packages against the modeled libraries available in the selected profile.
5. Generate commands that install only missing packages at runtime, e.g. pip install --quiet healpy && python3 analysis.py.
6. Write reana-env-report.md explaining imports, standard-library modules, profile-provided packages, and runtime-installed packages.

This keeps reana.yaml specific to the task while still using known AIP reana-env images when libraries are already available. Treat the profile library list as a curated availability model, not a formal package-lock file; run a smoke test when exact versions matter.

For a quick Python job:

python reana-workflows/reana-operator/scripts/reana_operator.py scaffold   --project /tmp/reana-demo   --script analysis.py   --code 'print("hello REANA"); open("output.txt", "w").write("hello\n")'   --output output.txt

python reana-workflows/reana-operator/scripts/reana_operator.py validate --project /tmp/reana-demo
python reana-workflows/reana-operator/scripts/reana_operator.py run --project /tmp/reana-demo --workflow reana-demo --timestamp

For an existing project:

python reana-workflows/reana-operator/scripts/reana_operator.py scaffold   --project ./my-analysis   --script analysis.py   --output result.png   --output summary.csv

The scaffold creates:

my-analysis/
  reana.yaml
  .reanaignore
  analysis.py          # only if --code was passed or a missing script should be stubbed
  requirements.txt     # optional, used if present

reana.yaml Defaults

The generated workflow uses the AIP-compatible serial pattern:

version: 0.9.0
inputs:
  files:
    - analysis.py
    - requirements.txt
workflow:
  type: serial
  specification:
    steps:
      - name: my-analysis
        environment: gitlab-p4n.aip.de:5005/p4nreana/reana-env:py311-astro-ml.2891a60c
        kubernetes_memory_limit: "32Gi"
        kubernetes_job_timeout: 7200
        compute_backend: kubernetes
        commands:
          - bash -lc 'if [ -f requirements.txt ]; then pip install --quiet -r requirements.txt; fi && python3 analysis.py'
outputs:
  files:
    - output.txt

Adjust the environment with --environment only when a known approved image is required.

Project-Type Handling

Supported now:

Input	Behavior
task --task ...	create task-specific REANA project and YAML; optionally submit with --run
--script analysis.py	parse imports, model package availability, run python3 analysis.py
--script run.sh	run bash run.sh
--command '...'	run the explicit shell command, with needed install prefix when detected
--code '...'	write code into the script, scaffold, validate, then run
--package healpy	declare extra dependency; install only if not modeled as available in the selected env profile

Planned/advanced cases to handle manually for now:

Project type	Recommended handling
notebook .ipynb	convert to a Python script or use a verified Jupyter-capable REANA image
Snakefile	use only after confirming a Snakemake-capable environment
Makefile	use --command 'make <target>' and declare outputs explicitly
large input data	stage only small scripts/config; fetch remote data inside the workflow

Validation Checklist

Before running, the helper checks:

• reana.yaml exists and parses as YAML
• all inputs.files exist locally
• no token/access-token text appears in reana.yaml
• outputs are declared
• memory is set to a 32 GiB/GB default
• no inline heredoc script block is present

Run explicitly:

python reana-workflows/reana-operator/scripts/reana_operator.py validate --project ./my-analysis

Workspace Hygiene

reana-client run may upload local project content. Keep projects small and explicit.

The scaffold writes .reanaignore with common exclusions:

.git/
.env
.reana/
__pycache__/
*.pem
*.key
*token*
*secret*
*password*

Before submitting, inspect the project tree and remove large or private files.

Operational Recipes

“What backend am I using?”

1. Run backends.
2. Report REANA_SERVER_URL, whether the token is set, client mode, and any local config profiles.
3. If credentials are set, run ping and report server/client versions.

“Show recent failed jobs”

python reana-workflows/reana-operator/scripts/reana_operator.py recent --status failed --limit 10

If exactly one recent failed job is relevant, follow with:

python reana-workflows/reana-operator/scripts/reana_operator.py logs <workflow> --tail 120

“Run this code on REANA”

1. Create a clean project directory.
2. Write the code to analysis.py.
3. Parse imports and choose --environment-profile astro-ml unless the task clearly needs another approved profile.
4. Generate reana-env-report.md and task-specific reana.yaml with declared outputs.
5. Validate.
6. Submit with task --run --timestamp or run --timestamp.
7. Return status/log/download commands.

“Create a REANA project for this repository”

1. Detect the main script or ask only if ambiguous.
2. Add/keep requirements.txt if present.
3. Generate reana.yaml with the script as an input file.
4. Validate all input files.
5. Do not submit until the user confirms if project upload size or output names are unclear.

Common Pitfalls

1. Credentials in files — never write REANA_ACCESS_TOKEN or access tokens into reana.yaml, .reana/config.yaml in a public repo, README, or logs.
2. Wrong working directory — always run scripts from $REANA_WORKSPACE inside REANA.
3. Missing input file — any script referenced by the command must be declared under inputs.files and exist locally.
4. Inline YAML scripts — avoid heredocs inside reana.yaml; use separate analysis.py or run.sh files.
5. Huge uploads — do not submit a full repository with large data, .git/, caches, or credentials. Use remote data access inside the workflow.
6. Output not declared — REANA will not expose artifacts reliably unless outputs are listed.
7. Ambiguous workflow names — use --timestamp for new submissions to avoid collisions.
8. Unverified environment images — only change --environment to a known REANA image.

Verification

• python scripts/reana_operator.py --help works.
• scaffold creates reana.yaml and .reanaignore.
• validate passes on the generated scaffold.
• backends reports unset/set credentials without exposing token values.
• With real environment variables set, ping connects to REANA.
• After run, status, logs, and download commands operate on the submitted workflow.