Cache selected columns from the ShBoost 2024 S3 dataset, create a hexbin CMD with log-scaled density, add optimized non‑overlapping stellar‑population annotations, and generate both a high‑resolution PNG and an MP4 animation.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
shboost_cmd_plot_and_animation
title
ShBoost CMD Plot and Population Animation
description
Cache selected columns from the ShBoost 2024 S3 dataset, create a hexbin CMD with log-scaled density, add optimized non‑overlapping stellar‑population annotations, and generate both a high‑resolution PNG and an MP4 animation.
author
hermes
When to Use
Cache selected columns from the ShBoost 2024 S3 dataset, create a hexbin CMD with log-scaled density, add optimized non‑overlapping stellar‑population annotations, and generate both a high‑resolution PNG and an MP4 animation.
Canonical Routing
This is a specialized or legacy example skill. For new work, start with astro-data-access-umbrella and route through:
s3-parquet-astro-access
astro-catalog-plotting-cache
Keep this skill for dataset-specific examples, but prefer the canonical skills for new implementations, live probes, REANA execution, and plotting/cache conventions.
Pitfalls
Do not hardcode credentials, tokens, or personal secrets.
Verify external service URLs, paths, and permissions before making changes.
Keep generated outputs reproducible and record input assumptions.
Verification
Confirm required inputs and credentials are available.
Run the smallest safe command or example before scaling up.
Check produced files, API responses, or plots before reporting success.
Overview
Automates the workflow for visualising the ShBoost 2024 colour‑magnitude diagram (CMD) from a massive S3 dataset. Handles efficient caching, sampling, hexbin density plotting, colour‑coded annotations, legend creation, and animation generation.
Prerequisites
Python 3.12+ with packages: dask[dataframe], pandas, matplotlib, seaborn.
Access to the public S3 bucket s3://shboost2024/shboost_08july2024_pub.parq/.
ffmpeg (optional; falls back to GIF via PillowWriter).
Constants (adjust as needed)
S3_ENDPOINT = "https://s3.data.aip.de:9000"
S3_PARQUET_GLOB = "s3://shboost2024/shboost_08july2024_pub.parq/*.parquet"
STORAGE_OPTS = {"use_ssl": True, "anon": True, "client_kwargs": {"endpoint_url": S3_ENDPOINT}}
CACHE_PATH = "shboost_full_cmd.parquet"# local cache of selected columns
TARGET_ROWS = 10**12# placeholder – full cache will be written regardless of size
PLOT_ROWS = 10_000_000# rows used for the actual figure (adjust for RAM)
1. Load or Create Cache
defload_or_fetch(force_refresh: bool) -> pd.DataFrame:
if os.path.isfile(CACHE_PATH) andnot force_refresh:
print(f"🔄 Loading cached full data from {CACHE_PATH}")
full_dd = dd.read_parquet(CACHE_PATH)
else:
print("⏬ Reading metadata from S3 …")
full_dd = dd.read_parquet(S3_PARQUET_GLOB, storage_options=STORAGE_OPTS)
full_dd = full_dd[["bprp0", "mg0"]]
total_rows = full_dd.shape[0].compute()
print(f"Dataset size: {total_rows:,} rows (full cache will be written)")
full_dd.to_parquet(CACHE_PATH, write_index=False)
print("Full cache saved.")
full_dd = dd.read_parquet(CACHE_PATH)
# Sample for plotting
total_rows = full_dd.shape[0].compute()
frac = min(PLOT_ROWS / total_rows, 1.0)
print(f"Sampling fraction for plot: {frac:.6%} → ~{min(PLOT_ROWS, total_rows):,} rows")
sample_dd = full_dd if frac >= 1.0else full_dd.sample(frac=frac, random_state=42)
pdf = sample_dd.compute()
print(f"Sampled rows for plot: {len(pdf):,}")
return pdf