Hunt down CoreWeave GPU cost leaks — idle reserved capacity, wrong-GPU-type
right-sizing waste, allocated-but-idle instances, and on-demand spend that should
be committed — then produce a CFO-grokkable, dollar-ranked FinOps report. CoreWeave
ships no cost dashboard and no billing API, so the spend view is built from PromQL
against its managed Grafana. Use when a user asks why their CoreWeave GPU bill is
high, wants to find wasted GPU spend or idle reservations, or needs a GPU FinOps
cost report. Trigger with "coreweave cost", "why is my coreweave bill",
"wasted GPU spend", "idle reserved capacity", "GPU cost leak".
Instrucciones de origen · Vista previa de solo lectura
name
coreweave-gpu-cost-leak-hunter
description
Hunt down CoreWeave GPU cost leaks — idle reserved capacity, wrong-GPU-type
right-sizing waste, allocated-but-idle instances, and on-demand spend that should
be committed — then produce a CFO-grokkable, dollar-ranked FinOps report. CoreWeave
ships no cost dashboard and no billing API, so the spend view is built from PromQL
against its managed Grafana. Use when a user asks why their CoreWeave GPU bill is
high, wants to find wasted GPU spend or idle reservations, or needs a GPU FinOps
cost report. Trigger with "coreweave cost", "why is my coreweave bill",
"wasted GPU spend", "idle reserved capacity", "GPU cost leak".
Community-contributed. Not affiliated with, endorsed by, or sponsored by
CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.
Audits a CoreWeave GPU cluster for real-dollar cost leaks — idle reserved capacity,
GPUs on the wrong SKU, allocated-but-idle instances, and steady on-demand spend that
should be committed — then emits a CFO-grokkable, dollar-ranked FinOps report.
Overview
CoreWeave ships no cost dashboard and no billing API (usage-monitoring
docs). There is also no single "dollars" metric — spend is reconstructed by
querying usage from CoreWeave's managed Grafana in PromQL and multiplying each
resource's usage by its rate-card price. This skill does exactly that, then ranks the
leaks by monthly dollar impact.
The math is deterministic: PromQL returns usage counts, and the bundled
scripts/rank-and-report.py does every multiplication, sum, and ranking — the agent
never eyeballs a number. Two of the four categories are billed waste (Confirmed);
the other two are a right-sizing model (Estimated) and a commitment decision
(At-risk), labeled so a CFO never reads a modeled number as recoverable cash.
Deep domain knowledge lives in references/, loaded only when a leak needs it.
Prerequisites
CoreWeave managed Grafana access — the Prometheus data source is reachable only
to a member of the admin, metrics, or write group in the CoreWeave Cloud
Console (usage-monitoring docs). This is the hard dependency; Step 1 probes it
and fails fast if the group is missing.
A Prometheus/Grafana query endpoint in $CW_PROM_URL (the Grafana data-source
proxy, e.g. https://grafana.ORG.coreweave.com/api/datasources/proxy/uid/UID)
and a bearer token in $CW_TOKEN for curl.
kubeconfig for the cluster (CoreWeave-issued) so kubectl get can corroborate
live GPU allocation and node labels.
The rate card — CoreWeave publishes no price metric, so on-demand and committed
rates are supplied to the ranker from references/gpu-right-sizing.md (dated
snapshot of coreweave.com/pricing) or the customer's contract.
jq and python3 for parsing query JSON and running the ranker.
Authentication. All auth comes from the environment ($CW_PROM_URL, $CW_TOKEN,
$KUBECONFIG) — no secrets are hardcoded. Grafana enforces the group membership above
on every query.
Instructions
The pipeline is detect → price → rank → report. PromQL returns usage; the dollar
arithmetic runs in scripts/; deep knowledge loads from references/ on demand:
Verify metric access, fail fast if the group is missing.
Pull the 30-day spend baseline (usage × rate card).
Detect Leak 4 — on-demand spend that should be committed (At-risk).
Rank by monthly dollar impact and render the CFO report.
Step 1: Verify Metric Access (fail fast, not mid-flow)
Probe billing:instance:total before anything else. An HTTP 401/403 or empty result
means the token's principal is not in admin/metrics/write — STOP and report it;
do not continue into the scans.
If status is not success with a non-empty result, load
references/promql-billing-setup.md and report
the missing group access verbatim. Stop here.
Step 2: Pull the Spend Baseline
Reconstruct 30-day GPU node-hours per instance type. CoreWeave has no dollars metric,
so this returns usage — the ranker multiplies by the rate card. Write the JSON to
the working dir for the ranker.
The rate card and the per-category PromQL live in
references/gpu-cost-leak-categories.md.
Load it now — the four scans below reference its recording-rule notes.
Reserved GPUs bill at the committed rate whether used or not. A reserved GPU
sitting below a utilization floor is confirmed waste — you paid for it and it did no
work. Cross reserved allocation (billing_gpu, filtered by the reservation label)
against SM-active from DCGM.
curl -sS -H "Authorization: Bearer $CW_TOKEN" --data-urlencode \
'query=sum by (instance_type,node) (avg_over_time(billing_gpu{reservation!=""}[30d:1h]))
and on(node) (avg by (node) (avg_over_time(DCGM_FI_PROF_SM_ACTIVE[30d:1h])) < 0.05)' \
"$CW_PROM_URL/api/v1/query" > "$OUT/leak1-idle-reserved.json"
The reservation label key is provider-specific — confirm yours with kubectl get nodes --show-labels. Waste = idle reserved GPU-hours × committed rate (ranker input).
H100/H200 running small-model (~7B–30B) inference is over-paying: for that regime
L40S is cheaper per token (directional — see gpu-right-sizing.md). Flag those
instance-hours; the ranker re-prices them at the L40S rate.
This is Estimated: the rate delta is exact rate-card math, but throughput
equivalence on L40S is a model. Confirm the served model size with the cluster owner
before acting; FP8 serving needs Hopper/Ada, not Ampere (see gpu-right-sizing.md).
On-demand GPUs that are allocated (billing) but running at low SM-utilization / low
MFU bill the full on-demand rate for no work — confirmed billed waste, the GPU twin of
an idle cluster.
curl -sS -H "Authorization: Bearer $CW_TOKEN" --data-urlencode \
'query=(avg by (node,instance_type) (avg_over_time(DCGM_FI_PROF_SM_ACTIVE[30d:1h])) < 0.05)
and on(node) (sum by (node) (avg_over_time(billing_gpu{reservation=""}[30d:1h])) > 0)' \
"$CW_PROM_URL/api/v1/query" > "$OUT/leak3-idle-ondemand.json"
Corroborate with kubectl get pods -A --field-selector=status.phase=Running to
confirm nothing is actually scheduled on the flagged node. Waste = idle on-demand
GPU-hours × on-demand rate.
Step 6: Leak 4 — On-Demand Spend That Should Be Committed (At-risk)
A stable on-demand floor — GPUs of one type always running across the window — is
paying on-demand for capacity a commitment discounts up to 60% (pricing).
Measure the always-on floor with min_over_time.
This is At-risk: the up-to-60% saving is pending a commitment decision, and a
commitment is itself a paid obligation — see the over-reservation caution in
gpu-cost-leak-categories.md. Savings = floor GPU-hours × on-demand rate × discount.
Step 7: Rank and Write the Report
Assemble one leak object per category from the PromQL usage results plus the rate
card, then pipe them to the deterministic ranker — the LLM does NOT do the arithmetic.
Because CoreWeave exposes no dollars metric, each object carries usage_gpu_hours and
its rate-card rate; the ranker multiplies usage × rate itself (and applies the
re-price or discount factor). Each object's kind (confirmed / estimated /
at-risk) tells the renderer to split the headline confirmed-vs-pending, rank
descending by monthly dollars, and stamp a Confidence column.
OUT="${OUT:-$(pwd)/cost-leak-out}" && mkdir -p "$OUT"# Each Step wrote a leak-N.json {category, root_cause, fix, kind, usage_gpu_hours,# rate_usd_per_gpu_hour, ...}; the ranker does usage × rate deterministically.
jq -s '.'"$OUT"/leak-*.json | \
python3 scripts/rank-and-report.py \
--monthly-spend 180000 --window-end "$WINDOW_END" \
--out "$OUT/cost-leak-report.md"
Render the output using the verbatim template in
references/cfo-output-format.md. Use Glob to
collect the per-leak JSON, Write the report, and Edit it to rescale the headline
spend on request.
Output
A CFO-grokkable report leading with a split headline that never sums
confirmed and unconfirmed dollars under one verb — A $180K/month CoreWeave GPU cluster is burning ~$40K/month (confirmed), plus up to ~$29K/month pending review —
each with a /year companion.
A trailing-30-day window stamp so every figure has an explicit calendar window.
The ranked leak table (# | Where it's leaking | $/month | Confidence | The fix), one row per category, highest dollar impact first, each fix a single change.
The #1-line callout — the top leak annualized, named, with its confidence.
Per-leak detail artifacts — the flagged nodes/instance types, the PromQL that
found them, and the underlying $/GPU-hour rates for the cluster engineer.
Error Handling
Error
Cause
Solution
HTTP 401/403 on /api/v1/query
Token principal not in admin/metrics/write
Run Step 1; report the group requirement from promql-billing-setup.md. Stop.
Empty result for billing:instance:total
Wrong data-source proxy UID, or org has no billing metrics enabled
Verify $CW_PROM_URL points at the Grafana Prometheus proxy; confirm in Grafana Explore.
DCGM_* series absent
DCGM exporter not scraped on the node pool
Skip Leaks 1/3 utilization filter for that pool; note "utilization unavailable" rather than reporting $0.
reservation label missing
Provider label key differs per org
Confirm the reservation/committed label with kubectl get nodes --show-labels; substitute it in the query.
Ranker prints ~$0/month confirmed
A kind value was mis-cased and dropped from the sum
The ranker normalizes case; verify each leak object's kind is one of the three tiers.
Examples
Example 1: "Why is my CoreWeave bill so high?"
Runs the full pipeline. The access probe passes, the four scans return rows, and the
ranker emits a split, confidence-stamped report:
### A $180K/month CoreWeave GPU cluster is burning **~$44,986/month** (confirmed), plus up to **~$29,110/month** pending review
Trailing 30 days ending 2026-06-22. Confirmed **~$540K/year**; up to **~$349K/year** more pending review. Spend is reconstructed from PromQL against CoreWeave's managed Grafana (no billing API). Every line below is one change.
| # | Where it's leaking | $/month | Confidence | The fix |
|---|---|--:|---|---|
| 1 | **Idle reserved GPUs** — reserved capacity billing around the clock below a utilization floor | **$26,280** | Confirmed | Right-size or release the reservation |
| 2 | **Allocated-but-idle on-demand GPUs** — nodes up at <5% SM-active, paying full rate for no work | **$18,706** | Confirmed | Scale-to-zero / deschedule the idle nodes |
| 3 | **H100/H200 on small-model inference** — L40S is cheaper per token in the 7B–30B regime | **$16,629** | Estimated | Move small inference to L40S |
| 4 | **Steady on-demand that should be committed** — an always-on floor paying on-demand | **$12,481** | At-risk | Commit the stable floor (up to 60% off) |
**The #1 line alone — idle reserved gpus (confirmed) — is ~$315K/year, fixed in one setting.**
Example 2: Idle-Reservation Sweep
User asks "are we paying for idle reserved GPUs?" The skill runs Step 3 only, crosses
billing_gpu{reservation!=""} against DCGM_FI_PROF_SM_ACTIVE, and reports each
reserved node below the floor with its 30-day committed spend.
