spend-watch

Today is ${today}. > **${var}** — scope selector + optional arm flag. > - **empty** / `all` → sweep every platform whose secret is present, emit one combined digest. > - `neon` | `vercel` | `railway` | `actions` → run one adapter only. > - prepend `arm:` (e.g. `arm:neon`, `arm:actions`) → authorize the adapter's **safe write levers** for this run. Without `arm:` the skill is read-only: it recommends, never mutates. > - `dry-run` appended anywhere → build the digest but do not `./notify` (for testing). > > Runs unattended — treat `${var}` as final, no confirmation step, except a delete/mutation always re-reads the target's current state before acting (see each adapter's arm rules). This skill is a cost **analyst**, not a bill alarm. Each run answers, per platform: 1. **Attribution** — what is consuming the most? Rank drivers by dollars (or the resource unit that maps to dollars), down to the specific object: service, branch, route, RPC method, workflow. Top-N with each line's % share. 2. **Root cause** — *why* is that line expensive? 3. **Recommendation** — a ranked action list, each carrying: the concrete lever, an effort/risk tag, whether it's armable now, and a **saving in real dollars ONLY when the platform's billing API returns real dollars** (Railway `currentUsage`, Actions overage). Everywhere else there is **no dollar figure** — the line carries its real signal instead (idle-awake %, % over the included allowance, cache-miss rate, stale-preview count, failure rate). Never invent a `$X/mo`. **Dollars-only-when-real is the core rule.** A fabricated "$5/mo" is worse than the true signal "idle-awake 71%". Rank each recommendation by: real-$ saving first (when known), then signal magnitude × how actionable it is (armable > 1-click > code-change > investigate). The recommendation is the deliverable; the signal justifies it; the dollar is a bonus only when the API hands it over. Across platforms (`all`): a roll-up — the **real** spend where billing exposes it (Railway $, Actions $), the biggest signal-ranked driver anywhere, and the top actions fleet-wide. No synthetic grand total. The monitoring (deltas, real budgets, signal thresholds) is the *trend context and the trigger*; the deliverable is the ranked recommendations. --- ## Shared setup (every run) 1. Read `memory/MEMORY.md` for context and `memory/spend-config.md` for real-$ budgets, signal thresholds, and ignore-lists (see the config schema at the bottom). If `spend-config.md` is missing, run with the built-in defaults and note `NO_CONFIG` in the log — recommendations still work; they rank by signal regardless. 2. Read the last 7 days of `memory/logs/` — used to detect *newly* expensive drivers vs ongoing, and to avoid repeat-nagging a recommendation already sent. 3. Parse `${var}`: ```bash RAW="$(printf '%s' "${var}" | tr '[:upper:]' '[:lower:]' | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" ARM=0; case "$RAW" in arm:*) ARM=1; RAW="${RAW#arm:}";; esac DRYRUN=0; case "$RAW" in *dry-run*) DRYRUN=1; RAW="$(printf '%s' "$RAW" | sed 's/dry-run//g' | tr -s ' ')";; esac RAW="$(printf '%s' "$RAW" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" case "$RAW" in ""|all) SCOPE=all ;; neon|vercel|railway|actions) SCOPE="$RAW" ;; *) SCOPE=all ;; # unrecognized -> full sweep, note it in the log esac ``` 4. Run the matching adapter(s). Each adapter **self-skips** if its secret is absent — log `<platform>: SKIP no-secret` and continue. In `all` mode, run every adapter whose secret is present, then run the **Synthesis** section. ### Common adapter contract Every adapter produces the same intermediate shape (the model builds it in memory, one entry per cost driver): ``` { platform, object, metric, amount, share_pct, signal, real_usd, saving_usd, trend, root_cause, recommendation, effort, armable } ``` - `effort` ∈ `armable` | `1-click` | `code-change` | `investigate`. - `real_usd` / `saving_usd` — populated **only** from a billing API that returns actual dollars (Railway `customer.currentUsage`, Actions summed `netAmount`/overage). Otherwise **`null`** — do not compute a dollar figure from a config rate. There is no cost-rate multiplication anywhere in this skill. - `signal` — the real, dollar-free magnitude that justifies the line and drives its rank when `saving_usd` is null: e.g. `idle-awake 71%`, `18% over included minutes`, `stale-previews 300`, `failure-rate 51%`, `RSS 4.6× working set`. Always present. - `metric`/`amount` — the raw usage unit (compute-hours, minutes, GB-hrs, requests) behind the signal. - `trend` ∈ `new` | `up` | `flat` | `down`, computed against the prior snapshot. ### State ledger Each adapter reads and rewrites `memory/state/spend-<platform>.json`: ```json { "updated_at": "<ISO8601>", "drivers": [ { "object": "...", "metric": "...", "amount": 0, "signal": "idle-awake 71%", "real_usd": null } ], "recommendations_sent": [ { "id": "neon:tighten-timeout:ep-x", "sent": "<ISO8601>", "signal": "idle-awake 71%" } ] } ``` `recommendations_sent` is how the skill avoids repeat-nagging: if a rec's `id` was sent within the config's `nag_cooldown_days` (default 14) and the driver hasn't grown, downgrade it to a one-line "(still open)" mention rather than re-ranking it at the top. ### Notify The `./notify` body is the **ranked recommendation list**, not a raw usage dump. Severity gate: - `critical` — a **real-$** breach: Railway `currentUsage` over `budgets_usd_month.railway` or its `usageLimit`, or Actions over the global included allowance (real overage $). Real dollars only — a signal alone never escalates to critical. - `warn` — an actionable recommendation exists, or a driver trending `up`/`new` past the config's `alert_share_pct` (signal-based, no dollars needed). - `info` — nothing worth acting on. **Send nothing** (silence is the signal, like `price-alert`). Just log. `dry-run` suppresses the send entirely. --- ## Adapter: actions (GitHub Actions — build-first, highest feasibility) Aeon runs *on* GitHub Actions, so this adapter optimizes the agent's own runner bill. Auth is `gh` (uses `GH_GLOBAL` ambiently as `GH_TOKEN`) — **not** `secretcurl` (`GH_GLOBAL` ends in `_GLOBAL`, which secretcurl does not substitute). ### 1. Pull usage — account-global first, per-repo only for attribution **Billing is ONE account-wide pool.** Included minutes are a single global allowance and overage is billed on the account total — NOT per repo. So the budget check is global; per-repo is only for attributing *where* the global minutes go. Never gate savings on a per-repo dollar figure. ```bash ME="$(gh api user --jq .login)" # Global account usage — the budget number. Probe both; branch on HTTP status. gh api "/users/$ME/settings/billing/usage" 2>/dev/null # enhanced: per-repo-per-day rows {quantity(min),grossAmount,discountAmount,netAmount,repositoryName}. SUM across ALL rows = global minutes + global net $ this cycle. gh api "/users/$ME/settings/billing/actions" 2>/dev/null # classic (410 on migrated accounts): total_minutes_used + included_minutes (the GLOBAL allowance) + minutes_used_breakdown{UBUNTU,MACOS,WINDOWS} gh api "/users/$ME/settings/billing/shared-storage" 2>/dev/null # global artifacts+packages storage GB ``` Note on the enhanced endpoint: it returns per-repo rows, but that's a reporting breakdown of one global pool — **sum them for the account figure**, group by `repositoryName` only to attribute. `netAmount` is real billed $ after the included-minutes discount; sum of `netAmount` = your true global Actions bill. Public repos are free/unlimited and don't draw the pool; only private-repo minutes consume the global allowance. ```bash # Attribution: which repos/workflows drive the global minutes. Scope = every repo with usage # in the billing rows this cycle (NOT a fixed private-list). Optional actions.repos in config # just narrows/orders which repos to fetch per-workflow detail for. for repo in <repos appearing in the billing rows>; do gh api "/repos/$repo/actions/cache/usage" 2>/dev/null gh api "/repos/$repo/actions/artifacts?per_page=100" --jq '[.artifacts[]|{name,size:.size_in_bytes,created:.created_at,expires:.expires_at,id}]' 2>/dev/null gh api "/repos/$repo/actions/runs?per_page=100" --jq '[.workflow_runs[]|{name,workflow_id,run_started_at,updated_at,conclusion}]' 2>/dev/null done ``` ### 2. Budget check (global) + attribute (per-repo/workflow) **Budget check — global only.** Compare account `total_minutes_used` vs `included_minutes` (classic) or summed `netAmount` vs `budgets_usd_month.actions` (enhanced). Inside the included allowance → net $0 → nothing to save (say so; it's `info`). Over the allowance → the overage $ is the *only* real Actions saving, and it's global. This is the single number that decides severity. **Attribution — per-repo → per-workflow.** Only once there's a global overage (or to pre-empt one) rank which repos/workflows drive the pool by **minutes** (the signal), so a trim targets the biggest draw. Dollars come only from real `netAmount`/overage; if billing $ is absent, rank by minutes and % of the included allowance — never multiply by a rate. ### 3. Root-cause + recommend (heuristic library) Savings are real **only when the account is over (or projected over) the global included allowance** — below it, trims free up pool headroom but save $0, so rank them as headroom/hygiene, not dollars. | Pattern detected | Recommendation | saving | effort | |---|---|---|---| | account over included minutes, one repo/workflow dominates the pool | trim that workflow's frequency/scope | share of the **global overage** | **armable** (aeon.yml PR) | | macOS/Windows job with no OS-specific need | switch to `ubuntu-latest` | 10x / 2x fewer pool minutes | code-change (PR) | | over-frequent cron (`*/5`,`*/30`) driving pool draw | trim frequency (`*/30`→hourly halves runs) | pool headroom (or overage $ if over) | **armable** | | artifact with long `retention-days` + large size | shorten retention / delete | global storage GB-month | **armable** | | workflow avg duration climbing | flag — likely a hung step / added work | n/a | investigate | ### 4. Arm (only if `ARM=1`) - **Delete stale artifacts** (any repo, since storage is global): re-read the list, delete artifacts older than `actions.artifact_stale_days` (default 14): `gh api -X DELETE "/repos/$repo/actions/artifacts/$id"`. Report count + freed GB. - **Trim a cron**: open a PR editing the workflow that's the biggest global-pool draw (`gh` checkout, edit the one schedule line, `gh pr create`). One PR per run, never force-merge. Ties into `auto-workflow`. Never lower a schedule the config marks `pin:`. ### 5. Notify + log Emit the ranked recommendation block (see Synthesis format). Log to `memory/logs/${today}.md` under `### spend-watch` (first bullet `- adapter: actions (var="${var}")`): the top drivers, the recs made with their `id`s, and any arm action taken. Write `memory/state/spend-actions.json`. --- ## Adapter: neon — feasibility HIGH Auth: `NEON_API_KEY` via `./secretcurl` (`{NEON_API_KEY}`). Base `https://console.neon.tech/api/v2`. ### 1. Pull usage ```bash ./secretcurl -sS --max-time 30 -H 'Authorization: Bearer {NEON_API_KEY}' -H 'Accept: application/json' \ 'https://console.neon.tech/api/v2/projects' # per project: ./secretcurl -sS -H 'Authorization: Bearer {NEON_API_KEY}' "https://console.neon.tech/api/v2/projects/$PID/branches" ./secretcurl -sS -H 'Authorization: Bearer {NEON_API_KEY}' "https://console.neon.tech/api/v2/projects/$PID/endpoints" # suspend_timeout_seconds, autoscaling_limit_min_cu/max_cu ./secretcurl -sS -H 'Authorization: Bearer {NEON_API_KEY}' \ "https://console.neon.tech/api/v2/consumption_history/projects?from=$FROM&to=$TO&granularity=daily&limit=100" # compute_time, active_time, storage, data_transfer ``` Print `-w '\nhttp=%{http_code}\n'` and branch on it — only degrade on a real non-2xx/timeout/empty; log the true reason, never "sandbox". **Neon returns no billable dollars on lower plans** (`consumption_history` is Scale-plan-gated → 403), so this adapter is **signal-only — no `$` figures**. The usage signal still comes from the branch/endpoint objects' cumulative fields (`compute_time_seconds`, `active_time_seconds`, storage) even when `consumption_history` 403s. ### 2. Attribute Rank drivers by the **signal**, not dollars: `idle-awake %` (`active_time / elapsed`), compute-hours share, storage GB-hrs, `max_cu` headroom. Attribute to project → branch: "project X's preview branches = 40% of compute-hours." `real_usd` stays `null`. ### 3. Recommend (heuristics — ranked by signal) | Pattern | Recommendation | signal | effort | |---|---|---|---| | endpoint `suspend_timeout_seconds` high + high idle-awake % | lower to 60s | idle-awake % (e.g. 71%) | **armable** (`PATCH`) | | `max_cu` never approached in the window | shrink autoscaling ceiling (e.g. 4→2) | peak CU vs ceiling headroom | **armable** (`PATCH`) | | branch idle > `neon.branch_stale_days` | delete branch | days idle | **armable · confirm** |