prerender-sizing

name	prerender-sizing
description	Pick or adjust the prerender server's pool envelope (`PRERENDER_PAGE_POOL_MIN` / `_MAX` / `_HIGH_PRIORITY_MAX` / `_HIGH_PRIORITY_THRESHOLD` / `_IDLE_CONTRACTION_MS` / `_SHARED_CONTEXT_CAP`) and ECS task CPU/memory using observed telemetry, not round-number intuition. Use this skill whenever an operator asks "are we sized right?", wants to change the envelope after a workload shift, considers a new ECS task size, or hits memory pressure / OOM on the prerender service. Layers on top of the `aws-access` skill for the underlying CloudWatch + DB plumbing; that skill is a prerequisite.
allowed-tools	Read, Grep, Glob, Bash

Prerender server sizing

The prerender pool's tab capacity is governed by a small set of SSM-driven knobs:

Env var	What it controls
PRERENDER_PAGE_POOL_MIN	Idle floor — pool never contracts below this.
PRERENDER_PAGE_POOL_MAX	Burst ceiling reachable by any priority.
PRERENDER_PAGE_POOL_HIGH_PRIORITY_MAX	Extra ceiling, reachable only when caller priority >= HIGH_PRIORITY_THRESHOLD.
PRERENDER_HIGH_PRIORITY_THRESHOLD	Priority bar that unlocks the upper tier.
PRERENDER_PAGE_POOL_IDLE_CONTRACTION_MS	Hysteresis window before each contraction tick.
PRERENDER_SHARED_CONTEXT_CAP	Absolute LRU cap for cached BrowserContexts.

Plus the ECS task definition's cpu and memory. All these together form the memory envelope that bounds how many warmed BrowserContexts the system can hold and how much burst headroom it has.

The right values aren't fixed once. As the workload shifts — more realms, larger card fan-outs, busier user traffic — the envelope needs re-tuning. This skill is the methodology to derive new values from telemetry rather than guessing.

When to use this skill

Trigger on any of:

• "Are we sized right for prerender?" / "should we resize the prerender ECS task?"
• "We're hitting memory pressure / OOM on the prerender service."
• "We added more realms / a bigger card schema, do we need more prerender capacity?"
• "Why does the dashboard show prerender memory peak at X%?"
• "Should I bump PRERENDER_PAGE_POOL_MAX from N to M?"

If the user is asking "why did this single render time out", that's the indexing-diagnostics skill, not this one. This skill is for capacity planning.

The sizing model

The pool's memory cost decomposes as:

total_memory_used = node_baseline + N × marginal_per_tab

where:

• node_baseline: Node + base Chrome process + standby + idle-state structures. Roughly 2 GB on the boxel prerender server today; varies with Node version, V8 heap retained, and whether standby refresh is active.
• N: number of warmed pool entries (active tabs + standby contexts the LRU is holding).
• marginal_per_tab: cost of one additional warmed BrowserContext + its cached fetches + tab queue state. Empirically derived per environment.

CPU follows a different shape. Each actively rendering tab consumes approximately one busy CPU core (Chromium docs / observed). But tabs alternate between rendering, host-side waits (fetches, store loads), and idle. So:

cpu_peak ≈ (# tabs rendering simultaneously) × 1 vCPU

The "rendering simultaneously" number is rarely the full pool — it's bounded by both the global render semaphore and the per-tab queue. Empirically on staging today, peak observed simultaneous rendering is 1–2 even when the pool is at 7 tabs.

These two facts mean memory is almost always the binding constraint for prerender sizing on Fargate. CPU rarely dominates; if a CPU peak appears it's usually a sign of synthetic stress, not steady-state.

Procedure

The skill has four required steps plus an optional fifth. Do steps 1–4 in order; the later ones depend on numbers you derive earlier. Step 5 (Fargate pricing comparison) is optional — only needed when the resize affects task size.

A note on units before you start: ECS / Fargate task memory is configured in MiB (binary, 2^20 bytes), and the CloudWatch MemoryUtilization percentage is computed against that allocated MiB value. So an "8 GB" task is really 8 GiB = 8192 MiB, and 98 % of it is 8030 MiB ≈ 7.84 GiB. The math below uses MiB / GiB throughout for consistency with what the AWS APIs actually return; if you see "GB" in this skill it's shorthand for "GiB" — never decimal gigabytes.

Step 1: Capture the telemetry window

Goal: a snapshot of how the existing fleet is behaving. Three sources:

CloudWatch — CPU and memory utilisation

Get 24-hour AND 7-day windows. The 24 h shows steady state; the 7 d catches bursts you'd otherwise miss. Run the snippet below twice — once with WINDOW=24h, once with WINDOW=7d — and compare.

ENV=staging  # or production
PROFILE=claude-${ENV}  # provisioned via the aws-access skill
WINDOW=24h   # change to 7d for the second run; the case statement below picks the matching --period
END=$(date -u +%FT%TZ)
case "$WINDOW" in
  24h) START=$(date -u -d '24 hours ago' +%FT%TZ); PERIOD=300  ;;  # 5-min datapoints
  7d)  START=$(date -u -d '7 days ago'  +%FT%TZ); PERIOD=3600 ;;  # 1-hour datapoints
  *) echo "WINDOW must be 24h or 7d" >&2; exit 1 ;;
esac

echo "=== CPU $WINDOW (% of allocated vCPU) ==="
aws --profile $PROFILE cloudwatch get-metric-statistics \
  --namespace AWS/ECS --metric-name CPUUtilization \
  --dimensions Name=ServiceName,Value=boxel-prerender-server-${ENV} \
               Name=ClusterName,Value=${ENV} \
  --start-time "$START" --end-time "$END" --period $PERIOD \
  --statistics Average Maximum \
  --query '{count: length(Datapoints), peakAvg: max(Datapoints[*].Average), peakMax: max(Datapoints[*].Maximum)}' \
  --output json

echo "=== Memory $WINDOW (% of allocated MiB) ==="
aws --profile $PROFILE cloudwatch get-metric-statistics \
  --namespace AWS/ECS --metric-name MemoryUtilization \
  --dimensions Name=ServiceName,Value=boxel-prerender-server-${ENV} \
               Name=ClusterName,Value=${ENV} \
  --start-time "$START" --end-time "$END" --period $PERIOD \
  --statistics Average Maximum \
  --query '{count: length(Datapoints), peakAvg: max(Datapoints[*].Average), peakMax: max(Datapoints[*].Maximum)}' \
  --output json

For the peak window, find the timestamp:

aws --profile $PROFILE cloudwatch get-metric-statistics \
  --namespace AWS/ECS --metric-name MemoryUtilization \
  --dimensions Name=ServiceName,Value=boxel-prerender-server-${ENV} \
               Name=ClusterName,Value=${ENV} \
  --start-time "$START" --end-time "$END" --period 3600 --statistics Maximum \
  --query 'sort_by(Datapoints, &Maximum) | [-3:].[Timestamp, Maximum]' \
  --output table

Queue-snapshot logs — what was the pool actually doing at peak?

This is the bridge from "memory was at X%" to "X% corresponds to N tabs". The prerender server logs prerender-queue-snapshot periodically with the live totalTabs and totalPending.

PEAK_START='2026-04-29 02:00:00 UTC'   # from the CloudWatch query above
PEAK_END='2026-04-29 02:30:00 UTC'

aws --profile $PROFILE logs filter-log-events \
  --log-group-name ecs-boxel-prerender-server-${ENV} \
  --start-time $(date -u -d "$PEAK_START" +%s)000 \
  --end-time $(date -u -d "$PEAK_END" +%s)000 \
  --filter-pattern '"prerender-queue-snapshot"' \
  --query 'events[*].message' --output text \
| tr '\t' '\n' \
| grep -oE "totalTabs=[0-9]+ totalPending=[0-9]+" \
| sort | uniq -c | sort -rn | head -10

The output is a histogram: how many snapshots saw each (totalTabs, totalPending) combination. The mode is the typical state during the peak; the max totalTabs is what the pool actually grew to.

DB — render-timing distribution

Confirms whether the system held under pressure (zero render-timeouts) or was at the edge. Queries the boxel_index.diagnostics JSONB column via the aws-access skill's port-forward + claude_readonly_user flow.

-- 7-day render-timing histogram. The `::int` casts assume the
-- diagnostic shape Prerenderer emits today; if a row is missing
-- a key (older diagnostic shape, partial write, manual edit) the
-- whole query errors. If that happens, narrow the WHERE clause to
-- skip the malformed rows: e.g. `AND diagnostics->'waits' ?
-- 'tabQueueMs'` (the JSONB `?` operator tests for a key) keeps
-- only rows with that key present.
SELECT
  count(*) AS rows_with_diag,
  count(*) FILTER (WHERE (diagnostics->>'totalElapsedMs')::int >= 145000) AS at_or_over_timeout,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY (diagnostics->>'totalElapsedMs')::int) AS p95_total_ms,
  percentile_cont(0.99) WITHIN GROUP (ORDER BY (diagnostics->>'totalElapsedMs')::int) AS p99_total_ms,
  max((diagnostics->>'totalElapsedMs')::int) AS max_total_ms,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY (diagnostics->'waits'->>'tabQueueMs')::int) AS p95_tabq_ms,
  max((diagnostics->'waits'->>'tabQueueMs')::int) AS max_tabq_ms,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY (diagnostics->'waits'->>'semaphoreMs')::int) AS p95_sem_ms,
  max((diagnostics->'waits'->>'semaphoreMs')::int) AS max_sem_ms
FROM boxel_index
WHERE diagnostics IS NOT NULL
  AND diagnostics ? 'totalElapsedMs'
  AND (diagnostics->>'indexedAt') ~ '^[0-9]+$'
  AND (diagnostics->>'indexedAt')::bigint > extract(epoch from now() - interval '7 days')*1000;

Key signals to look for:

• at_or_over_timeout > 0: the system is already dropping renders. Sizing change is needed urgently.
• max_tabq_ms of seconds-to-tens-of-seconds: the user was waiting for a tab. This is the UX-visible pressure that priority routing + dynamic expansion exists to mitigate.
• max_sem_ms of seconds-to-tens-of-seconds: global render-semaphore saturation. Indicates pool is too small or fleet is too small.
• p99_total_ms near 145000 (the timeout budget): system was at the edge. Even if no timeouts fired, you're one bad burst from a 504.

Step 2: Derive the marginal cost

From the peak snapshot — memory used, baseline, and active tabs:

marginal_per_tab = (peak_memory_bytes − baseline_bytes) / N_at_peak

peak_memory_bytes = peak_memory_pct × allocated_memory. If the task is 8 GB and peak is 98 %, that's 7.84 GB used.

baseline_bytes is approximately 2 GB on the current boxel prerender. If the workload changes substantially (Node major version, Chrome major version, observability instrumentation), measure it directly: tail the prerender server logs at idle and capture the RSS of the process after standby refresh has settled and the pool has 0 active tabs.

N_at_peak is the max totalTabs from the queue-snapshot histogram above. Note that this includes standby (the standby contexts also pin memory).

The number you derive is environment-specific. The original CS-10976 plan estimated 625 MB/tab from a quieter window; refreshed staging telemetry on 2026-04-30 measured 836 MB/tab under heavier load. Always re-derive from current data, never reuse a stale number.

Step 3: Project the proposed envelope's memory footprint

Make a table of memory at each candidate MAX / HP_MAX value against each candidate task size. Threshold: stay under 80 % of allocated memory under sustained operation.

N_tabs    memory_used = baseline + N × marginal
                       % of allocated for each task size

Example with baseline=2 GB, marginal=836 MB:

N tabs | Memory used | 8 GB task | 12 GB task | 16 GB task
   2   |    3.7 GB   |    46 %   |     31 %    |    23 %
   4   |    5.4 GB   |    67 %   |     45 %    |    34 %
   6   |    7.1 GB   |    89 %   |     59 %    |    44 %    ← MAX
   8   |    8.7 GB   |   109 %   |     72 %    |    55 %    ← HP_MAX
  10   |   10.4 GB   |   130 %   |     86 %    |    65 %

The 80 % line is the working ceiling. Anything that exceeds it under expected operation is undersized (see fallback below).

HP_MAX is the largest the pool can grow to — that's the value that has to fit. MAX is the most likely steady-state ceiling but HP_MAX is the memory invariant.

Step 4: Choose the values

Now apply the projection plus operational judgment:

• MIN — the idle floor. From queue-snapshot data: what's the typical low-load tab count? Pick MIN slightly above that so the manager's warm-vacancy routing has cached affinities to route to. On boxel today, MIN=2 covers the steady state.
• MAX — the any-priority burst ceiling. From queue-snapshot data: what's the observed peak totalTabs? Pick MAX at-or-just-above that. Values above the 80 % memory line are undersized; values below the observed peak will throttle under existing workload. On boxel today, MAX=6 covers all 7 d observations.
• HP_MAX — the high-priority ceiling. Should give priority-10 traffic 1–2 reserved expansion slots beyond MAX for the worst-case "low-priority workload has saturated MAX, user comes in" scenario. Has to fit memory at 80 %. On boxel today, HP_MAX=8 fits 16 GB at 55 %.
• HIGH_PRIORITY_THRESHOLD — the bar that unlocks HP tier. Above the system-initiated tiers (systemInitiatedPriority = 1, systemInitiatedPrerenderHtmlPriority = 0), at-or-below the user-initiated tiers (userInitiatedPriority = 10, userInitiatedPrerenderHtmlPriority = 9). Default 5 leaves room for an intermediate priority level (e.g. live-refresh) to also benefit without re-tuning.
• IDLE_CONTRACTION_MS — the hysteresis window. Long enough to absorb sequential render trains from a typical fan-out; short enough that contraction reaches MIN within a few minutes. Default 60 000 ms (1 minute) works for most workloads.
• SHARED_CONTEXT_CAP — the absolute LRU cap on cached BrowserContexts. Default HP_MAX × 1.5 keeps the LRU stable across expansion + contraction cycles.

Step 5 (optional): Pricing comparison

If the resize affects task size, do a Fargate pricing comparison. us-east-1 on-demand:

• 1 vCPU @ $0.04048/hr
• 1 GB @ $0.004445/hr

So:

Task size	$/hr	/month per task
1 vCPU / 4 GB	$0.058	$42
2 vCPU / 8 GB	$0.117	$85
2 vCPU / 16 GB	$0.152	$111
4 vCPU / 8 GB	$0.197	$144
4 vCPU / 16 GB	$0.233	$170

If the resize is "swap memory for CPU" (the typical case for prerender — memory-bound, CPU over-provisioned), the cost may actually drop. Always show the pricing delta in the PR description. It's a meaningful data point for the resize decision.

Worked example — the 2026-04-30 staging resize

Captured on 2026-04-30 ~20:00 UTC for the CS-10976 PR 12 staging activation.

Telemetry

Metric	24 h	7 d
CPU avg of 5-min Avg	1.1 %	1.5 %
CPU 5-min peak	67.5 %	97.5 %
Memory avg of 5-min Avg	35 %	39 %
Memory 5-min peak	64 %	98.3 %

7-d render-timing histogram from boxel_index.diagnostics:

• 58,549 rows with diagnostics
• 0 timeouts (totalElapsedMs ≥ 145 000)
• All-time DB-wide timeouts: 0
• p95 totalElapsedMs = 3 675 ms
• p99 totalElapsedMs = 13 804 ms
• max totalElapsedMs = 100 984 ms (under but approaching the 145 s budget)
• max tabQueueMs = 90 019 ms (90 s of waiting!)
• max semaphoreMs = 17 620 ms

Queue-snapshot at the memory peak:

115 totalTabs=5 totalPending=0
 60 totalTabs=6 totalPending=0
 25 totalTabs=6 totalPending=4
  9 totalTabs=6 totalPending=5
  4 totalTabs=7 totalPending=4

Marginal cost

• Peak memory: 98.3 % × 8 GB = 7.86 GB
• Baseline: 2 GB
• Peak tabs: 7
• Marginal per tab: (7.86 − 2.0) / 7 = 836 MB / tab

(Original plan estimated 625 MB. Re-derived against actual peak: 836 MB. Use this number, not the plan's.)

Memory projection

N tabs	Memory used	8 GB (today)	16 GB (resized)
2 (MIN)	3.7 GB	46 % ✓	23 % ✓
4	5.4 GB	67 % ✓	34 % ✓
6 (MAX)	7.1 GB	89 % ✗	44 % ✓
8 (HP_MAX)	8.7 GB	109 % ✗ OOM	55 % ✓

The 16 GB resize is what makes HP_MAX=8 safe. On the existing 8 GB task, MAX=6 is already tight; HP_MAX=8 would OOM.

Chosen values

• MIN = 2
• MAX = 6
• HP_MAX = 8
• HIGH_PRIORITY_THRESHOLD = 5
• IDLE_CONTRACTION_MS = 60_000
• SHARED_CONTEXT_CAP = 12 (HP_MAX × 1.5)

Cost delta

4 vCPU / 8 GB ($0.197/hr) → 2 vCPU / 16 GB ($0.152/hr) = −23 % per task. Doubles the memory ceiling while reducing spend.

Pitfalls

• Don't reuse stale marginal-cost numbers. Re-derive every time. Workload changes shift the per-tab cost.
• Don't ignore standby in the tab count. The pool maintains maxPages + 1 standbys; those count against memory the same as active tabs.
• CPU < memory in priority for prerender. If you're tempted to bump CPU, check the simultaneous-rendering count first. Almost always you can shrink CPU and grow memory for net wins.
• 80 % is the working ceiling, not the maximum. Reserve the last 20 % for LRU growth, standby refresh churn, runtime-exception capture buffers, and Node V8 retained-heap overhead.
• HP_MAX is the memory invariant, not MAX. The pool can reach HP_MAX under any priority-10 burst; size memory for that worst case.
• Validate with a synthetic stress test before promoting to prod. Concurrent catalog full reindex + simulated user reindex; pass criteria: high-priority p95 tabQueueMs < 1 s, memory peak < 80 %, CPU peak < 80 %, zero 145 s timeouts.

Where to set the values

Two surfaces:

• SSM parameters at /<env>/boxel/PRERENDER_* drive the runtime envelope. The activation procedure (including the exact aws ssm put-parameter calls, the required force-deploy of the ECS service after, the validation gate, and the rollback path) lives in docs/aws-operations.md → "Activate the dynamic-pool prerender server". Use that runbook rather than ad-hoc shell — it covers the pre-requisites that make the difference between a real activation and a runtime no-op.
• ECS task definition is in cardstack/infra at configs/boxel-prerender/base/main.tf (cpu and memory properties). Resize via Terraform PR + apply.

After applying values, re-run Step 1 at +1 hour, +24 hours, +48 hours to validate. If memory stays under 80 % and tab-queue waits don't regress, promote to prod with the same values (or adjust per the projection if prod's load profile differs).