| name | bulk-briefing-cohort |
| description | Use when manually triggering a bulk run of meeting briefings — dispatching a cohort of meeting_briefing agent jobs against prod via the imminence-gated endpoint, monitoring them to completion, and producing the outcome/cost/runtime analysis and CSV export. Irreversible prod spend (~$6 per dispatched briefing); requires gp-admin AWS access and this repo's packages/gp-api. |
Runbook: dispatch a cohort of meeting_briefing jobs in prod and analyze them
You are running a one-shot bulk dispatch of meeting_briefing agent jobs against
prod, using the product's gate (serve-ICP, fail closed + 3-day imminence
window + future-briefing dedupe), then monitoring to completion and producing an
outcome/cost/runtime analysis. Target for this run: 200 dispatched briefings.
The dispatch script enforces a fleet cap: at most 100 meeting_briefing agents
RUNNING at once (--max-in-flight, default 100). For targets above the cap it
dispatches the first wave fast, then trickles the rest in as agents complete
(~15 min each), so a 200-target dispatch phase takes roughly 30-60 min, not ~5.
Do not "fix" the slowness by raising the cap or running a second copy.
This is an irreversible, money-spending prod action (about $6 per dispatched
briefing, so ~$1,200 at 200). Do the read-only pre-flight first, show the human
the dry-run numbers, and get an explicit go before the real dispatch.
Prerequisites
- AWS access via SSO profile
gp-admin (run aws --profile gp-admin sts get-caller-identity to confirm; if it fails, the human runs aws sso login --profile gp-admin).
- This repo's
packages/gp-api on develop or master (the script scripts/dispatch-imminent-briefings.ts and the useImminenceGate endpoint are merged). Verify the checked-out branch before running.
- Prod gp-api must be running a build where
dispatchManual enforces the serve-ICP gate under useImminenceGate (the same fail-closed check the daily cron uses; landed 2026-06-11). The script has no client-side ICP filter — the server gate is the only enforcement, so if prod predates it the cohort WILL include non-ICP offices. Verify before dispatching: cd packages/gp-api && git log master --oneline -S 'skipping gated manual dispatch' -- src/meetings/services/meetingBriefings.service.ts must show a commit, and prod must be deployed at or past it. Abort and tell the human if not.
psql not required; everything goes through npx tsx.- The daily meetings cron is ENABLED in prod as of 2026-07-15 (
MEETINGS_AUTOMATION_ENABLED=true, dispatchDailyBriefings at @Cron('0 7 * * *') in meetingBriefings.service.ts). It writes experimentType: meeting_briefing rows to the same table your manual cohort does, so a bare createdAt time window can now pick up cron-dispatched runs too if your window overlaps 7am UTC. Prefer diffing against records.push'd electedOfficeIds from your own run's JSONL log over a pure time-window query, or run outside the cron's firing window.
- The cron path (
dispatchBriefingIfNeeded) also applies a 30-day user-inactivity gate (isInactiveUser in src/shared/util/userActivity.util.ts) that dispatchManual — the endpoint this script calls — does not enforce itself. The script now replicates that gate client-side (filters the office pool before dispatching) so a manual run doesn't diverge from what the cron would actually do, but the underlying gap in dispatchManual is still open; don't assume the endpoint is gate-complete if you call it directly instead of through this script.
Secrets and where they live (you gather these, do not ask the human to set env)
- Prod DB password + prod Clerk keys: AWS Secrets Manager secret
GP_API_PROD.
- Prod DB host:
gp-api-db-prod.cluster-cmb1uukjsfbe.us-west-2.rds.amazonaws.com, db gpdb, user gpuser. Reachable directly (no tunnel) as of 2026-06.
- The M2M caller secret
GP_PROD_MACHINE_SECRET is gp-admin's, NOT gp-api's. It lives in the gp-admin-web Vercel project (prj_ZT7POAebSPy3jFf2u0xKIUZTQpcT, production target). Pull it via the Vercel API using VERCEL_TOKEN + VERCEL_TEAM_ID from GP_API_PROD (no vercel login needed). Minting with gp-api's own GP_WEBAPP_MACHINE_SECRET fails with 401.
- Artifacts (schedule + briefing JSON) live in S3 bucket
gp-agent-artifacts-prod, keys meeting_briefing/<runId>/artifact.json and the schedule equivalent.
Gotchas
- For the AWS SDK (used by the analysis tsx for S3),
unset AWS_PROFILE after eval "$(aws --profile gp-admin configure export-credentials --format env)", or the SDK tries to resolve the SSO profile in-process and fails. Also export AWS_REGION=us-west-2.
- Throwaway tsx analysis scripts must live INSIDE
packages/gp-api (not /tmp) so node_modules resolves. Delete them after.
- The dispatch script's
PROD_DATABASE_URL must be URL-encoded (password may contain special chars).
Step 1: assemble env + non-destructive pre-flight probe ($0)
The probe mints the token and POSTs a bogus office id with the gate on. New prod
code returns 201 {dispatched:false} (no dispatch); old code 404s; bad auth 401/403.
This proves auth AND that prod is serving the gated endpoint, with zero spend.
cd packages/gp-api
SECRET_JSON=$(aws --profile gp-admin secretsmanager get-secret-value --secret-id GP_API_PROD --query SecretString --output text)
read VTOKEN VTEAM < <(echo "$SECRET_JSON" | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d["VERCEL_TOKEN"],d["VERCEL_TEAM_ID"])')
PID=prj_ZT7POAebSPy3jFf2u0xKIUZTQpcT
ENVJSON=$(curl -s -H "Authorization: Bearer $VTOKEN" "https://api.vercel.com/v9/projects/$PID/env?teamId=$VTEAM")
getval(){ eid=$(echo "$ENVJSON"|python3 -c "import json,sys;d=json.load(sys.stdin);e=d.get('envs',d);print(next((x['id'] for x in e if x.get('key')=='$1' and 'production' in (x.get('target') or [])),''))"); curl -s -H "Authorization: Bearer $VTOKEN" "https://api.vercel.com/v9/projects/$PID/env/$eid?teamId=$VTEAM"|python3 -c 'import json,sys;print(json.load(sys.stdin).get("value","") or "")'; }
export GP_PROD_MACHINE_SECRET=$(getval GP_PROD_MACHINE_SECRET)
export CLERK_SECRET_KEY=$(getval CLERK_SECRET_KEY)
export CLERK_PUBLISHABLE_KEY=$(getval NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY)
export PROD_API_URL='https://gp-api.goodparty.org'
eval "$(echo "$SECRET_JSON" | python3 -c 'import json,sys,urllib.parse;d=json.load(sys.stdin);pw=urllib.parse.quote(d["DB_PASSWORD"],safe="");print(f"export PROD_DATABASE_URL=\"postgresql://gpuser:{pw}@gp-api-db-prod.cluster-cmb1uukjsfbe.us-west-2.rds.amazonaws.com:5432/gpdb\"")')"
TOKEN=$(npx tsx -e 'import{createClerkClient}from"@clerk/backend";(async()=>{const c=createClerkClient({secretKey:process.env.CLERK_SECRET_KEY,publishableKey:process.env.CLERK_PUBLISHABLE_KEY});const m=await c.m2m.createToken({machineSecretKey:process.env.GP_PROD_MACHINE_SECRET,secondsUntilExpiration:600});process.stdout.write(m.token||"")})().catch(e=>{console.error(e);process.exit(2)})')
curl -s -o /dev/stderr -w "HTTP %{http_code}\n" -X POST "$PROD_API_URL/v1/meetings/briefings/dispatch" -H "content-type: application/json" -H "authorization: Bearer $TOKEN" -d '{"electedOfficeId":"__probe__","kind":"briefing","useImminenceGate":true}'
Step 2: dry run (read-only), then get human go
In the SAME shell (env does not persist across calls):
npx tsx scripts/dispatch-imminent-briefings.ts --dry-run --target=200
Show the human: total offices (1,800), how many the local 30-day activity
gate skips, the resulting eligible pool, target 200, max cost ($1,200). Note
offices already covered by a future briefing (e.g. the prior cohort's
briefing_ready ones) are skipped by the gate dedupe, and offices whose
position is not serve-ICP are skipped fail-closed server-side. Neither the
ICP status nor coverage dedupe is visible in the pool count (only the activity
gate is applied client-side) — until the Databricks is_serve_icp backfill is
broadly populated, expect far more gate-skips per dispatch than the reference
run, and possibly an unreachable target (the script just exhausts the pool and
reports fewer dispatched). Wait for explicit go.
Step 3: real dispatch
echo y | npx tsx scripts/dispatch-imminent-briefings.ts --target=200
It walks the shuffled pool calling the gated endpoint until 200 briefings
dispatch (hard cap, no overshoot), logging one JSONL line per call to
scripts/output/.
The script also enforces the fleet cap itself: before each dispatch it checks
how many meeting_briefing runs are active in prod (RUNNING or
AWAITING_RESUME — paused runs respawn as new Fargate tasks, so they count; DB
count refreshed every 30s, plus the script's own un-reconciled dispatches
counted conservatively) and pauses while that is >= --max-in-flight
(default 100). Expect it to print
throttled: N agents active >= 100 cap lines and sit idle for stretches —
this is correct behavior, leave it running in the same shell (it re-mints its
Clerk token automatically if the run outlasts the 1h TTL). A 200-target run
takes ~30-60 min end to end.
SAVE the final summary JSON, especially reconcile.runStart
and reconcile.runEnd (UTC). Those two timestamps define your cohort.
Step 4: monitor to completion (once a minute until all terminal)
Re-assemble PROD_DATABASE_URL each call. Query experiment_run for the cohort
window and group by status. meeting_briefing agents take ~15 min each and run
~1:1 as Fargate tasks in the pmf-engine-prod ECS cluster; the stale-run sweeper
force-FAILs anything still RUNNING at 45 min.
SECRET_JSON=$(aws --profile gp-admin secretsmanager get-secret-value --secret-id GP_API_PROD --query SecretString --output text)
eval "$(echo "$SECRET_JSON"|python3 -c 'import json,sys,urllib.parse;d=json.load(sys.stdin);pw=urllib.parse.quote(d["DB_PASSWORD"],safe="");print(f"export PROD_DATABASE_URL=\"postgresql://gpuser:{pw}@gp-api-db-prod.cluster-cmb1uukjsfbe.us-west-2.rds.amazonaws.com:5432/gpdb\"")')"
npx tsx -e '
import { PrismaClient } from "./src/generated/prisma"
const p=new PrismaClient({datasourceUrl:process.env.PROD_DATABASE_URL})
const START="<runStart>", END="<runEnd>" // from the dispatch summary
;(async()=>{const w={experimentType:"meeting_briefing",createdAt:{gte:new Date(START),lte:new Date(END)}}
const g=await p.experimentRun.groupBy({by:["status"],where:w,_count:{_all:true}})
const c={RUNNING:0,COMPLETED:0,FAILED:0} as any; for(const r of g) c[r.status]=r._count._all
const agg=await p.experimentRun.aggregate({where:{...w,status:"COMPLETED"},_avg:{durationSeconds:true},_sum:{costUsd:true}})
console.log(new Date().toISOString(),c, "avgDur", agg._avg.durationSeconds, "cost", agg._sum.costUsd)
await p.$disconnect()})()'
Schedule a wakeup ~60s apart and repeat until COMPLETED+FAILED == 200. On the
last poll, also fetch FAILED runId + error.
Step 5: analysis (run after all terminal)
Outcome breakdown lives in the S3 artifact (briefing_status), not the DB. Pull
the cohort runs, fetch each COMPLETED artifact, bucket by briefing_status
(briefing_ready = meeting found, awaiting_agenda, no_meeting_found), and
join cost/duration. Also count meeting_resource_location rows (type AGENDA,
linked by experimentRunId) and diagnose each FAILED run.
Write a tsx file INSIDE packages/gp-api (e.g. ./_tmp_cohort_analysis.ts), set
AWS SDK creds first, run, then rm it:
eval "$(aws --profile gp-admin configure export-credentials --format env)"; unset AWS_PROFILE; export AWS_REGION=us-west-2
npx tsx ./_tmp_cohort_analysis.ts && rm ./_tmp_cohort_analysis.ts
The script should:
experimentRun.findMany where type=meeting_briefing, createdAt in [runStart,runEnd] -> runId, organizationSlug, status, costUsd, durationSeconds, artifactBucket, artifactKey, error.- For COMPLETED rows,
@aws-sdk/client-s3 GetObject on gp-agent-artifacts-prod/meeting_briefing/<runId>/artifact.json, parse briefing_status.
- Report, per bucket (briefing_ready / awaiting_agenda / no_meeting_found / FAILED): count, cost sum + avg, duration avg/min/max.
- Count
meetingResourceLocation rows where experimentRunId in the cohort (expect ~1 per COMPLETED run).
- For FAILED: print runId, org, duration, full error. Note:
upstream stream ended unexpectedly = transient (re-dispatchable); Artifact rejected: Raw HTML not allowed = validation, needs agent-side fix, may recur.
Step 6: CSV export
Emit scripts/output/meeting_briefing_cohort_<date>.csv with columns:
elected_office_id, meeting_briefing_id, run_id, org_slug, briefing_status.
elected_office_id: resolve via electedOffice.findMany where organizationSlug in the cohort slugs (covers all rows incl. failures).meeting_briefing_id: meetingBriefing.findMany where experimentRunId in cohort runIds (populated only for briefing_ready; blank otherwise, since only ready/user-provided statuses persist a MeetingBriefing row).
Visual review (optional)
To eyeball briefings rendered in the real webapp UI, use the view-briefing-gallery
skill — it pulls <runId>/artifact.json into a local dev-only gallery at
http://localhost:4000/dev/briefings. Its pull helper defaults to the dev bucket;
for this prod cohort, copy the artifacts from gp-agent-artifacts-prod into
.local-briefings/<runId>.json yourself (or point LOCAL_BRIEFINGS_DIR at a dir you
populate).
Reference numbers from the 2026-06-08 cohort of 100 (for sanity-checking yours)
- ~1,850 elected offices; ~1,612 have a real (found) schedule; cadence ~1.6 meetings/30d (mostly twice-monthly).
- Outcome mix of completed: ~51% briefing_ready, ~34% awaiting_agenda, ~15% no_meeting_found.
- ~96% success; ~$6/dispatched briefing actual (not the $3.90 code estimate); ~15 min avg runtime.
- 100 dispatches took 346 endpoint calls (the rest gate-skipped). For 200 expect to walk a larger slice of the pool, and note prior-cohort
briefing_ready offices are deduped out.
- That cohort predates both the serve-ICP gating and the in-flight cap: expect a worse dispatched/calls ratio (server-side ICP skips) and a longer wall-clock dispatch phase (throttling) than those numbers suggest.
- That cohort also predates the client-side 30-day activity gate (added 2026-07-15): expect a smaller eligible pool up front than 1,800, on top of the ICP/coverage skips that happen per-call.
