data-programs

# Data Programs A **data program** is a named, stored, agent-authored JS script that fetches, joins, filters, or aggregates data — provider APIs via `providerFetch` / `providerFetchAll` / `providerSearchAll`, app data via `appAction`, or Resources-backed workspace files — and emits a small `{ rows, schema }` result that gets cached in SQL. Any app can render that result through its own table/chart components; nothing about the primitive is analytics-specific. This ships automatically wherever `run-code` is available — there is nothing to mount, wire, or configure per app. If your app already has run-code enabled, `save-data-program` / `preview-data-program` / `run-data-program` / `list-data-programs` / `get-data-program` / `delete-data-program` are already registered actions. ## Why this exists Without it, "build a live view of X" tends to end in one of two bad places: a hardcoded action per vendor/query shape (brittle, doesn't scale to every customer's custom fields), or re-running the same fetch/join logic from scratch on every page view (slow, and it re-spends the same provider quota every time someone looks at a chart). A data program is the middle ground: write the fetch/join/aggregate logic once as ordinary `run-code`, save it, and let the cache + refresh policy handle "how often does this actually need to re-run." ## Authoring workflow 1. Prototype the fetch/join/aggregate logic as a normal `run-code` script and confirm it returns the rows you expect. 2. Change it to call `emit(rows, schema?)` exactly once at the end instead of returning or printing the result. 3. Save it with `save-data-program({ name, title, description, code, defaultParams?, refreshMode?, refreshTtlMs?, background? })`. This **dry-runs the code with `defaultParams` before persisting** and rejects the save with a structured error if it fails, so a broken program is never stored. On success it returns `{ programId, rowCount, columns, sampleRows }` — use that as proof the program produces the rows you expect. 4. Bind whatever UI needs the result. Your app's own read path calls `run-data-program({ programId, params?, forceRefresh?, includeRows? })` and gets back rows/schema/cache metadata to hand to your own table/chart components — there is no dashboard-specific plumbing baked into the primitive itself. (The analytics template's `"program"` dashboard panel source is one such adoption, not a required pattern — see its `data-programs` skill if you want a concrete UI wiring reference.) 5. Iterate without persisting via `preview-data-program({ code, params? })` — same dry-run path, no stored row. ## The sandbox surface Exactly the `run-code` globals (`providerFetch`, `providerFetchAll`, `providerSearchAll`, `appAction`, `webFetch`, `workspace*` Resources helpers), plus: - `params` — a **frozen** global object holding the params for this run. Read-only; mutating it throws. - `emit(rows, schema?)` — call exactly once, at the end. `rows` must be an array of plain objects. `schema`, when provided, is `{ name: string, type: string }[]`; when omitted it's inferred from the first 50 rows (a column is `"json"` if rows disagree on primitive type, otherwise `"number"` / `"string"` / `"boolean"`). `console.log` remains free for debugging and never interferes with parsing the emitted result. Caps enforced on every run: | Limit | Value | | --- | --- | | Max emitted rows | 10,000 (excess dropped, `truncated: true`) | | Max emitted result size | 4 MiB (rows dropped from the end to fit; a single oversized row is a hard failure) | | Max active programs per app | 200 | | Minimum refresh TTL | 60,000 ms | | Run rows kept per (program, params) | 5 most recent | Truncation is always explicit (`truncated: true`) — never a silent drop. ## Caching and refresh model Every `(programId, paramsHash)` pair has its own run history: | Situation | Behavior | | --- | --- | | Fresh cache (younger than `refreshTtlMs`, or `refreshMode: "manual"` with a prior success) | Returns cached rows instantly, `cacheHit: true` | | Stale cache | Re-executes synchronously (25s budget for view-triggered reads, 120s for agent/manual calls), replacing the cache on success | | `refreshMode: "manual"` | Only refreshes on explicit `forceRefresh: true` | | No cache yet | Executes synchronously like a normal first fetch | | `background: true` program with no fresh cache | Serves the last good run with `stale: true` and enqueues a durable background execution; a later call finalizes it once complete | | Background program still running, no prior success | `background_pending` failure — surface an explicit "still computing" state, never a blank result | | Execution fails | Structured `{ code, message }` failure; a previous success is attached as `lastGoodRun` for stale-serve | | Program archived (soft-deleted) | Explicit `archived` failure | Use `background: true` only for scripts that routinely exceed the foreground timeout (large multi-provider joins, deep pagination). Everything else should stay foreground. ## Security notes - **Credentials are always the calling viewer's, never the program author's.** `providerFetch` inside a program resolves auth using the caller's own request context. A viewer without a configured provider key sees that provider's normal auth error on that program's result — never another user's data. - **Programs are shareable org-internal only — never public.** Sharing a program means teammates can view its cached result; because execution uses the *viewer's* credentials, only bind to or view a program you trust. - Raw provider tokens never reach the model or the browser. Only the `{ rows, schema }` an `emit()` call produces ever leaves the sandbox. - The mutating/executing actions (`save-data-program`, `preview-data-program`, `run-data-program`, `delete-data-program`) are not callable from a sandboxed extension/iframe bridge — only from the agent or a server-side read path. ## Related skills - `actions` for `run-code` conventions and the sandbox globals this primitive is built on. - `sharing` for the org-internal, never-public sharing model programs use. - `security` for credential-handling and access-scoping invariants.