data-programs

name	data-programs
description	Save a run-code fetch/join/aggregate script as a stored, refreshable data source that dashboard panels bind to and render. Use when an ad-hoc provider-api-request or run-code analysis should become a live dashboard panel other users see, or when a cohort/join needs an arbitrary property filter, an IN-search, or a cross-provider stitch a canned action can't express.

Data Programs

A data program is a named, stored, agent-authored JS script that fetches, joins, filters, or aggregates provider or app data, and is rendered by a dashboard panel. It is the generic answer to "I can query this once in chat — now make it a panel other people can see, that refreshes on its own."

There is no new credential machinery here. A data program is exactly a run-code script (same providerFetch / providerFetchAll / providerSearchAll / appAction / workspace* globals you already use for ad-hoc analysis) plus two things: a stored, named identity, and a small emit(rows, schema) contract so the result can be cached and charted. Read provider-api first if you have not already — data programs assume you know provider-api-catalog / provider-api-docs / provider-api-request.

When to reach for this vs. a canned action

• The common case (won/lost by pipeline, a standard date-range rollup) already has a first-class action (hubspot-deals, etc.) — use that.
• A one-time chat question — just query it with run-code or provider-api-request and answer directly. Don't save a program for something you'll never look at again.
• A cohort needs an arbitrary property filter, an IN-search over custom values, or a join against a second provider, and the user (or a dashboard) will want to see it again with fresh data — that's a data program.

Authoring workflow

1. Prototype in chat first. Write the fetch/join/aggregate logic as a normal run-code script and confirm it returns the rows you expect.
2. Wrap it to call emit(rows, schema) exactly once at the end instead of returning/printing the result (see contract below).
3. Save it with save-data-program:

   save-data-program(
     name: "risk-meeting-cohort",      // stable slug, unique per your account
     title: "Risk Meeting — HubSpot cohort",
     description: "...",
     code: "<the run-code script text>",
     defaultParams: { riskStatuses: ["Churn Risk"] },  // optional
     refreshMode: "ttl",               // "ttl" | "manual"
     refreshTtlMs: 900000,             // optional, floor 60000
     background: false,                // true for slow multi-minute joins
   )
   

   This dry-runs the code with defaultParams before persisting and rejects the save with a structured error if it fails — a broken program is never stored. On success it returns { programId, rowCount, columns, sampleRows }: use that as your proof the program actually produces the rows you expect before telling the user it's ready.
4. Bind a dashboard panel to it. Panels reference programs through the "program" source; the panel's sql field carries a JSON descriptor instead of a query string:

   { "programId": "dp_...", "params": { "riskStatuses": ["Churn Risk"] } }
   

   The panel renders through the exact same chart/table components as every other panel source — it only ever receives { rows, schema }.
5. Iterate without re-saving with preview-data-program(code, params?) — same dry-run path, no persisted row. Use this while tuning a script before calling save-data-program again.
6. Refresh on demand with run-data-program(programId, params?, forceRefresh?, includeRows?) — this is what the agent calls to get a compact proof-of-result (, , ) without waiting for a panel view. Pass only when you need the full (capped) row set back in context.

The sandbox surface

Inside a data program you get exactly the run-code globals, plus one addition:

• providerFetch(provider, path, init?), providerFetchAll(...), providerSearchAll(...) — the same generic provider-access helpers used for ad-hoc analysis. See provider-api for the pagination/search option shapes.
• appAction(name, params), webFetch, workspace* Resources helpers — same as run-code.
• params — a frozen global object: the params passed into this run (defaultParams merged by the caller, or whatever was passed to run-data-program / the panel descriptor). Read it, never mutate it.
• emit(rows, schema?) — call this exactly once, at the end, with your result. A second call throws. schema is optional — { name: string, type: string }[] — inferred automatically from the first 50 rows when omitted (a column is "json" if rows disagree on primitive type, otherwise "number" / "string" / "boolean"). console.log remains completely free for debugging; it never interferes with emit() parsing and is captured separately as a truncated log tail on the run record.

Caps enforced on every run (not configurable per-program):

Limit	Value
Max emitted rows	10,000 (rows beyond this are dropped, truncated: true)
Max emitted result size	4 MiB (rows dropped from the end to fit; a single row over 4 MiB is a hard result_too_large failure — nothing to safely keep)
Max active programs per app	200 (archive unused ones to free room)
Minimum refresh TTL	60,000 ms
Run rows kept per (program, params)	5 most recent (older ones are pruned automatically)

Truncation is always honest: a partial result comes back with truncated: true, never a silent drop.

Caching and refresh model

Every (programId, paramsHash) pair (params are canonicalized and hashed, so equivalent param objects share a cache entry) has its own run history. What happens on each call to runDataProgram / a panel view:

Situation	Behavior
Fresh cache hit (younger than refreshTtlMs, or refreshMode: "manual" with any prior success)	Returns cached rows instantly, cacheHit: true
Cache is older than the TTL	Re-executes synchronously (panel views get a 25s budget; agent/manual calls get 120s), replacing the cache on success
refreshMode: "manual"	Never auto-refreshes; only forceRefresh: true (or manual_refresh from the UI) re-runs it
No cache yet	Executes synchronously like a normal first fetch
background: true program, no fresh cache	Serves the last good run immediately with stale: true and enqueues a durable background execution (10-minute budget); the next call finalizes it if it has completed
Background program, execution still running, no prior success	background_pending failure — the panel shows an explicit "running in background" error card, never a blank chart
Execution fails (timeout, sandbox error, bad emit() shape)	Failure result with a structured error code; if a previous successful run exists, it is attached as lastGoodRun so the panel can stale-serve instead of going blank
Program was archived (delete-data-program)	Explicit archived failure — panels show "This data program was archived", never a silent blank

Use background: true only for programs that routinely take longer than the foreground budget (multi-provider joins over large cohorts, deep pagination). Everything else should stay foreground — it is simpler to reason about and serves fresh data faster.

Error codes

Code	Meaning	What to do
program_not_found	No program with that id	Check the id via list-data-programs
access_denied	Caller can't see this program	Don't leak existence; ask the owner to share it
archived	Program was soft-deleted	Repoint the panel or unarchive by resaving under the same name
timeout	Didn't finish within the run's budget	Reduce scope, page in fewer items, or mark background: true
emit_missing	Script never called emit(...)	Add the emit(rows, schema?) call at the end
emit_shape_invalid	emit() args weren't (array-of-plain-objects, optional-schema-array)	Fix the shape; check for a stray second emit() call
result_too_large	A single row (or the whole result) exceeds the byte cap	Trim large string fields before emitting
sandbox_error	Uncaught exception in the script	Read the thrown message; guard optional provider fields
run_code_unavailable	The run-code module isn't available in this build	Not something a program author can fix
background_pending	Background program has no result yet	Wait and retry, or check lastGoodRun if present

Every failure is a structured { code, message }, never a bare string — treat code as the thing to branch on, message as the thing to show a human.

Security notes

• Credentials are always the viewer's, never the author's. When a shared program's panel is viewed, providerFetch inside it resolves auth using the viewing user's own configured provider credentials — not the program author's. A teammate missing a HubSpot key sees a HubSpot auth error on that panel, not someone else's data.
• Only share programs you trust. Because execution uses the viewer's credentials, only bind to or view a shared data program if you trust its code — it runs with your own access, exactly like opening someone else's extension.
• Data programs are org-internal only, never public. Unlike dashboards, a data program can never be shared publicly — sharing is restricted to members of your org. This is a hard invariant, not a default.
• Raw provider tokens never reach the model or the browser. The only thing that ever leaves the sandbox is the { rows, schema } your emit() call produces — the same trust boundary as every other dashboard panel source.
• The actions that execute or mutate stored code (save-data-program, preview-data-program, run-data-program, delete-data-program) are not callable from the sandboxed extensions/iframe bridge — only from the agent or the server-rendered panel path.

Worked example: Risk Meeting (HubSpot x Pylon)

This is the reference example installed by the ensure-risk-meeting-dashboard action (templates/analytics/actions/ensure-risk-meeting-dashboard.ts), which idempotently upserts both programs below plus a two-panel "Risk Meeting" dashboard. It demonstrates the three capabilities a hardcoded action can't anticipate for every customer: an arbitrary HubSpot property IN-search, a batched multi-hop association join, and a join against a second, unrelated provider.

risk-meeting-cohort.js — HubSpot deals in a configurable risk_status cohort, resolved to their company domain, joined against Pylon account sentiment by domain:

// params: { riskStatuses?: string[], enterpriseOnly?: boolean }
const DEFAULT_RISK_STATUSES = [
  "On the Radar",
  "Churn Risk",
  "Confirmed Churn",
  "No Save Attempted",
];
const riskStatuses =
  Array.isArray(params.riskStatuses) && params.riskStatuses.length > 0
    ? params.riskStatuses
    : DEFAULT_RISK_STATUSES;
const HIGH_RISK_SENTIMENTS = ["frustrated", "high_risk_detractor"];

function chunk(items, size) {
  const out = [];
  for (let i = 0; i < items.length; i += size)
    out.push(items.slice(i, i + size));
  return out;
}

async function main() {
  // 1. Arbitrary property IN-search + open-close-date filter — not
  //    expressible through the canned hubspot-deals filters.
  const dealSearch = await providerFetchAll(
    "hubspot",
    "/crm/v3/objects/deals/search",
    {
      method: "POST",
      body: {
        filterGroups: [
          {
            : [
              {
                : ,
                : ,
                : riskStatuses,
              },
              {
                : ,
                : ,
                : (.()),
              },
            ],
          },
        ],
        : [
          ,
          ,
          ,
          ,
          ,
          ,
          ,
          ,
          ,
          ,
        ],
        : ,
      },
      : ,
      : {
        : ,
        : ,
        : ,
      },
    },
  );
   deals = dealSearch. || [];
   dealIds = deals.( d && d.).();

  
  
   companyByDeal = {};
   ( batch  (dealIds, )) {
     (batch. === ) ;
     assoc =  (
      ,
      ,
      { : , : { : batch.( ({ id })) } },
    );
     ( r  (assoc && assoc.) || []) {
       dealId = r && r. && r..;
       companyId =
        r && .(r.) && r.. >  ? r.[]. : ;
       (dealId && companyId) companyByDeal[dealId] = companyId;
    }
  }

  
   companyIds = .( (.(companyByDeal))).(
    ,
  );
   domainByCompany = {};
   ( batch  (companyIds, )) {
     (batch. === ) ;
     companies =  (
      ,
      ,
      {
        : ,
        : { : batch.( ({ id })), : [] },
      },
    );
     ( r  (companies && companies.) || []) {
       domain =
        r && r. && r..
          ? (r..).()
          : ;
       (r && r. && domain) domainByCompany[r.] = domain;
    }
  }

  
   pylonAccounts =  (, , {
    : ,
    : {
      : ,
      : ,
      : ,
    },
  });
   sentimentByDomain = {};
   ( account  pylonAccounts. || []) {
     (!account) ;
     domain =
       account. ===  ? account..() : ;
     (!domain) ;
     (
      params. &&
      account. !== 
    )
      ;
     (account.)
      sentimentByDomain[domain] = account.;
  }

  
  
   rows = deals.( {
     props = (deal && deal.) || {};
     dealId = deal && deal.;
     companyId = dealId ? companyByDeal[dealId] : ;
     domain = companyId ? domainByCompany[companyId] : ;
     sentiment = domain ? sentimentByDomain[domain] ||  : ;
     {
      : dealId || ,
      : props. || ,
      : props. || ,
      : props. || ,
      : props. || ,
      : props. || ,
      : props. || ,
      : (props. || ),
      : props. || ,
      : props. || ,
      : props. || ,
      : domain || ,
      : sentiment,
      : (
        domain && .(sentiment),
      ),
    };
  });

  (rows, [
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
    { : , :  },
  ]);
}

 ();

risk-meeting-pylon-early-warning.js — the complementary "support signal outrunning CRM signal" view: enterprise Pylon accounts already flagged at-risk by sentiment that have not yet shown up in the HubSpot cohort above (a same-shape HubSpot domain lookup, minus the Pylon domain set — see templates/analytics/seeds/data-programs/risk-meeting-pylon-early-warning.js for the full script). This is the third validation capability: excluding one provider's cohort from another's, entirely inside one program.

Both programs are installed and kept up to date by ensure-risk-meeting-dashboard — an idempotent action that upserts each program by its stable name and a two-panel "Risk Meeting" dashboard whose panels use source: "program". Installing the dashboard requires no HubSpot/Pylon credentials up front; provider auth resolves per-viewer only when a panel is actually rendered or run-data-program is called.

Related skills

• provider-api — the catalog/docs/request/staging actions a data program's providerFetch* calls are built on.
• dashboard-management — creating and laying out the dashboard a program's panel lives on.
• cross-source-analysis — identity-stitching discipline (stable id + email, de-duplication, provenance) for joins like the one above.
• security — credential handling and access-scoping invariants this primitive relies on.