recoup-internal-task-email-audit

name

recoup-internal-task-email-audit

description

INTERNAL — Recoup staff tooling. Never use for customer-facing or artist requests. Audit the health of scheduled-task emails over a time window: what the customer-prompt-task runs actually delivered, how many empty/malformed sends the API guard blocked, per-account/per-recipient breakdown, and whether agents used the reliable helper vs hand-rolled curl. Use when asked "how did the tasks run last night", "what emails went out from tasks", "are we still sending empty emails", "task email health", "did the empty-email fix hold". Produces a per-run table (one row per Trigger.dev `customer-prompt-task` run: run_id, recipient, prompt, email subject, body kb) plus a headline, a per-run **tool-call trace**, and a **`has_hallucinated_data`** / `expected_data_source` content check. Reads the Trigger.dev run list + `email_send_log` and `chats`/`chat_messages` (Supabase).

Task-email health audit

How Recoup staff check whether scheduled task emails are healthy — real reports delivered, no empty "Message from Recoup" footer-only sends — over any window. This is the standing audit for the pipeline fixed in recoupable/chat#1829 (empty-email bug, resolved 2026-07-01).

When to use

• "recoup-internal — how did last night's tasks go? What emails went out?"
• "Are we still shipping empty footer-only emails?"
• "Task email health for the last 24h / this week."
• "Did the #729 guard actually block empties in production?"
• "Is the data in this task email real, or hallucinated? Why did it fabricate?" (→ §§ 4–5)
• "Show me the exact tool calls a task run made." (→ § 4)

Background (what you're measuring)

Scheduled tasks (customer-prompt-task, Trigger.dev) each run the runAgentWorkflow agent, which sends its report by calling POST /api/emails from inside the sandbox. Before the 2026-07-01 fix, a malformed/empty body was silently delivered as a footer-only "Message from Recoup" email with success:true. The fix: POST /api/emails now rejects empty/unparseable bodies (400) and logs every attempt to email_send_log (sent / send_failed / rejected) with the full raw body. That log is the audit's source of truth.

Data sources

A. Task runs — Trigger.dev Management REST API (the run list + run_id)

Every scheduled task run is a Trigger.dev run of the customer-prompt-task task (lib/trigger/createSchedule.ts), tagged account:<accountId>. Enumerate them with the bundled script — it pages the Management API, filters to customer-prompt-task in the window, and prints the account array for the correlation SQL:

node scripts/fetch_task_runs.mjs --hours 24 > runs.json   # add --account <uuid> for one account

Requires the PROD TRIGGER_SECRET_KEY — a tr_dev_* key returns 0 prod runs (the #1 gotcha; the script warns you). Pull it without clobbering your .env, from the recoupable/api repo (linked to recoup/api):

vercel env pull /tmp/prod.env --environment=production --yes
export TRIGGER_SECRET_KEY=$(grep '^TRIGGER_SECRET_KEY=' /tmp/prod.env | cut -d= -f2- | tr -d '"')
# … run the script …
rm /tmp/prod.env                                          # scrub the secrets when done

runs.json rows are { run_id, account, status, createdAt, finishedAt }. For the exact per-run prompt, runs.retrieve(runId) (@trigger.dev/sdk) has payload.prompt; but scheduled_actions.prompt per account (in the SQL below) is cheaper and usually sufficient.

B. Emails + agent activity — Supabase (project godremdqwajrwazhbrue)

• email_send_log — the send-attempt log. One row per /api/emails attempt: status (sent | send_failed | rejected), account_id, chat_id, resend_id, raw_body (full request body), created_at. Exists only from 2026-06-30 (when api#731 shipped).
• chats / chat_messages / sessions — every run creates a chat; the agent's bash/tool activity (how it sent) is in chat_messages.parts.
• scheduled_actions — the scheduled-task definitions (title, prompt, account_id). Resolve a run's prompt/title without a runs.retrieve call.

⚠️ Do NOT use workflow_runs or scheduled_actions.last_run/next_run — stale (not maintained by the current scheduler; workflow_runs stopped updating 2026-05-27). Run history is Trigger.dev; activity is chats/chat_messages.

Query Supabase via the MCP (execute_sql) or psql/DATABASE_URL. Read-only.

The audit (run these, newest-first)

Substitute the window in interval '24 hours'. Always exclude test noise — sends to [e-mail] and preview-auth-probe bodies are staff test runs, not customer traffic.

1. Headline — tasks run, delivered vs. blocked, guard-block rate, forced retries

tasks_run = the number of customer-prompt-task runs in the window — count them from the Trigger.dev run list (§ A). The remaining columns come from email_send_log:

with w as (
  select status,
    substring(raw_body from '"to"\s*:\s*\[\s*"([^"]+)"') as rcpt,
    created_at
  from public.email_send_log
  where created_at > now() - interval '24 hours'
    and raw_body not like '[e-mail]%'
    and raw_body not like '%preview-auth-probe%'
)
select
  count(*) filter (where status='sent')                                   as delivered,
  count(*) filter (where status='rejected')                               as blocked_empty,
  round(100.0*count(*) filter (where status='rejected')/nullif(count(*),0),0) as pct_blocked,
  count(distinct rcpt) filter ( status)                        recipients_served,
  ()  ( status
     rcpt  ( rcpt  w  status))                  blocked_then_delivered;

tasks_run ≠ email tasks — segment the noise. Many customer-prompt-task runs are not email tasks: profile-sync/maintenance (one account fired 33 runs in 24h), health-checks, and misconfigured prompts (literally "22", or "New task — replace with your instructions."). Report tasks_run (all runs) and email_tasks (prompt mentions email/send and the account resolves a recipient) separately, and list the non-email / high-frequency / misconfigured ones as a short "ops noise" callout — otherwise the delivery rate is judged against a polluted denominator (the first real run read "6 of 90", but ~33 of those 90 were one maintenance task).

2. Per-run table — one row per Trigger.dev run (the main deliverable)

Columns: run_id · recipient · prompt (truncated) · subject · body (kb). Paste the array[...]::uuid[] that the script printed into this one correlation query (returns recipient + prompt + delivered subject/body per account — this exact query is validated, mind the inline gotchas):

with racct as (
  select unnest( /* paste the account array from fetch_task_runs.mjs here */ ) as account_id
)
select substr(ra.account_id::text, 1, 8)                          as acct,
       ae.email                                                   as recipient,
       left(sa.prompt, 60)                                        as prompt,
       es.subject,
       es.body_kb
from racct ra
-- account_emails has NO created_at → limit 1, no order
left join lateral (select email from public.account_emails
                   where account_id = ra.account_id limit 1) ae on true
left join lateral (select prompt from public.scheduled_actions
                   where account_id = ra.account_id order by updated_at desc limit 1) sa on true
-- the delivered email: newest 'sent' row for the account in the window
left join lateral (
  select substring(raw_body from '"subject"\s*:\s*"([^"]{0,60})')  subject,
         round(length(raw_body)  , )                       body_kb
   public.email_send_log
   account_id  ra.account_id  status  
     created_at  now()   
    created_at  limit 
) es  
  (es.subject  ) , ra.account_id;   

Then join runs.json to this result by account to fill run_id. Notes:

• subject is null ⇒ the account delivered no email in the window (ran but didn't send, or its send was a blocked empty — rejected rows carry a null account_id, so they don't show here; the § 1 headline counts them).
• An account with >1 run in the window (a task firing hourly) maps to one account row here — attribute the delivered email to the run whose finishedAt is nearest the send.

Render:

run_id	recipient	prompt	subject	body (kb)
run_abc…	[e-mail]	"Daily social trends for PinkPanth…"	Daily Social Trends: PinkPantheress – July 1	15.2
run_def…	[e-mail]	"Leonardo daily social trends…"	— (blocked, no send)	—

Join caveat. email_send_log.chat_id is null (agents don't pass it), so the run→email link is account + timing, not an exact key — reliable when an account fires one task per window, ambiguous when several fire together. Threading a real chat_id into /api/emails (the footer-404 follow-up) makes run_id → email exact.

3. Delivered reports (the real emails that went out)

select
  substring(raw_body from '"subject"\s*:\s*"([^"]{0,70})') as subject,
  substring(raw_body from '"to"\s*:\s*\[\s*"([^"]+)"')     as recipient,
  account_id, to_char(created_at,'MM-DD HH24:MI') as at
from public.email_send_log
where status='sent' and created_at > now() - interval '24 hours'
  and raw_body not like '[e-mail]%'
order by created_at desc;

4. Per-run tool-call trace — why an email is empty / rich / hallucinated

To debug a single run (what it actually fetched, whether it fabricated), pull its tool calls from chat_messages.

First, find the run's chat. email_send_log.chat_id is null, so locate the chat by recipient (or a subject marker):

select chat_id, min(created_at) started
from public.chat_messages
where created_at > now() - interval '24 hours'
  and parts::text like '[e-mail]%'
group by chat_id order by started desc;

Then extract the tool calls. ⚠️ chat_messages.parts is an object {id, role, parts:[…]}, not a bare array — unnest parts::jsonb->'parts'. A run is usually one assistant message holding every part:

select row_number() over (order by ord) as n,
  p->>'type' as tool,
  left(coalesce(p->'input'->>'command', p->'input'->>'skill', p->'input'->>'url',
                p->>'text', (p->'input')::text), 200) as input_or_text,
  left(coalesce(p->'output'->>'stdout', p->'output'->>'content',
                (p->'output')::text, p->>'state'), 200) as 
 public.chat_messages cm,
   jsonb_array_elements(
      jsonb_typeof(cm.parts::jsonb)  cm.parts::jsonb
          cm.parts::jsonb 
  )  ordinality  t(p, ord)
 cm.chat_id  
   p  (,,,,)
  ord;

The text parts are the agent's narration — usually the smoking gun (e.g. "the API doesn't have direct CPM metrics, I'll generate … sample data").

5. Content audit — is the reported data real? (has_hallucinated_data)

For each delivered email, set has_hallucinated_data from the run's tool calls (§ 4): a metric is hallucinated if it isn't backed by a successful data-fetch this run. Red flags:

• the data call errored / returned empty (/socials → "Artist not found", organizations: []) yet the email still reports numbers;
• no web_fetch in a trends/research task (used_web_fetch=false);
• the HTML/narration contains "sample data" / "estimated" / "industry average" / "realistic data" / chart-placeholder;
• the agent used the wrong id for a sub-resource (gotchas below) → 404 → no data.

Report two columns alongside the delivered table: has_hallucinated_data (bool; mark ✅ verified vs ⚠️ inferred) and expected_data_source (where accurate data should come from — YouTube CPM → YouTube Analytics connector; trends → web_fetch + platform trending; followers/streams → Recoup socials / Research).

Id gotchas when spot-checking real data (skip these and you'll log false "no data"):

• /api/artists/{id}/* sub-resources key on the account_id, not the list's top-level id. And socials are embedded in /api/artists (account_socials) — check there before calling /socials at all.
• POST /api/socials/{id}/scrape keys on social_id, not the id field.

How to read it

• delivered = real reports that reached customers. Spot-check subjects in query 2 — they should carry a real title + the correct date (a wrong year, e.g. "2024", is a stale/hallucinated-date content bug worth flagging).
• blocked_empty = empty/malformed sends the #729 guard stopped. Every one of these would have been a footer-only "Message from Recoup" email delivered to a real customer before 2026-07-01. This is the guard working, not a failure.
• blocked_then_delivered = the agent's first attempt was empty (blocked), then it retried into a real send to the same recipient — the guard's 400 forced a correction.
• pct_blocked = the underlying empty-generation rate; the guard makes it harmless.

Caveats

• Classify every run sent / correct no-send / failed — never assume no-send = failure. Some runs aren't email tasks at all, and some prompts are explicitly conditional ("if no new video, send nothing"). Read the run's final text part (§ 4): an honest no-op conclusion is correct behavior.

• rejected ≠ always guard-blocked. An expired sandbox ephemeral key (runs

  ~20 min) logs every retry as rejected too — one run can emit dozens. Check the trace: 401 "Unauthorized" = key expiry (delivery-infra); "a non-empty html or text body is required" = a real guard block.

• Part counts can shrink mid-run — a workflow-step retry resets the persisted assistant message. Re-query before concluding a run's final state.

• Needs the PROD TRIGGER_SECRET_KEY — a tr_dev_* key returns 0 runs; the script warns you. Pull it to a scratch file and scrub it after (see § A). If vercel env pull fails with an invalid-token error, the CLI login has expired — ask the operator to run vercel login (or paste the key) before proceeding.

• email_send_log.chat_id is null on nearly all sends — agents don't pass it. So you can't join sends to a specific run/task by chat_id; attribute by recipient and account_id (on sent rows) instead. This null also means task emails currently render no footer "continue on Recoup" link — a known follow-up.

• account_id is null on rejected rows — the guard runs before auth, so a 400 has no resolved account. Attribute blocked empties by recipient/subject.

• Always exclude staff test traffic ([e-mail], preview-auth-probe).

• No email_send_log data before 2026-06-30.

Reference

• History: chat#1829 (delivery, resolved 2026-07-01) · chat#1833 (hallucination, resolved 2026-07-02) — baselines, fix PRs, and benchmarks live there.