perlantir-fleet

Internal admin tools for inspecting, monitoring, and improving judge outputs — including raw output inspection, calibration drift detection, missing evidence detection, low-signal output flagging, and a feedback quality dashboard.

Block filler, detect low-signal text, score specificity, and enforce evidence-anchored observations across all LLM-generated feedback in Bouts — with banned phrase detection, specificity scoring, retry-with-critique, and dimension name normalization.

Generate insights comparing an agent's performance to top performers, peers, and their own history — only from real data with minimum sample guards, including percentile comparisons, counterfactual rank calculation, and "surprisingly strong" lane detection.

Evaluation-specific visualization patterns for Bouts — radar charts, multi-judge comparison bars, rank distribution dots, confidence overlays, and percentile bands using Recharts with full accessibility and missing-data handling.

What users actually need after losing or winning a competitive AI evaluation — emotional state design for winners, close misses, and clear losses, with TSX layout and copy patterns that make feedback feel fair rather than algorithmic.

Expose when a judgment is high-confidence vs thin-evidence without undermining the platform's authority — covering the confidence trilemma, per-tier UI patterns, data model, copy patterns, and hard rules on when NOT to show confidence indicators.

Recharts-based data visualization in Next.js/React — percentile bars, trend lines, comparative charts, responsive containers, and accessibility standards.

Design and implement the analytics pipeline powering lane score distributions, repeated weakness patterns, agent progress over time, and challenge-level learning analytics — using materialized views, pg_cron refresh, and a clean separation between operational and analytical tables.

Design lane structure, scoring dimensions, weighting logic, calibration rules, and anchor-based criteria so Bouts produces defensible, consistent AI judgments instead of vibes.

Convert raw judge outputs into premium, evidence-anchored user feedback — with suppression rules, fallback logic, contradiction handling, and zero generic filler.

How to make dense evaluation output readable fast — the 3-second rule, progressive disclosure architecture, scan-first layout, expandable evidence panels, and concrete before/after redesign for Bouts result pages.

Schema design for Bouts feedback data — model for future questions not just current queries, normalize evidence refs, build audit trails, know when to use JSONB vs columns, and evolve schemas safely without breaking existing records.

Purpose-built LLM classifier design for failure mode taxonomy — 15-code classification with confidence scoring, anti-convergence patterns, evidence anchoring, and anti-generic prompt enforcement.

Turning Bouts evaluation breakdowns into coaching items that are concrete enough to act on — specificity ladder, failure code → next step derivation, deduplication across submissions, and a priority-ordered TSX coaching display component.

Define the raw output contract for Bouts AI judges — lane scores, dimension breakdowns, evidence refs, confidence, flags, and integrity adjustments — with Zod validation, SQL storage, and multi-judge reconciliation logic.

Surface structured coaching insights across multiple bouts for the same competitor — repeated failures, recurring strengths, trendline improvement, and same-lane persistence — grounded in real data with suppression rules that prevent surfacing patterns too early.

Multi-stage async LLM pipeline design — stage isolation, structured handoffs, idempotency, concurrency-safe profile updates, and failure handling across chained LLM calls.

Render partial, legacy, missing, and evolving evaluation result data without crashes — covering every null/undefined edge case in Bouts lane scores, evidence refs, confidence fields, and partial judge results.

Design the Bouts post-match breakdown page — information hierarchy, component architecture, partial result handling, loading states, mobile/desktop layout, and psychological flow that makes users feel the result is trustworthy and earned.

Open-window ranking, provisional placement, finalization triggers, edge case handling (ties, disqualifications, late submissions), rank history storage, and clear status communication for Bouts.

Full system design for Bouts evaluation replay — event model, SQL schema, TypeScript discriminated union, phase grouping, evidence ref linking, legacy record handling, and a virtualized TSX timeline component for 500+ events.

Track which feedback blocks users actually engage with — expand, copy, revisit, dwell on — and use that engagement signal to improve feedback structure over time without compromising user trust.

Making the Bouts scoring system legible to competitors — evidence vs inference labeling, provisional vs final copy, methodology disclosure, and dispute acknowledgment patterns that open the black box without overwhelming users.

The Bouts first 90-day marketing operating plan with week-by-week content, distribution, enterprise outreach, and growth targets from 0 to 500 agents enrolled and first data licensing revenue. Use when planning or executing the Bouts launch phase marketing operation.

Systematically test headlines, content formats, CTAs, channels, and posting timing for Bouts marketing with a one-variable-at-a-time discipline, minimum sample sizes, winner criteria, and immediate application of results. Use when running any Bouts content optimization test.

Systematic 4-week outreach system for AI labs (Anthropic, OpenAI, Google, Meta, Cognition, Cursor, and 13 others) to sell Bouts data licensing and private benchmarks — from warm-up through commercial conversation. Use when prospecting AI labs, writing outreach copy, or building the pipeline for data licensing revenue.

Track Bouts marketing performance weekly across growth, content, and revenue metrics with UTM attribution, a structured weekly analytics report, and decision rules for scaling or cutting channels. Use when reporting on Bouts marketing performance, diagnosing what is or is not working, or building the analytics infrastructure for the Bouts launch.

The self-executing weekly Bouts marketing operation that runs Monday through Friday without manual triggers — data pull, content production, publishing, distribution, community engagement, and analytics review. Use as the master operational guide for running the Bouts content machine autonomously each week.

Market the Bouts Benchmark API to AI labs with the right value proposition, API documentation strategy, landing page copy structure, and quick-start framing. Use when writing API documentation, the /benchmark landing page, or any outreach targeted at AI labs that need programmatic access to contamination-resistant evaluation.

Market Bouts agent certification tracks to enterprises and individual builders as independent proof of capability, with landing page copy, badge system, API verification, and the enterprise procurement angle. Use when creating certification marketing materials, writing the /certification landing page, or building the enterprise sales motion around verified agent capability.

Respond to competitive moves, benchmark launches, methodology criticism, and market changes as Bouts' calm, data-rich voice in AI evaluation debates. Use when a competitor launches, a critic raises objections, an AI lab releases its own eval, or any situation requiring a public competitive response.

Monitor AI benchmark competitors weekly including SWE-bench, HumanEval, LiveCodeBench, Aider, CodeClash, and new entrants — tracking methodology changes, adoption, community growth, and market signals that should feed into Bouts content, positioning, or feature decisions.

Plan and execute Bouts conference presence at NeurIPS, ICML, AI Engineer Summit, and developer conferences with talk submissions, poster proposals, live competition events, and follow-up sequences. Use when planning conference strategy, writing talk abstracts, or maximizing the business value of any Bouts conference presence.

Prepare Bouts content for international reach starting with language-neutral English and defining the localization roadmap for Chinese, Japanese, Korean, German, and French markets by AI developer density. Use when writing English content to ensure it is globally accessible, or when planning future localization efforts.

Track every Bouts content piece, run monthly performance reviews, identify patterns, and systematically improve content mix based on what actually drives agent signups and enterprise inquiries. Use when conducting monthly content reviews, spotting high/low-performance patterns, or deciding where to shift content investment.

Turn every Bouts weekly intelligence report and technical blog post into 10+ channel-specific pieces for X, LinkedIn, email, Reddit, HN, Discord, and social sharing. Use when maximizing reach from existing Bouts content and enforcing the rule that nothing is created once for one channel.

Optimize every stage of the Bouts conversion funnel from awareness to retention with specific targets, copy guidance, friction rules, and measurement approach. Use when diagnosing drop-off, writing onboarding copy, or improving the visit-to-active-competitor rate.

Respond to Bouts crises including methodology criticism, challenge exploits, AI lab complaints, downtime, and negative press with specific response templates, escalation rules, and the principle of always responding with data and transparency instead of defensiveness.

Coordinate Launch with Gauntlet (data), Pixel (visuals), Counsel (compliance), MaksPM (pipeline), Scout (intel), and Maks (product) with specific requests, cadence, and handoff formats. Use when requesting assets from other agents, reporting status, or coordinating multi-agent deliverables for Bouts marketing.

Write and distribute content that sells Bouts data licensing tiers (Index Access at $2K/mo, Benchmark API at $5K/mo, Private Lane at $10K/mo, Enterprise custom) to AI labs and enterprises through specific pitch copy and value propositions per tier. Use when writing sales one-pagers, landing page copy, or outreach materials for the data licensing business.