Skip to main content

agent-traffic: classify agents and crawlers

Classifies HTTP requests as agents, AI crawlers, search crawlers, bots, or humans with the agent-traffic dataset and package.

Source facts

Repository
mintlify/agent-traffic
Last source activity
November 10, 2026 at 18:43
Detected SKILL.md language
English
Stars
78
Forks
3

Examples

Documented test cases in the pinned SKILL.md (not a SkillsMP site test, and not runtime-verified in this draft): curl/8.7.1 should classify as bot / http-client; a listed desktop Chrome user agent should classify as human (or no match in SQL); a listed VS Code Electron user agent must not be classified as GitHub Copilot; Claude-Code-Docs-Fetcher/3.0 must not be classified as Claude Code. See the Test it table in skills/agent-traffic/SKILL.md at the pinned HEAD.

Uses

Use this Skill when adding agent or bot detection to request handlers, middleware, analytics, server logs, or SQL queries, or when someone asks how much of their traffic comes from agents.

Prerequisites

Every result needs at least the user agent. The Accept header and the real client IP make classification much more accurate; if logs can be extended, the Skill recommends logging those fields. For JavaScript or TypeScript at request time, the Skill documents npm i @mintlify/agent-traffic. That install command was documented only and was not run. The site's general install module covers Skill-file installation. GUIDELINES.md, the dataset JSON files, and src/classify.ts are cited in the Skill but were not attached and were not used as pinned capabilities; the bound facts are those stated in the pinned SKILL.md. This draft did not execute the Skill or the package.

How to use

Ask the agent to classify traffic or to wire agent-traffic into your stack, and show it where requests are handled or how logs are stored. The agent first checks which fields exist and whether an IP is the real client or a proxy address, then picks request-time classification, a lookup table over distinct user agents, or a generated SQL CASE expression. It should test the implementation against the examples documented in the Skill before reporting the work done, and it must keep agent and ai-crawler counts separate when it reports numbers. Those integration and test steps are agent work, not prerequisites you must already have completed.

Limitations

Do not reimplement the rules when the package can be used. Use the real client IP from a trusted proxy header such as CF-Connecting-IP; never use the leftmost X-Forwarded-For value, which the client controls. Only pass headersComplete true when browser headers arrive intact. Do not call unverified identities verified: with no real client IP, every identity is self-declared. Keep agent and ai-crawler separate. A markdown fetch by curl, axios, or a script is a bot, not an agent, unless it identifies itself or asks for markdown in Accept. Do not match bare product words, IP-check entries marked runsOn user, treat a stale range-file mismatch as spoofing when the file is older than 180 days, or copy a pattern list into code by hand. Agents that send a browser user agent and ask for markdown will show as human in user-agent-only SQL unless the markdown Accept rule is also applied; the Skill says to state that in any report. The parent thread's “489M requests” and the middle tweet's “98 agents and crawlers” wording are author or post claims and are not in the pinned SKILL.md. The parent and middle tweets are context only and are not second posts entries. The repository is also a dataset and library; this card covers only skills/agent-traffic/SKILL.md. The repository-root README is an unattached library README and is not in sourceUrls. Official reuse is unknown; no official production target id was invented. This draft did not execute the Skill or install the package.

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

Showing SKILL.md

SKILL.md
Source instructions · Read-only preview
nameagent-traffic
descriptionClassify HTTP traffic as agents, AI crawlers, search crawlers, bots or humans using the agent-traffic dataset. Use when adding agent or bot detection to request handlers, middleware, analytics, server logs or SQL queries, or when someone asks how much of their traffic comes from agents.