| name | server-log-crawler-analyst |
| description | Analyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims. |
Server Log Crawler Analyst
Use this skill when credential-free monitoring evidence is enough, or when credentialed
Search Console, rank tracking, analytics, or assistant citation tools are unavailable.
Read references/data-source-tiers.md before producing a monitoring report.
Owns
- credential-free monitoring baseline;
- server access-log summary with IP privacy protection;
- crawler/user-agent observation reports;
- public HTTP checks for robots, llms.txt, sitemaps, RSS, and representative URLs;
- data-source tiering;
- monitoring limitations and unknowns;
- backlog handoff for Search Console, rank/SERP, analytics, and LLM citation monitoring.
Does Not Own
- credentialed Search Console or analytics API work;
- scraping search engines;
- bypassing bot protections;
- live firewall or WAF changes;
- claims about rankings or assistant citations without direct evidence;
- storing raw IP logs in public artifacts.
Workflow
- Identify available evidence: public URL fetches, access-log snippet, exported summary, or synthetic fixture.
- Label the evidence tier and collection time.
- If logs are provided, summarize requests by path, status, user-agent class, and redacted IP hash.
- Separate known crawler hits from unknown bot-like traffic.
- Check public discovery endpoints when requested.
- Record limitations: no Search Console, no rank data, no assistant citation data unless explicitly provided.
- Produce a monitoring report using assets/crawler-monitor-report.template.md.
- Hand off credentialed or citation tasks to the backlog.
Non-Negotiables
- Do not publish raw IP addresses.
- Do not store credentials, cookies, API keys, or private analytics exports in skill files.
- Do not infer ranking, indexing, or citation success from server logs alone.
- Do not claim a user-agent is verified Google/OpenAI/Claude/Perplexity unless verification method is documented.
- Do not scrape SERPs or assistants against terms of service.
- Do not change firewall, WAF, robots, or server config from this skill.
Safety And Privacy Boundaries
- Treat raw logs as private by default.
- Use aggregated counts and redacted identifiers.
- Keep exact timestamps only when needed for evidence; otherwise aggregate by hour/day.
- If the user asks to publish a report, remove raw IP, query strings, session IDs, auth paths, and admin URLs unless explicitly needed and safe.
- Public reports should say "observed in provided logs" rather than "definitely crawled by" unless bot verification is complete.
Evidence Labels
Observed: directly in log, HTTP response, or exported report.Verified: observed and checked by an accepted verification method.Inferred: likely based on user-agent/path/status, but not verified.Open: no evidence available in this run.
Validation
Validate skill edits with:
python3 <codex-skills-dir>/senior-skill-architect/scripts/lint_production_skill.py ./skills/server-log-crawler-analyst
python3 ./skills/server-log-crawler-analyst/scripts/analyze_access_log.py ./skills/server-log-crawler-analyst/fixtures/sample-access.log --report <tmp-dir>/server-log-crawler-report.json
python3 ./plans/seo-llm-skill-cluster/scripts/lint_skill_cluster.py .
Forward tests live in evals.json.
Output Shape
Return:
- Evidence source and timestamp.
- Data tier and privacy handling.
- Observed crawler/user-agent classes.
- Public endpoint status checks if performed.
- Suspicious or irrelevant bot traffic summary.
- Unknowns and claims refused.
- Monitoring backlog and next evidence needed.
