| name | d-research |
| description | Browser-first deep research and lawful public-data collection for AI agents. Use for web research, source discovery, public-data scraping, literature or market/technical research, due diligence, policy/standards analysis, cultural research, atomic fact verification, single-URL inspection, public social-post archival, public-role person lookup, semantic corpus retrieval, evidence ledgers, execution gates, and blocker reports. Read-only; never bypasses logins, paywalls, captchas, robots restrictions, or rate limits. |
D Research
Mission
Maximize reachable public evidence under available tools and open-web constraints.
Default browser automation: Playwright.
Use for deep web research, public data collection, source discovery, academic
and literature review, market/technical research, due diligence, policy and
standards analysis, creative/cultural research, dynamic-page evidence, and
blocker reports.
Never use this skill to bypass access controls, login walls, paywalls, captchas,
rate limits, robots restrictions, or explicit access restrictions.
Safety invariants (never allowed)
- Bypass login, authentication, paywalls, or subscription checks.
- Solve or evade captchas. Config
access.allowCaptchaSolving must be false;
true fails validation. Captcha solving is never allowed.
- Stealth plugins or anti-detection. Config
access.allowStealthEvasion must be
false; true fails validation. Stealth evasion is never allowed.
- Evade rate limits or anti-bot systems.
- Stolen cookies, leaked tokens, or credentials not provided by the user.
- Private/personal/sensitive data without authorization.
- Ignore robots when acting as a crawler (
--no-respect-robots hard-fails).
- External mutation without explicit opt-in (Wayback Save Page Now requires
--submit-archive).
Lawful authenticated access with user-provided credentials is distinct from
bypass/evasion and remains read-only by default.
Full policy: references/safety-and-access-policy.md.
Tool priority by phase
- Discovery: user context → web search.
- Probe: Playwright/browser → fetch → web-search-only fallback.
- Extraction: public file/API/static DOM → browser-rendered content.
Anti-bot blocked tier-1 public sources: run the bounded chain in
references/anti-bot-fallback.md once (API/static → archive → cache/snippet →
fetch-only → blocker). Never use it to bypass controls.
Adapter policy: references/tool-adapter-policy.md.
Data access layers
- Web pages and files (browser/fetch)
- Public APIs (REST/GraphQL/SPARQL) —
references/api-access-workflow.md
- Academic databases —
references/academic-databases.md
- Wikidata —
adapters/wikidata.md
- Read-only databases when user provides access —
adapters/database-readonly.md
- Specialized domains —
references/specialized-domains.md
Intake and routing
Step 0: Classify with references/research-intake.md before opening sources.
Assign shape labels, depth (fast/standard/completeness-first), safety posture,
output artifact, authority basins, required ledgers/gates. Ask the user only when
ambiguity changes safety, legality, scope, or deliverable.
Long-horizon outer loop: When more than 5 sub-questions, more than 50 sources,
multi-context runtime, or audit-grade output, use research plan schema 2.0
(references/research-plan-protocol.md):
node scripts/run_python.mjs scripts/research_plan.py init --slug <slug> [--title "..."] (generic draft)- Fill tasks with
phase: research|synthesis, sub-questions, outputs under
research-output/
configure-execution → render → gate --gate plan_ready → approve →
gate --gate execute_ready- Dispatch only after
execute_ready/dispatch_ready passes; write findings to
declared outputs immediately
gate --gate synthesize_ready (research phase only; real HMAC via
D_RESEARCH_LEDGER_KEY; completed checklist)- Compose report from
report.draft.md / section inputs
gate --gate release_ready (synthesis complete; exact report path; 100 percent
claim coverage; stopping criteria)
Migrate v1 plans: scripts/research_plan.py migrate. Always report workspace path.
Before final synthesis: Apply references/execution-gates.md unless a
fast-path branch says otherwise. Do not claim completeness unless gates pass.
Route table
| Route | Reference |
|---|
| Atomic fact | references/fact-verification.md |
| Social post archive | references/social-media-archival.md |
| Named public-role person | references/person-aggregation.md |
| Semantic corpus query | references/semantic-retrieval.md |
| Broad multi-source research | references/workflow-routes.md |
| Due diligence / red flags | intake due_diligence_or_investigation |
| Policy / standards / RFC | intake policy_or_standards_analysis |
| Creative / cultural | intake creative_or_cultural_research |
| Technical research | references/source-discovery.md, references/workflow-routes.md |
| Market / competitor | references/source-discovery.md, references/source-quality-rubric.md |
| Legal / government / financial | references/specialized-domains.md |
| Medical / safety | references/specialized-domains.md, references/source-quality-rubric.md |
| Dataset collection | references/data-extraction-toolbox.md, references/data-processing-pipeline.md |
| Academic / literature | references/academic-research-protocol.md, references/citation-management.md |
| Systematic / PRISMA | references/systematic-review-protocol.md |
| Single URL | references/browser-first-crawl.md, references/anti-bot-fallback.md |
| API collection | references/api-access-workflow.md |
| Large-scale (100+) | references/large-scale-collection.md |
| Monitoring / change detection | references/monitoring-change-detection.md |
| Visualization / rendered report | references/data-visualization.md, references/report-generation.md |
| Multilingual / Vietnamese | references/multilingual-research.md, references/vietnamese-source-discovery.md |
| Thin recall / jargon | references/register-and-jargon-expansion.md |
| Evidence gaps | references/frontier-search.md |
Narrative branch detail: references/workflow-routes.md.
Machine-readable routes: templates/route-manifest.json.
Core deep research workflow
- Restate goal, entities, timeframe, geography, language, output, source constraints.
- Decompose (
references/topic-decomposition.md): sub-questions, facets, aliases,
source classes, stopping criteria.
- Source map (
references/source-discovery.md): official, primary, papers, APIs,
datasets, archives.
- Query fanout (
references/query-patterns.md): broad, exact, official, primary,
filetype, site, dataset, recent, contradiction; register variants when needed.
- Probe with browser-first access; classify access state; never force blocked pages.
- Extract least-invasively: public files → public APIs → static markup → rendered text.
- Expand via links/sitemaps/APIs within crawl limits; respect robots.
- Maintain evidence ledger (
references/evidence-ledger.md); sign with
scripts/evidence_ledger.py sign for long-horizon plans and audit-grade work.
- Contradiction pass; score sources (
references/source-quality-rubric.md).
- Blocker reports (
references/blocker-report.md) for unreachable tier-1 sources.
- Synthesize only after gates; use
references/final-report-template.md.
Default crawl limits: depth 2, 30 pages/domain, 100 total, 1000 ms delay, robots true.
Adapters and fallbacks
Default adapter: adapters/playwright.md. Alternatives:
adapters/generic-browser.md, adapters/fetch-only.md,
adapters/web-search-only.md, adapters/graphql.md,
adapters/citation-resolver.md, adapters/translation.md.
If Playwright is unavailable, use the configured browser adapter. If no browser
exists, use fetch. If fetch is unavailable, use web search and mark limitations.
Blocked relevant public tier-1 sources: one pass of
references/anti-bot-fallback.md then references/blocker-report.md. Record
failed attempts as low-confidence process ledger rows.
Crawl and expansion defaults
When acting as a crawler:
- max depth: 2
- max pages per domain: 30
- max total pages: 100
- delay between page loads: 1000 ms
- respect robots: always true (policy hard-fail if disabled)
- follow external links: false unless needed for source discovery
- TLS verification on by default;
--ignore-tls-errors is opt-in and must be
recorded as a limitation
Plan gates summary (schema 2.0)
| Gate | Meaning |
|---|
| plan_ready | Complete filled plan, rendered PLAN.md, not started |
| execute_ready / dispatch_ready | Approved and ready to run |
| synthesize_ready | Research phase terminal; ledger valid+HMAC; checklist complete |
| release_ready | Synthesis done; exact report+citations; full claim coverage |
Canonical assertion sets live in templates/route-manifest.json and are enforced
by scripts/research_plan.py. Standard gates cannot be emptied.
Evidence and output contract
Every important claim needs source, type, dates, access method, evidence,
contradiction status, and confidence. Separate facts, inferences, speculation,
and unknowns.
Ledger rows may set record_type: claim (default), process, or blocker.
Release requires full narrative coverage of claim rows via [ref:claim_id] in
authored text only (not generated Evidence Summary or References blocks).
For broad and non-trivial routes, the final answer includes: direct answer, key
findings, evidence summary, data collected, sources reached/blocked,
contradictions/caveats, confidence, and next steps. Narrow fast paths follow
their branch-specific output contract instead of manufacturing unused sections.
Never present results as complete unless the relevant execution gates passed.
Render/lint: scripts/report_render.py. Claim coverage:
report_render.py lint --workspace <dir> [--report <exact-path>] --strict.
High-stakes and privacy boundaries
Hard-stop before broad research when intake indicates refusal, access-control
bypass, private-person profiling, minors, harassment/stalking/doxxing framings,
or high-stakes advice that must be reframed as evidence synthesis.
Person aggregation (references/person-aggregation.md) saturates at 25 ledger
rows or three sources adding no new verified claims. Never re-identify
pseudonyms; never aggregate home address, family, personal contact, photos,
medical/financial/orientation/whereabouts.
Social archival (references/social-media-archival.md) refuses the same privacy
boundary before any HTTP call. Tier A uses direct public APIs; Tier B is
archive lookup-only unless --submit-archive.
Signing and reproducibility
For every long-horizon plan workspace, and for audit-grade work on any route:
- Maintain
evidence-ledger.csv (23-column canonical; legacy 14/19/22 OK).
- Sign with
scripts/evidence_ledger.py sign --file evidence-ledger.csv --key-env D_RESEARCH_LEDGER_KEY.
- Complete
reproducibility-checklist.md (no unchecked boxes; N/A as checked with reason).
- Render:
scripts/report_render.py render --workspace <dir>.
- Lint exact report:
scripts/report_render.py lint --workspace <dir> --report <path> --strict.
- Gate release:
scripts/research_plan.py gate --gate release_ready.
Tampered ledgers with stale HMAC sidecars must fail every release gate.
Optional helpers
Scripts under scripts/ are optional. Key entry points:
- Browser:
playwright_probe.mjs, playwright_extract.mjs, playwright_crawl.mjs
- Plan/report:
research_plan.py, report_render.py, evidence_ledger.py
- Network:
api_fetch.mjs, web_search.mjs, http_cache.py
- Academic:
citation_export.py, citation_render.py, citation_resolver.py
- Social/archive:
social_snapshot.py, wayback.py
- Quality:
score_source.py, run_dogfood.py, check_contract.py
Full inventory: references/script-inventory.md.
Configuration
Obey project research.config.json or research.config.example.json defaults.
Access is read-only. Captcha solving and stealth evasion are never allowed.
Field reference: references/config-reference.md.
Compatibility notes
- Research plan v1 loads with a one-shot deprecation warning until v4; run migrate.
- Ledgers with 14, 19, or 22 columns remain valid; missing
record_type is claim.
- CLI alias
--paginate remains with one deprecation warning; prefer --pagination.
- Root
report.md is deprecated; prefer declared outputs under research-output/.
Further reading
- Methodology:
references/research-bibliography.md
- Routes:
references/workflow-routes.md
- Scripts:
references/script-inventory.md
- Config:
references/config-reference.md
- Contributors:
CONTRIBUTING.md
