| name | ingest-link |
| description | Use when the user wants to ingest web URLs, online PDFs, rendered web pages, standards, manuals, or articles through qt-web-extractor into the normal document-ingest workflow. |
Ingest Web Links
Use this skill when the user already has one or more URLs and wants them pulled into the ScholarAIO library as document-style records.
When to Use
Use this skill when the user wants to:
- ingest a webpage directly into the local knowledge base
- ingest an online PDF or report from a URL
- capture technical documentation, standards, manuals, or web articles as
document items
- route rendered web content through the normal ScholarAIO ingest/index flow
Do not use this skill when:
- the task is only to search the local library; use
search
- the task is mainly about arXiv preprints; use
arxiv
- the user only wants broad web discovery without ingesting; use external web search first, then come back if ingestion is desired
Core Workflow
1. Confirm the source shape
- If the user already provides URL(s), go straight to
scholaraio ingest-link
- If the user only has a topic but no URL, discover URLs first through the host agent's native web search, then ingest selected results
2. Use the CLI entrypoint
Basic ingestion:
scholaraio ingest-link https://example.com/page
scholaraio ingest-link https://example.com/page https://example.com/report.pdf
Preview only:
scholaraio ingest-link https://example.com/page --dry-run
Ingest without rebuilding indexes immediately:
scholaraio ingest-link https://example.com/page --no-index
Force PDF extraction mode when the backend needs a hint:
scholaraio ingest-link https://example.com/report.pdf --pdf
3. Keep the backend model clear
- ScholarAIO does not render webpages itself in this flow
- It depends on an external
qt-web-extractor service
- The value of that service is rendered-content extraction, not just raw HTML download
- Default HTTP endpoint:
http://127.0.0.1:8766
- Preferred MCP endpoint:
http://127.0.0.1:8766/mcp
- Prefer
config.yaml -> webextract.transport: mcp plus webextract.mcp_url / webextract.mcp_tool
- HTTP fallback uses
webextract.base_url / webextract.api_key
- Environment variables
WEBEXTRACT_TRANSPORT, WEBEXTRACT_MCP_URL, WEBEXTRACT_URL, and WEBEXTRACT_API_KEY still work as overrides
4. Understand what gets stored
- The extracted page is written into a temporary document inbox
- ScholarAIO reuses the existing document ingest flow
- Final records stay in the current
document family, not a separate webdocument type
- Provenance fields such as
source_url, source_type, and extraction_method are preserved in meta.json
- If the extractor returns rendered HTML, ScholarAIO discovers image references, downloads reachable images into a local
images/ directory during ingest, and rewrites Markdown links to local relative paths.
- If an image download fails because of access control, SSL, 404, or similar issues, ScholarAIO keeps the original external URL in the Markdown instead of breaking the content.
Practical Heuristics
- Prefer
--no-index when ingesting many links and you plan to rebuild once at the end
- Let the backend auto-detect normal web pages and PDF URLs first; prefer
--pdf only when detection seems unreliable
- If the backend is unavailable, report that clearly instead of pretending ScholarAIO can fetch/render the page alone
Output Style
- Make it clear that the content came from a URL rather than a local file
- Mention the source URL in summaries when it matters
- If ingestion fails, surface the backend/service reason directly
