vs-crawler

name	vs-crawler
description	Crawl websites (news, blogs, papers, GitHub, product docs, RSS feeds) into a fixed-schema JSONL file, then create a dataset and a searchable application in Viking AI Search. Supports one-time crawl and scheduled recurring crawl with automatic incremental sync.
category	workflow
applies_to	codex, agents, external-agent
requires_cli	>=0.2.0
keywords	web crawler, scheduled crawl, content ingestion, news crawler, blog crawler, paper crawler, github crawler, docs crawler, rss crawler
commands	connector export, connector init, connector run, connector status, connector stop, dataset import-url, dataset infer-schema, dataset infer-result, dataset create, data write, app create, app attach-dataset

Viking Content Crawler

When to Use

Use this skill when the user wants to crawl content from websites and import it into Viking AI Search to build a searchable knowledge base. This covers news sites, blogs, academic papers, GitHub repositories, product documentation, RSS feeds, and similar web content sources.

The agent writes crawler code tailored to the target sites, outputs data in a fixed JSONL schema, and then hands off to the vs-item-onboarding skill for dataset creation and import.

Do not use this skill when:

• The user already has a local file ready to import (use vs-item-onboarding directly).
• The user wants to import from a database (use vs-item-onboarding directly with MySQL).

Fixed Schema

All crawled records MUST conform to this schema. Every record is a flat JSON object written as one line in a JSONL file.

Field	Type	Required	Description
id	string	yes	Unique identifier. Use a source-native stable ID (e.g., arXiv ID, GitHub owner/repo, post slug) when available; otherwise derive a deterministic ID from title + author + published_at. Must be deterministic so re-crawling the same item produces the same ID.
title	string	yes	Content title (headline, post title, paper title, repo name, doc page title).
summary	string	yes	Short abstract or description (100-500 characters recommended).
content	string	yes	Full text body with HTML stripped to plain text. For GitHub repos, concatenate README content. For PDF/DOC documents, extract the text content directly into this field.
category	string	yes	One of: news, blog, paper, github, docs, other.
source	string	yes	Human-readable source name, e.g. "Hacker News", "arXiv", "Viking Docs".
author	string	no	Author name(s); multiple authors separated by commas.
published_at	string	no	ISO 8601 datetime, e.g. "2026-07-16T10:30:00Z". Use crawl time if unavailable.
tags	array<string>	no	Tags, keywords, or topics.
language	string	no	ISO 639-1 code: "en", "zh", etc.
source_url	string	no	Canonical URL of the source page (the URL the record was crawled from). Must be a fully-qualified URL with scheme and host.

Example Record

{
  "id": "viking-blog-introducing-viking-ai-search",
  "title": "Introducing Viking AI Search",
  "summary": "Viking AI Search is a new generation of hybrid search engine combining BM25 and vector search...",
  "content": "Full article text with HTML removed and paragraphs separated by newlines...",
  "category": "blog",
  "source": "Viking Blog",
  "author": "Jane Doe",
  "published_at": "2026-07-15T08:00:00Z",
  "tags": ["search", "vector database", "hybrid search"],
  "language": "en",
  "source_url": "https://viking.example.com/blog/introducing-viking-ai-search",
  "metadata": {
     
     
     
  

Standard Metadata Fields

Only these keys are allowed in metadata. Do not add custom keys — every crawled record, regardless of source or category, must use exactly these keys when the data is available, and omit keys whose data is unavailable. This guarantees schema consistency across all crawl sources so downstream consumers (schema inference, search relevance tuning) see a uniform shape.

Key	Type	Category	Description
read_time	string	content	Estimated reading time, e.g. "8 min".
word_count	number	content	Word count of the article / document body.
views	number	engagement	View count or page view count.
likes	number	engagement	Like / upvote / thumbs-up count.
comments	number	engagement	Comment count.
shares	number	engagement	Share count.
stars	number	repo / paper	GitHub stars (for github category) or citation-equivalent metric.
forks	number	repo	GitHub fork count (for github category).
citations	number	paper	Citation count (for paper category).
venue	string	paper	Publication venue, e.g. "NeurIPS 2025", "arXiv".
doi	string	paper	Digital Object Identifier, e.g. "10.1234/abcde".

Values must be flat scalars (string / number / boolean). No nested objects, no arrays. If a data point does not map to any standard key, omit it rather than inventing a new key.

Preconditions

• vs CLI >= 0.2.0 is installed and authenticated (vs auth status and vs doctor succeed).
• The crawl target is reachable from the execution environment.
• A suitable runtime is available (Python 3.8+ with requests and beautifulsoup4 recommended).

Commands

This skill delegates dataset creation and import to vs-item-onboarding. The crawler workflow itself uses:

Stage	Action	Purpose
Crawl	Run agent-written crawler script	Fetch content and write JSONL
Onboard	Invoke vs-item-onboarding skill	Create dataset, infer schema, import data, optionally start sync
Schedule	Set up cron/launchd wrapper	For scheduled mode: periodically re-crawl and append new lines

Workflow

Run in strict order.

1. Confirm crawl mode — resolve whether the user wants one-time crawl or scheduled recurring crawl. Only skip the question when the request contains an explicit, unambiguous signal (apply detection to whatever language the user is writing in):

   • Explicit one-time: phrases carrying "once", "one-time", "just this time", or equivalent single-crawl semantics.
   • Explicit scheduled: phrases carrying "daily", "scheduled", "keep updated", "auto-crawl", "sync", "incremental", or equivalent recurring semantics.
   • If the request is neutral — e.g. "crawl X", bare "crawl", mentions target sites but says nothing about scheduling/once — you MUST ask the user to choose. The bare crawl verb is NOT a one-time signal; it is ambiguous. Never silently default to one-time.

2. Identify crawl targets and write the crawler. Based on the user's target sites, write a crawler script. The crawler MUST:

   • Output records conforming to the Fixed Schema as JSONL (one record per line).
   • Write output to a stable path: /tmp/viking/crawler/<job-name>/items.jsonl.
   • For scheduled mode: support incremental crawling — track the last crawl cursor (most recent published_at or last seen item IDs) in /tmp/viking/crawler/<job-name>/state.json so subsequent runs only fetch new content.
   • Deduplicate by id within each run and against previous state.
   • Strip HTML to plain text; never include raw HTML in content.
   • When encountering PDF, DOC, or other document links, download the document and extract its text content directly into the content field. Use available libraries (e.g. PyPDF2/pypdf for PDF, python-docx for DOCX, beautifulsoup4 for HTML) to extract readable text. Do not store document links in records; put the extracted full text in content.
   • Be polite: set a descriptive User-Agent, respect robots.txt, add 1-3 second delays between requests, retry transient errors with backoff.
   • Prefer structured sources (RSS/Atom feeds > sitemap.xml > official APIs > HTML scraping).
   • Log per-item errors and continue; do not abort on single-page failures.
   • Strictly follow the Fixed Schema defined above — the same field names, types, and metadata key set, regardless of the source. Do not add source-specific top-level fields or metadata keys. Print a summary to stdout: crawled count, new count, output path.

Customer Environment Principle

• In customer environments, assume repository source code is unavailable.
• Execute tasks using only the installed skills, the packaged vs CLI surface (--help, command output, observed runtime behavior), and explicit user-provided information.
• If the installed CLI behavior conflicts with a skill, trust the installed CLI behavior first.

Constraints

1. Never write raw HTML into content. Always strip to plain text.
2. Never hardcode credentials in crawler code. Use environment variables for API keys.
3. Always generate a stable id. Use a source-native stable ID (e.g., arXiv ID, GitHub owner/repo, post slug) when available; otherwise derive a deterministic ID from title + author + published_at. Must be deterministic so re-crawling the same item produces the same ID.
4. All datetime values MUST be ISO 8601 (e.g., "2026-07-16T10:30:00Z").
5. All output MUST be valid JSONL: one JSON object per line, UTF-8 encoded.
6. The category field MUST use the predefined values (news, blog, paper, github, docs, other).
7. Dataset creation and import MUST go through vs-item-onboarding. Do not call vs dataset create, vs data write, etc. directly from this skill.
8. For scheduled mode, incremental sync is handled by the JSONL file connector (configured by vs-item-onboarding). The scheduled job only needs to run the crawler to append new lines to the JSONL file; the connector daemon picks up new lines automatically.
9. Respect rate limits and robots.txt. Add polite delays between requests.
10. Extract text from PDF/DOC documents. When encountering PDF, DOCX, or other document links, download the file and extract its text content directly into the content field using appropriate libraries (e.g., pypdf for PDF, python-docx for DOCX). Do not store document links in output records.
11. Auto-confirm Schema Confirmation during onboarding. The crawler produces records against the Fixed Schema defined above, which is stable and well-defined. When handing off to vs-item-onboarding, instruct it to automatically reply yes at the Schema Confirmation step without surfacing the prompt to the user. Only pause and surface schema details if the backend inference returns genuine errors (e.g. missing primary-key BizAttr) that require user intervention.
12. After completes its hand-off (printing console links + readiness reminder), end your turn. Do NOT run , , or any polling loop to wait for the Ready state. Readiness is an asynchronous backend process; tell the user to check the console links themselves.

Recovery Hints

• Crawler returns zero records → verify target site/feed accessibility, check for rate limiting (HTTP 429), review error logs.
• Duplicate records appear → verify id generation is deterministic (same item always produces the same ID).
• Content extraction produces garbled text → ensure HTTP response encoding is correctly detected.
• PDF text extraction fails or is garbled → try a different PDF library (e.g., switch from pypdf to pdfplumber) or fall back to extracting abstract/metadata only.
• Sync is not picking up new lines → verify the JSONL connector daemon is running via vs connector status --job <job>.