web-scraping-rag

name	web-scraping-rag
description	Use this skill when you need to scrape a website and generate a structured RAG-optimized knowledge base document for a chatbot. Triggers include: scraping a company website, building a knowledge base, generating a knowledge_base.md, creating RAG documents, or any task involving web scraping followed by document generation for chatbot ingestion.

SKILL: Web Scraping & RAG-Optimized Knowledge Base Generation

Purpose: Scrape a target website, extract clean structured content, and produce a single knowledge_base.md optimized for vector embedding and RAG retrieval. Every decision — chunk size, overlap, metadata, deduplication — is made to maximize retrieval precision and minimize hallucination in downstream chatbots.

---

1. Environment Setup

Install All Dependencies

pip install requests trafilatura beautifulsoup4 lxml markdownify \
            langchain-text-splitters tiktoken python-dotenv \
            urllib3 fake-useragent tqdm hashlib

Optional — for JavaScript-heavy / SPA sites:

pip install playwright
playwright install chromium

Library Roles (Know What You're Using)

Library	Role
trafilatura	Primary HTML → clean text extractor. Strips nav, ads, footers automatically.
beautifulsoup4 + lxml	Fallback parser when trafilatura returns empty/partial content.
markdownify	Converts cleaned HTML into structured Markdown (preserves headings, lists, tables).
playwright	Full browser render for JS-heavy / React / Next.js sites.
langchain-text-splitters	RecursiveCharacterTextSplitter with configurable separators and overlap.
tiktoken	Accurate token counting using cl100k_base (GPT-4/Claude compatible).
fake-useragent	Rotates realistic browser User-Agent headers to avoid bot detection.
tqdm	Progress bar for multi-page scrapes.
hashlib	SHA-256 deduplication of chunk content.

---

2. Pre-Scrape Planning

Before writing a single line of scraping code, answer these questions:

1. Is the site static or JS-rendered?

   • Check: Disable JS in browser DevTools. If content disappears → use Playwright.
   • Static HTML → requests + trafilatura is sufficient.

2. What pages contain the real knowledge?

   • Docs pages, FAQs, blog articles, product pages, support articles = high value
   • Landing pages, pricing tables, legal/footer pages = low value (usually skip)

3. Does the site have a sitemap?

   • Check https://[domain]/sitemap.xml or https://[domain]/robots.txt
   • If yes → parse the sitemap to get a clean URL list instead of crawling.

4. What is the domain's robots.txt?

   • Always fetch and parse it. Never scrape Disallow: paths.
   • Add a 1.5–3s random delay between requests. Never hammer a server.

---

3. Fetching & Extraction

3.1 Primary: trafilatura (Preferred)

import trafilatura
import time
import random
from fake_useragent import UserAgent

ua = UserAgent()

def fetch_and_extract(url: str) -> str | None:
    headers = {"User-Agent": ua.random}
    downloaded = trafilatura.fetch_url(url, headers=headers)
    if not downloaded:
        return None
    text = trafilatura.extract(
        downloaded,
        include_comments=False,
        include_tables=True,
        no_fallback=False,         # allow internal fallback
        favor_precision=True,      # reduce noise over completeness
        output_format="markdown",  # directly get markdown
    )
    time.sleep(random.uniform(1.5, 3.0))
    return text

Key trafilatura.extract() flags to understand:

• favor_precision=True → aggressive boilerplate removal. Use for docs/articles.
• favor_recall=True → keep more content. Use if precision is cutting real content.
• include_tables=True → important for comparison pages and specs.
• output_format="markdown" → skips a markdownify conversion step.

3.2 Fallback: BeautifulSoup + markdownify

Use when trafilatura returns None or < 200 characters.

import requests
from bs4 import BeautifulSoup
from markdownify import markdownify as md

def fetch_bs4_fallback(url: str) -> str | None:
    headers = {"User-Agent": ua.random}
    try:
        res = requests.get(url, headers=headers, timeout=10)
        res.raise_for_status()
    except Exception as e:
        print(f"[FETCH ERROR] {url}: {e}")
        return None

    soup = BeautifulSoup(res.text, "lxml")

    # Remove noise elements before conversion
    for tag in soup(["nav", "header", "footer", "aside", "script",
                     "style", "noscript", "form", ".cookie-banner",
                     "#cookie-notice", ".advertisement"]):
        tag.decompose()

    # Target the main content area — adjust selector per site
    main = soup.find("main") or soup.find("article") or soup.find("body")
    if not main:
        return None

    raw_md = md(str(main), heading_style="ATX", bullets="-", strip=["a"])
    time.sleep(random.uniform(1.5, 3.0))
    return raw_md

3.3 Playwright (JS-Rendered / SPA Sites)

Use for sites built with React, Next.js, Vue, Angular, or any site where requests returns a shell HTML with no real content.

from playwright.sync_api import sync_playwright
import trafilatura

def fetch_playwright(url: str) -> str | None:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(
            user_agent=ua.random,
            extra_http_headers={"Accept-Language": "en-US,en;q=0.9"}
        )
        page.goto(url, wait_until="networkidle", timeout=30000)
        page.wait_for_timeout(2000)  # Let lazy-loaded content settle
        html = page.content()
        browser.close()

    text = trafilatura.extract(html, output_format="markdown",
                               include_tables=True, favor_precision=True)
    return text

---

4. Multi-Page Crawling

4.1 Sitemap-Based (Preferred — Most Accurate)

import xml.etree.ElementTree as ET

def get_urls_from_sitemap(sitemap_url: str) -> list[str]:
    res = requests.get(sitemap_url, timeout=10)
    root = ET.fromstring(res.content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    urls = [loc.text for loc in root.findall(".//sm:loc", ns)]
    return urls

4.2 Link-Based Crawl (Fallback — Same Domain Only)

from urllib.parse import urljoin, urlparse

def crawl(start_url: str, max_pages: int = 100) -> list[str]:
    domain = urlparse(start_url).netloc
    visited, queue = set(), [start_url]
    results = []

    while queue and len(results) < max_pages:
        url = queue.pop(0)
        if url in visited:
            continue
        visited.add(url)

        try:
            res = requests.get(url, headers={"User-Agent": ua.random}, timeout=10)
            soup = BeautifulSoup(res.text, "lxml")
            results.append(url)

            for a in soup.find_all("a", href=True):
                abs_url = urljoin(url, a["href"])
                # Stay on same domain, skip anchors/query-heavy URLs
                if urlparse(abs_url).netloc == domain and \
                   abs_url not in visited and "#" not in abs_url:
                    queue.append(abs_url)
        except Exception:
            continue
        time.sleep(random.uniform(1.5, 2.5))

    return results

Filtering rules (apply before crawling):

• Skip URLs containing: /login, /signup, /cart, /checkout, /cdn-cgi, .pdf, .png, .jpg, .zip
• Skip URLs with query strings longer than 2 params (usually pagination/search results)
• Prefer URLs containing: /docs, /help, /guide, /faq, /blog, /about

---

5. Text Cleaning & Normalization

Apply after extraction, before chunking. This is where quality is made or lost.

import re

def clean_markdown(text: str) -> str:
    if not text:
        return ""

    # Normalize Windows line endings
    text = text.replace("\r\n", "\n").replace("\r", "\n")

    # Remove HTML entities that slip through
    text = re.sub(r"&[a-zA-Z]+;", " ", text)

    # Remove inline HTML remnants
    text = re.sub(r"<[^>]+>", "", text)

    # Collapse excessive blank lines (max 2 consecutive)
    text = re.sub(r"\n{3,}", "\n\n", text)

    # Remove lines that are pure noise (single chars, dividers, etc.)
    lines = text.split("\n")
    lines = [l for l in lines if not re.fullmatch(r"[-_=*|\\/ ]{3,}", l.strip())]
    lines = [l for l in lines if len(l.strip()) > 1 or l.strip() == ""]

    # Remove cookie/GDPR banners that sneak through (common patterns)
    noise_patterns = [
        r"(?i)we use cookies.*?(\n|$)",
        r"(?i)accept all cookies.*?(\n|$)",
        r"(?i)subscribe to our newsletter.*?(\n|$)",
        r"(?i)sign up for.*?updates.*?(\n|$)",
    ]
    text = "\n".join(lines)
    for pattern in noise_patterns:
        text = re.sub(pattern, "", text)

    # Collapse again after noise removal
    text = re.sub(r"\n{3,}", "\n\n", text)

    return text.strip()

What to PRESERVE (never strip these):

• Heading hierarchy: #, ##, ### — critical for section path metadata
• Numbered steps: 1., 2. — procedural content must stay sequential
• Code blocks: ```python ``` — do NOT split inside code blocks
• Tables: |---|---| — keep complete
• Q&A pairs: question + answer must stay in the same chunk

---

6. RAG-Optimized Chunking

6.1 Token Budget Logic

Chunk Size	Effect on RAG
< 200 tokens	Too small — loses context, retrieval becomes noisy
300–500 tokens	Good for FAQ-style, definition-heavy content
500–700 tokens	Sweet spot — preserves reasoning chains, fast retrieval
700–900 tokens	Acceptable for complex technical docs with code examples
> 900 tokens	Too large — dilutes embedding signal, retrieval degrades

Target: 90% of chunks between 350–750 tokens.

6.2 Chunking Code

from langchain_text_splitters import RecursiveCharacterTextSplitter
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

def chunk_document(text: str, source_url: str) -> list[dict]:
    splitter = RecursiveCharacterTextSplitter(
        separators=[
            "\n## ",    # Major sections first
            "\n### ",   # Sub-sections
            "\n#### ",
            "\n\n",     # Paragraph breaks
            "\n",       # Line breaks
            ". ",       # Sentence boundaries
            " ",        # Word boundaries (last resort)
            ""
        ],
        chunk_size=650,         # tokens
        chunk_overlap=80,       # tokens overlap for context continuity
        length_function=count_tokens,
        is_separator_regex=False,
    )

    chunks_raw = splitter.split_text(text)
    chunks = []

    for chunk in chunks_raw:
        token_count = count_tokens(chunk)

        # Drop fragments that are too small to be meaningful
        if token_count < 250:
            continue

        # Flag oversized chunks (will be caught in validation)
        if token_count > 900:
            # Try a hard split at the nearest paragraph
            sub_chunks = chunk.split("\n\n")
            for sc in sub_chunks:
                sc_tokens = count_tokens(sc)
                if sc_tokens >= 250:
                    chunks.append({
                        "content": sc.strip(),
                        "tokens": sc_tokens,
                        "source": source_url,
                    })
            continue

        chunks.append({
            "content": chunk.strip(),
            "tokens": token_count,
            "source": source_url,
        })

    return chunks

6.3 Section Path Injection

Every chunk must carry the heading context it came from. This prevents the "orphan chunk" problem where retrieved content has no context about what section it belongs to.

def inject_section_paths(text: str) -> str:
    """
    Prepends the nearest parent heading to each paragraph/block.
    Converts:
        ## Installation
        Run this command...
    Into:
        [Installation] Run this command...
    """
    lines = text.split("\n")
    current_path = []
    output = []

    for line in lines:
        if line.startswith("### "):
            heading = line.lstrip("# ").strip()
            if len(current_path) >= 2:
                current_path[1] = heading
            else:
                current_path.append(heading)
        elif line.startswith("## "):
            heading = line.lstrip("# ").strip()
            current_path = [heading]
        elif line.startswith("# "):
            heading = line.lstrip("# ").strip()
            current_path = [heading]
        elif line.strip() and current_path:
            path_str = " > ".join(current_path)
            output.append(f"[{path_str}] {line}")
            continue

        output.append(line)

    return "\n".join(output)

---

7. Deduplication

Two types of duplicates destroy RAG quality: exact duplicates (copied boilerplate) and near-duplicates (same content with slightly different wording from pagination).

import hashlib

def deduplicate_chunks(chunks: list[dict]) -> list[dict]:
    seen_hashes = set()
    unique = []

    for chunk in chunks:
        # Normalize before hashing: lowercase, collapse whitespace
        normalized = re.sub(r"\s+", " ", chunk["content"].lower()).strip()
        h = hashlib.sha256(normalized.encode("utf-8")).hexdigest()

        if h not in seen_hashes:
            seen_hashes.add(h)
            chunk["hash"] = h[:8]  # Store short hash as chunk ID
            unique.append(chunk)

    print(f"[DEDUP] {len(chunks)} → {len(unique)} chunks after deduplication")
    return unique

Near-duplicate detection (optional but recommended for large sites):

# Simple Jaccard similarity for near-dup detection
def jaccard_similarity(a: str, b: str) -> float:
    set_a = set(a.lower().split())
    set_b = set(b.lower().split())
    intersection = len(set_a & set_b)
    union = len(set_a | set_b)
    return intersection / union if union else 0.0

def remove_near_duplicates(chunks: list[dict], threshold: float = 0.85) -> list[dict]:
    filtered = []
    for i, chunk in enumerate(chunks):
        is_dup = False
        for prev in filtered[-10:]:  # Only compare with recent 10 (performance)
            if jaccard_similarity(chunk["content"], prev["content"]) > threshold:
                is_dup = True
                break
        if not is_dup:
            filtered.append(chunk)
    return filtered

---

8. Metadata Schema

Each internal chunk (used during processing, not in final output) carries:

chunk = {
    "hash":    "a3f1b8c2",              # SHA-256 first 8 chars
    "source":  "https://example.com/about",
    "path":    "About > Our Mission",
    "tokens":  512,
    "content": "..."
}

The final knowledge_base.md does not expose raw chunk metadata to the chatbot. Instead, chunks are synthesized into a structured human-readable document (see Section 9). Metadata is retained in a separate chunks_index.json for debugging and re-ingestion.

---

9. Output Format — Structured Company Knowledge Base

Critical design decision: The final knowledge_base.md is NOT a dump of raw scraped chunks. It is a synthesized, structured document that organizes all scraped content into a predictable skeleton. This makes retrieval dramatically more accurate because section boundaries align with question intent (e.g. "what services do they offer" → hits ## Services, not a random chunk mid-page).

9.1 Required Document Skeleton

The output document MUST follow this structure. Every section is mandatory. If the site has no content for a section, write _No information available on the website._ — never omit the heading.

# [Company Name] - Company Knowledge Base

> Source: [website URL]
> Generated: [ISO date]
> Sections: Company Overview · Services · Process · Team · Clients · FAQs · Contact

---

## Company Overview

A 3–5 sentence summary covering: what the company does, where it operates,
who it serves, and its core value proposition. Written in third person.
Do not use bullet points here — prose only.

**Founded:** [year if available]
**Headquarters:** [City, State/Country]
**Industry:** [primary industry]
**Company Type:** [e.g. Consulting, SaaS, Agency, Staffing]

---

## Services

> Each service gets its own ### subsection. Never merge multiple services
> into a single block. This is the most-queried section of any company KB.

### [Service Name 1]

**What it is:** 1–2 sentence plain-English description.

**What's included:**
- [Specific deliverable or capability]
- [Specific deliverable or capability]
- [Specific deliverable or capability]

**Who it's for:** [Target customer segment]
**Key differentiator:** [What makes this service distinct, if mentioned on site]

### [Service Name 2]
[Same structure as above]

### [Service Name N]
[Same structure as above]

---

## How It Works / Our Process

> Include this section ONLY if the site describes a process, methodology,
> or step-by-step workflow. Skip with a note if not available.

### Step 1: [Step Name]
[Description of what happens in this step]

### Step 2: [Step Name]
[Description]

[Continue for all steps found on site]

---

## Industries Served

List each industry the company explicitly mentions serving.
For each, include a brief note on what they offer that industry if stated.

- **[Industry 1]:** [context if available]
- **[Industry 2]:** [context if available]

---

## Team & Leadership

> Include only what is publicly stated on the website.
> Never infer or fabricate titles/names.

| Name | Role | Notable Background |
|------|------|--------------------|
| [Name] | [Title] | [Bio snippet if available] |

If no team info is available: _Team information not publicly listed on the website._

---

## Clients & Partners

List all clients, partners, or case studies mentioned on the site.

**Clients:**
- [Client Name] — [Industry or context if mentioned]

**Partners / Integrations:**
- [Partner Name] — [Nature of partnership if stated]

If a case study is present, summarize it in 2–3 sentences under a `#### Case Study:` subheading.

---

## Pricing & Engagement Models

> Include ONLY if the website mentions pricing, plans, or engagement types.
> Never guess or fill in pricing that isn't stated.

[Describe pricing structure, packages, or how to get a quote as stated on site]

If not available: _Pricing is not listed publicly. Contact the company directly for a quote._

---

## FAQs

> Extract every Q&A from the site. If no dedicated FAQ page exists,
> synthesize implicit FAQs from the content (e.g. "What industries do you serve?"
> from an Industries page). Aim for minimum 8 Q&A pairs.

**Q: What does [Company] do?**
A: [Answer synthesized from overview content]

**Q: What services does [Company] offer?**
A: [Answer listing all services]

**Q: What industries does [Company] serve?**
A: [Answer from industries section]

**Q: How do I get started / contact [Company]?**
A: [Answer from contact info]

**Q: Where is [Company] located?**
A: [Address or city/region]

**Q: [Any FAQ explicitly on the site]**
A: [Verbatim or paraphrased answer]

[Add more Q&As as found — do not cap at a fixed number]

---

## Contact Information

**Email:** [email address]
**Phone:** [phone number with country code]
**Address:** [full address if listed]
**Website:** [URL]

**Office Hours:** [if stated]
**Response Time:** [if stated, e.g. "within 24 hours"]

### Social Media
- LinkedIn: [URL]
- Twitter/X: [URL]
- [Other platforms if listed]

### Contact Form
[Note if a contact form exists and what it captures, e.g. "Name, Email, Message, Service Interest"]

---

## Additional Notes

> Use this section for anything important on the site that doesn't fit
> the sections above. Examples: certifications, awards, news mentions,
> blog summaries, job openings, legal/compliance info, return policy, etc.

### [Subsection title]
[Content]

---

9.2 Writing Rules for the Structured Document

These rules apply when synthesizing scraped content into the skeleton above:

Accuracy rules:

• Never invent or infer information not present on the site
• If a data point is ambiguous, quote the site's exact wording in quotes
• Mark anything uncertain with [unconfirmed — verify on site]

Prose rules:

• Company Overview → prose paragraphs only, no bullets
• Services → always use the sub-structure (What it is / What's included / Who it's for)
• FAQs → always full Q: / A: format, never collapsed
• Do not editorialize. No phrases like "impressive", "industry-leading", "best-in-class" unless quoted from the site

Completeness rules:

• Every ### service gets its own block — never merge two services
• Every FAQ found on the site must appear verbatim + additional synthesized ones
• Contact section must be 100% accurate — phone/email errors break chatbot utility

Length targets per section:

Section	Min Length	Max Length
Company Overview	150 words	350 words
Each Service	80 words	250 words
FAQs	8 Q&A pairs	No cap
Contact Info	All fields found	—
Total document	1,500 words	6,000 words

---

9.3 Export Code

from datetime import date

def export_structured_kb(
    structured_content: str,
    chunks: list[dict],
    company_name: str,
    source_url: str,
    output_md: str = "knowledge_base.md",
    output_index: str = "chunks_index.json"
):
    import json

    today = date.today().isoformat()

    # Write the human-readable structured document
    with open(output_md, "w", encoding="utf-8") as f:
        f.write(structured_content)
    print(f"[EXPORT] Structured KB → {output_md}")

    # Write the raw chunks index separately (for vector store ingestion)
    index = {
        "company": company_name,
        "source": source_url,
        "generated": today,
        "total_chunks": len(chunks),
        "token_stats": {
            "min": min(c["tokens"] for c in chunks),
            "max": max(c["tokens"] for c in chunks),
            "avg": round(sum(c["tokens"] for c in chunks) / len(chunks))
        },
        "chunks": chunks
    }
    with open(output_index, "w", encoding="utf-8") as f:
        json.dump(index, f, indent=2, ensure_ascii=False)
    print(f"[EXPORT] Chunks index → {output_index}")

Two files are always produced:

• knowledge_base.md — the structured document. Feed this to your chatbot as the primary context source.
• chunks_index.json — raw chunks with metadata. Use this for vector embedding and semantic search.

---

10. Validation Checklist

Run these checks before feeding the file into your vector store:

def validate_knowledge_base(chunks: list[dict]) -> bool:
    issues = []

    total = len(chunks)
    in_range = [c for c in chunks if 350 <= c["tokens"] <= 750]
    coverage = len(in_range) / total * 100

    if coverage < 90:
        issues.append(f"Only {coverage:.1f}% of chunks in 350–750 token range (need 90%)")

    if total < 10:
        issues.append(f"Only {total} chunks — likely scraping failure or very thin content")

    if total > 5000:
        issues.append(f"{total} chunks is very high — check for crawl duplication or noise")

    for c in chunks:
        if not c.get("source"):
            issues.append(f"Chunk {c.get('hash')} missing source URL")
        if not c["content"].strip():
            issues.append(f"Chunk {c.get('hash')} has empty content")

    if issues:
        print("[VALIDATION FAILED]")
        for issue in issues:
            print(f"  ✗ {issue}")
        return False

    print(f"[VALIDATION PASSED] {total} chunks, {coverage:.1f}% in optimal range")
    return True

Manual checklist before using the knowledge base:

• Every chunk is readable and self-contained out of context
• No chunk contains only navigation text, button labels, or UI strings
• Code blocks are complete (not cut mid-function)
• Tables are not split across chunks
• Q&A pairs are not split (question and answer in same chunk)
• 90%+ of chunks fall in the 350–750 token window
• All chunks have Source: and Path: metadata
• File opens cleanly without encoding errors

---

11. Complete Orchestration Script

The pipeline runs in two passes:

• Pass 1 (Scrape): Crawl all pages → extract → clean → chunk → deduplicate → save chunks_index.json
• Pass 2 (Synthesize): Feed all cleaned text into GLM 5.1 with the document skeleton → produce knowledge_base.md

# run_scrape.py
from tqdm import tqdm
import json

TARGET_URL  = "https://your-target-site.com"   # ← Replace
COMPANY_NAME = "Company Name"                   # ← Replace
SITEMAP_URL  = f"{TARGET_URL}/sitemap.xml"
MAX_PAGES    = 150

# ── PASS 1: SCRAPE ──────────────────────────────────────────────

print("[1/6] Discovering URLs...")
try:
    urls = get_urls_from_sitemap(SITEMAP_URL)
    print(f"  Found {len(urls)} URLs from sitemap")
except Exception:
    print("  Sitemap not found — falling back to crawl")
    urls = crawl(TARGET_URL, max_pages=MAX_PAGES)

skip_patterns = ["/login", "/signup", "/cart", "/cdn-cgi",
                 ".pdf", ".png", ".jpg", ".zip", ".css", ".js"]
urls = [u for u in urls if not any(p in u for p in skip_patterns)]
print(f"  {len(urls)} URLs after filtering")

print("[2/6] Scraping & cleaning pages...")
all_chunks = []
all_cleaned_text = []  # Accumulated for Pass 2 synthesis

for url in tqdm(urls[:MAX_PAGES]):
    raw_text = fetch_and_extract(url)
    if not raw_text or len(raw_text) < 200:
        raw_text = fetch_bs4_fallback(url)
    if not raw_text or len(raw_text) < 200:
        continue

    clean_text = clean_markdown(raw_text)
    annotated_text = inject_section_paths(clean_text)
    all_cleaned_text.append(f"<!-- SOURCE: {url} -->\n{clean_text}")

    chunks = chunk_document(annotated_text, url)
    all_chunks.extend(chunks)

print("[3/6] Deduplicating...")
all_chunks = deduplicate_chunks(all_chunks)
all_chunks = remove_near_duplicates(all_chunks)

print("[4/6] Validating...")
validate_knowledge_base(all_chunks)

# Save raw chunks index
with open("chunks_index.json", "w", encoding="utf-8") as f:
    json.dump(all_chunks, f, indent=2, ensure_ascii=False)
print("  Saved chunks_index.json")

# ── PASS 2: SYNTHESIZE ──────────────────────────────────────────

print("[5/6] Synthesizing structured knowledge base...")
full_scraped_text = "\n\n---\n\n".join(all_cleaned_text)

# This text gets passed to GLM 5.1 (see Section 12 for the exact prompt)
# GLM writes the structured knowledge_base.md using the skeleton in Section 9.1
# Save the raw material for the LLM call:
with open("scraped_raw.txt", "w", encoding="utf-8") as f:
    f.write(full_scraped_text)
print("  Saved scraped_raw.txt — pass to GLM 5.1 for synthesis")

print("[6/6] Done.")
print("  Next: run GLM synthesis prompt (see SKILL.md Section 12)")
print("  Output files: knowledge_base.md, chunks_index.json")

---

12. GLM 5.1 Prompting via OpenCode

12.1 Initial Scrape Prompt

Use this to kick off the full pipeline:

Apply SKILL.md to scrape [TARGET_URL].

Company name: [Company Name]
Site type: [static HTML / WordPress / Next.js SPA / docs site]
Priority pages: [list key paths like /services, /about, /faq, /contact]
Skip: [/blog, /careers, or any sections irrelevant to the chatbot]

Run both passes:
1. Scrape → clean → chunk → save chunks_index.json
2. Synthesize → write knowledge_base.md using the Section 9.1 skeleton

Report: chunk count, token range, pages that returned empty content, 
and any sections of the skeleton that had no source data.

12.2 Synthesis-Only Prompt (Pass 2, after scrape)

Use this if you already have scraped_raw.txt and just need the document:

You are writing a structured company knowledge base for a customer-facing chatbot.

Below is the raw scraped content from [Company Name]'s website ([URL]).
Using ONLY the information present in the scraped content:

1. Fill in the knowledge base skeleton from SKILL.md Section 9.1 exactly.
2. Follow all writing rules from Section 9.2 strictly.
3. If a section has no data, write: _No information available on the website._
4. Never invent, infer, or embellish. Quote exact wording for anything uncertain.
5. FAQs: include ALL Q&As from the site + synthesize at least 8 from the content.

Output the complete knowledge_base.md. Nothing else — no preamble, no summary.

--- SCRAPED CONTENT START ---
[paste contents of scraped_raw.txt here]
--- SCRAPED CONTENT END ---

12.3 Audit & Fix Prompt (after document is generated)

Review the knowledge_base.md I just generated against SKILL.md Section 9.2 rules.

Check for:
- Any section that is missing entirely
- Any service that is missing a "What it is / What's included / Who it's for" structure  
- FAQ count below 8
- Any invented information not found in the scraped content
- Contact section completeness (email, phone, address, hours)
- Any section exceeding its max word count

List each issue with the exact section name and what needs to be fixed.
Then apply all fixes and output the corrected document.

---

13. Common Failure Modes & Fixes

Problem	Symptom	Fix
JS-rendered site	trafilatura returns < 100 chars	Switch to Playwright fetch
Anti-bot blocking	403 / 429 responses	Add random delay 2–5s, rotate User-Agent, retry with exponential backoff
Over-chunking	500+ tiny chunks from one page	Increase chunk_size to 750, raise minimum to 300 tokens
Under-chunking	3 massive chunks per page	Reduce chunk_size to 500, add \n\n as a hard separator
Boilerplate leak	Chunks full of nav/footer text	Tighten BS4 selector to <main> or <article>, add more noise patterns to clean_markdown
Duplicate content	Same paragraph in 20 chunks	Lower Jaccard threshold from 0.85 to 0.75
Missing context in retrieval	Retrieved chunks are confusing alone	Verify inject_section_paths is running before chunking
Encoding errors	\x84, \xb4 in output	Force encoding="utf-8" on all open() calls, add errors="replace"
Empty knowledge base	0 chunks output	Check if site requires authentication; try Playwright; check robots.txt

---

14. Usage Instructions

Quick Start

1. Save this file as SKILL.md in your project root.
2. Set TARGET_URL and COMPANY_NAME in run_scrape.py.
3. Run: python run_scrape.py → produces scraped_raw.txt + chunks_index.json
4. Run the GLM 5.1 synthesis prompt (Section 12.2) with scraped_raw.txt → produces knowledge_base.md
5. Optionally run the audit prompt (Section 12.3) to catch and fix gaps.

What You Get

File	Purpose
knowledge_base.md	The structured document — feed this to your chatbot as primary context
chunks_index.json	Raw chunks with metadata — use for vector embedding + semantic search
scraped_raw.txt	Intermediate raw text — keep for re-synthesis without re-scraping

Feeding Into Your Chatbot

• Simple approach: Load knowledge_base.md as a system prompt or context document. Works for small companies where the full doc fits in context.
• RAG approach: Embed chunks from chunks_index.json, store metadata as payload, retrieve top-K chunks by cosine similarity, inject into LLM context per query.
• Hybrid (recommended): Use the structured knowledge_base.md sections as semantic anchors — embed by section, not by raw chunk. Section headers become natural query targets.