wayback

name	wayback
description	Query and explore the Wayback Archiver personal web archiving system

Wayback Archiver

Wayback Archiver is a self-hosted personal web archiving system that captures and preserves web pages with full fidelity — HTML, CSS, JavaScript, images, fonts, and all resources.

Repository: https://github.com/icodeface/wayback-archiver

When to use

Use this skill when the user wants to:

• Search archived web pages
• View snapshots of a specific URL
• List recent archives or filter by date range
• Get details about a specific archived page
• Read page content in Markdown format (for AI/LLM consumption)
• Explore the archiving system's data

Prerequisites

Before using this skill, ensure the Wayback Archiver server is running:

• Chrome or Firefox with Tampermonkey extension (v5.3+)
• Database (optional): SQLite (default, zero-config) or PostgreSQL 14+ (for remote deployment)

Quick Start

1. Download Pre-built Binaries

Download from Releases:

• Server binary: wayback-server-<os>-<arch>.tar.gz (or .zip for Windows)
• Userscript: wayback-userscript.js

Extract the server binary:

# macOS/Linux
tar -xzf wayback-server-*.tar.gz

# Windows: extract the .zip file

Building from source? See docs/BUILD.md.

2. Setup Database

By default, Wayback Archiver uses SQLite (zero-config, single-file database). For remote deployment with multiple users, PostgreSQL is recommended.

Option 1: SQLite (Default)

No setup required! The database file will be created automatically at ./data/wayback.db on first run.

Option 2: PostgreSQL

# Configure database connection in .env
echo "DB_TYPE=postgres" >> .env
echo "DB_NAME=wayback" >> .env

Server startup automatically creates the configured PostgreSQL database and tables. If the database does not exist yet, the configured user must have permission to create databases.

Upgrade note for existing installs: Any installation upgrading from < 1.1.0 to 1.1.0 or later needs the pages.snapshot_state column. The server applies this automatically during startup when the database user has sufficient privileges. Otherwise, run the following idempotent SQL manually first:

ALTER TABLE pages ADD COLUMN IF NOT EXISTS snapshot_state VARCHAR(16);
UPDATE pages SET snapshot_state = 'ready' WHERE snapshot_state IS NULL OR snapshot_state = '';
ALTER TABLE pages ALTER COLUMN snapshot_state SET DEFAULT 'pending';
ALTER TABLE pages ALTER COLUMN snapshot_state SET NOT NULL;

3. Start Server

./wayback-server

Server runs at http://localhost:8080 by default.

4. Install Browser Script

1. Download wayback-userscript.js from Releases
2. Open Tampermonkey dashboard
3. Create new script and paste the content
4. Save and enable

Chrome users: Enable "Allow user scripts" in Tampermonkey's extension settings (right-click icon → Manage extension). Firefox does not require this.

API Usage

Base URL: http://localhost:8080

Authentication (Optional)

When AUTH_PASSWORD is set, use HTTP Basic Auth:

• Username: wayback
• Password: $AUTH_PASSWORD

Endpoints

List All Pages

curl "http://localhost:8080/api/pages?limit=100&offset=0"

Search Pages

curl "http://localhost:8080/api/search?q=$KEYWORD"

Filter by Date Range

curl "http://localhost:8080/api/pages?from=2026-03-01&to=2026-03-12"

Get Page Details

curl "http://localhost:8080/api/pages/$PAGE_ID"

Get Page Content as Markdown

Returns the page body as clean Markdown (strips scripts, nav, footer, etc.). Ideal for AI/LLM consumption.

curl "http://localhost:8080/view/$PAGE_ID/md"

Create Public Share Link

Creates an unauthenticated link for one fixed snapshot. The returned token is only shown once. The /view/:id snapshot header also has a Share button that creates and copies the public URL, then exposes Revoke for that share.

curl -X POST "http://localhost:8080/api/pages/$PAGE_ID/shares" \
  -H "Content-Type: application/json" \
  -d '{}'

Get Timeline for URL

curl "http://localhost:8080/api/pages/timeline?url=$ENCODED_URL"

View Archived Page

Open in browser:

http://localhost:8080/view/$PAGE_ID

Data Models

Page

{
  id: number;
  url: string;
  title: string;
  captured_at: string;      // ISO 8601 timestamp
  html_path: string;
  content_hash: string;     // SHA-256
  first_visited: string;    // ISO 8601 timestamp
  last_visited: string;     // ISO 8601 timestamp
  created_at: string;       // ISO 8601 timestamp
}

Resource

{
  id: number;
  url: string;
  content_hash: string;     // SHA-256
  resource_type: string;    // "css", "js", "image", "font", etc.
  file_path: string;
  file_size: number;
  first_seen: string;       // ISO 8601 timestamp
  last_seen: string;        // ISO 8601 timestamp
}

Storage Layout

data/
├── html/                     # HTML snapshots, organized by date
│   └── 2026/03/12/
│       └── <timestamp>_<hash>.html
└── resources/                # Deduplicated static resources
    └── ab/cd/
        └── <sha256>.css

Features

• High-fidelity replay: CSSOM serialization, computed styles inlining
• Full-page capture: HTML, CSS, JS, images, fonts
• Cross-origin resource recovery: Server-side download of CORS-blocked resources
• Content-hash deduplication: SHA-256 based, shared resources stored once
• Version history: Multiple snapshots per URL, distinguished by timestamp
• Smart dedup: Session-level + server-level prevents redundant captures
• Dynamic content support: Captures live DOM state, auto-updates on significant mutations
• SPA-aware: Detects SPA navigation, resets capture state per route
• Anti-refresh protection: Archived pages frozen (timers, WebSockets, navigation APIs neutralized)
• Hardened cross-origin iframe capture: Embedded cross-origin documents are still captured, but bridge requests are signed and frame HTML returns over a private MessageChannel
• Capture body guardrail: /api/archive and /api/archive/:id reject decompressed JSON bodies above 32 MiB
• Private page filter: Browser side skips loopback, RFC1918, link-local, ULA/loopback IPv6, and .local pages

Testing

See docs/BUILD.md for build and test instructions.

Troubleshooting

Issue	Solution
Server won't start	Check database configuration; for PostgreSQL ensure it's running and database exists
Pages not archiving	Verify Tampermonkey script is enabled and server is reachable
Missing resources	Check proxy settings if behind corporate firewall
Authentication errors	Verify AUTH_PASSWORD env var matches your curl credentials

Configuration

Create .env file in the project root (or set environment variables directly):

The server automatically loads .env from the working directory if it exists.

# Database
DB_TYPE=sqlite  # "sqlite" (default) or "postgres"
DB_PATH=./data/wayback.db  # SQLite database file path (when DB_TYPE=sqlite)

# PostgreSQL configuration (when DB_TYPE=postgres)
DB_HOST=localhost
DB_PORT=5432
# DB_USER defaults to your current system username; falls back to postgres if unavailable
# DB_USER=
DB_PASSWORD=
DB_NAME=wayback
DB_SSLMODE=disable

# Server
SERVER_HOST=0.0.0.0  # 默认 127.0.0.1，设置 0.0.0.0 监听所有网卡
SERVER_PORT=8080

# Storage
DATA_DIR=./data

# Resource metadata cache
RESOURCE_METADATA_CACHE_MB=256  # Optional; RESOURCE_CACHE_MB is a legacy alias

# Authentication (optional)
AUTH_PASSWORD=

# Debug APIs (disabled by default; enable only while troubleshooting)
ENABLE_DEBUG_API=false

License

MIT