| name | add-comic-source |
| description | Add a new comic/manga website source to the comic_downloader project. Use when the user asks to add a new comic site, manga source, or downloader for a specific website URL. Covers the full workflow from site exploration to implementation, testing, and registration. |
Add Comic Source
Add a new ComicSource subclass for a comic/manga website. The project lives in the repo root; source adapters go in downloader/sources/adapters/ with configs in configs/.
Workflow
- Explore the target site - determine URL patterns and page structure
- Choose browser mode -
requests, seleniumbase, or cloakbrowser
- Create config and source module - JSON config + Python class
- Register the source - add to
downloader/sources/registry.py
- Lint, format, and test - ruff + live XPath validation + pytest
Step 1: Explore the Target Site
Use curl (via Bash) and WebFetch to analyze three page types:
| Page | What to find |
|---|
| Search results | URL pattern (/search?q={kw} etc.), result container CSS class, name/URL extraction |
| Manga detail | Name selector (<h1>), metadata (author, status, genre), chapter list container and item structure |
| Chapter/reader | Image loading mechanism: <img src>, <img data-original>, JS variables (chapterImages, etc.) |
Fetch raw HTML with curl -s -L -H "User-Agent: Mozilla/5.0 ..." "<url>" to inspect script tags, CSS classes, and DOM structure that WebFetch may abstract away.
Step 2: Choose Browser Mode
Decision tree based on site behavior:
REQUESTS_MODE (default) - HTML is server-rendered, no JS needed. Fastest. Prefer this.SELENIUMBASE_MODE - Site returns 403/429 to plain requests, or images loaded via JS (e.g., chapterImages variable), or lazy-loaded with scroll.CLOAKBROWSER_MODE - Heavy anti-bot protection, Cloudflare challenges, requires full browser fingerprinting.
Set browser_mode on the class AND in the config JSON. The resolved SourceProfile applies generic keys from configs/{module_name}.json and runtime config overrides.
For image parsing (__parse_imgs__), if images require JS execution, set download_requires_driver = True and use self.driver.get(url) + self.execute_js_safely().
Step 3: Create Config and Source Module
See references/code-patterns.md for templates:
- Config JSON template with XPath expressions
- Source module template with
search(), info(), __parse_imgs__()
downloader/sources/registry.py registration entry
Key conventions:
- Class name:
{SiteName}Comic (e.g., TukuComic)
- Module name: lowercase site name (e.g.,
tuku.py)
- Source module path:
downloader/sources/adapters/{module_name}.py
- Config file:
{module_name}.json
- Import from
downloader.browser.modes for mode constants
- Import from
downloader.comic for Comic, ComicBook, ComicVolume, ComicSource, logger
- Use
self.parse_xpath_list(root, xpath, extract_map) for config-driven list extraction
- Use
self.config['key'] to access XPath expressions from JSON
- Use
''.join(node.itertext()) instead of node.text_content() - lxml.etree._Element lacks text_content()
- URL assembly: handle relative paths with
self.base_url + href if href.startswith('/') else href
Step 4: Register the Source
Add a SourceDefinition entry to the SOURCE_DEFINITIONS tuple in downloader/sources/registry.py. downloader.spec already collects downloader.sources.adapters with collect_submodules(), so do not add a legacy top-level source-module hidden import.
Step 5: Lint, Format, and Test
Run in project root using uv run:
uv run python -m ruff check downloader/sources/adapters/{module}.py
uv run python -m ruff format downloader/sources/adapters/{module}.py
uv run python -c "from downloader.sources import load_source_classes; c = load_source_classes(); print(list(c.keys()))"
uv run python -m pytest tests/test_downloader_spec.py -q
uv run python -c "..."
uv run python -m pytest tests/ -v
