add-provider

name	add-provider
description	Add a new feed provider to feed-forge. Use when the user asks to "add a provider", "support a new feed source", "wire up <site> feed", "create a new feed source", or anything else that means a new entry in internal/<name>/ that implements providers.FeedProvider and self-registers with the global registry.

Add a new feed-forge provider

This project is a unified RSS/Atom feed generator with a provider registry. Every source (Hacker News, Tildes, YouTube, Oglaf, Feissarimokat, Fingerpori, …) is a self-contained package in internal/<name>/ that:

1. Implements providers.FeedProvider.
2. Self-registers in an init() via providers.MustRegister.
3. Reuses the shared feed generator in pkg/providerfeed instead of writing its own Atom serialization.

Follow the steps below in order. Tildes (internal/tildes/) is the cleanest read-only Atom-passthrough example; YouTube and Oglaf add multi-feed and content-DB caching respectively. Ignore internal/reddit-json/ — it carries Reddit-specific auth/proxy/OG machinery that does not generalize.

1. Decide the provider's shape upfront

Answer these before writing code — they drive the rest of the file layout:

• Source format. Atom (XML), RSS (XML), JSON, or HTML scrape? Atom/RSS providers reuse encoding/xml directly (see internal/tildes/api.go, internal/youtube/api.go). HTML providers regex-extract (see internal/oglaf/provider.go for stripImgRegex).
• Stateful or stateless. Stateless = re-fetch every run, no DB (Tildes, YouTube, Feissarimokat). Stateful = needs a content DB for incremental fetch / image backfill / dedup (Oglaf has oglaf.db). Drives DatabaseConfig.UseContentDB.
• Single feed or many. A provider can fan out across multiple feed URLs (YouTube feed-urls/channel-ids, Tildes topics). If yes, plan a normalize helper and a per-feed-config branch in feedConfig().
• Comments link distinct from article link. If yes (Tildes, Reddit), Item.Link() returns the external article and Item.CommentsLink() returns the discussion page. If no, both return the same URL.
• Score / comment counts available. If not, the Item.Score() / Item.CommentCount() methods return 0 — that is fine, the template handles zero gracefully.

2. Files to create

Create internal/<name>/ containing:

File	Purpose
provider.go	Provider struct, Config, factory, init() registration, previewInfo, NewXProvider, FetchItems, feedConfig (only if config varies from previewInfo.Config).
types.go	XML/JSON structs for the upstream format + Item struct implementing providers.FeedItem. Small providers can keep types in provider.go (see Oglaf) but a separate types.go is the norm.
api.go	HTTP fetch + parse + any normalization helpers (normalizeTopics, buildFeedURL, cleanContent).
provider_test.go	At minimum a fixture-driven TestFetchItemsAgainstFixture that spins up httptest.NewServer from testdata/. Pattern: see internal/tildes/provider_test.go.
testdata/<source>.xml (or .json)	Recorded upstream response used by the fixture test.

Then touch these shared files:

• cmd/feed-forge/main.go — add the import (triggers init()), CLI sub-struct, buildProviderConfig case, and switch ctx.Command() case.
• templates/<name>-atom.tmpl — Atom template (start by copying templates/tildes-atom.tmpl and trimming).
• config_example.yaml — example YAML block.

3. Implement Provider, Config, and registration (provider.go)

Template (replace xname / XName):

package xname

import (
    "fmt"
    "log/slog"
    "sort"

    "github.com/lepinkainen/feed-forge/pkg/feedmeta"
    "github.com/lepinkainen/feed-forge/pkg/providerfeed"
    "github.com/lepinkainen/feed-forge/pkg/providers"
)

var previewInfo = &providers.PreviewInfo{
    Config: feedmeta.Config{
        Title:       "XName",
        Link:        "https://example.com/",
        Description: "XName feed generated by Feed Forge",
        Author:      "Feed Forge",
        ID:          "https://example.com/feed",
    },
    ProviderName: "XName",   // shown in preview UI + OPML
    TemplateName: "xname-atom", // must match templates/xname-atom.tmpl
}

type Provider struct {
    *providers.BaseProvider
    // ...per-provider fields (FeedURL, Limit, etc.)
}

type Config struct {
    providers.GenerateConfig `yaml:",inline"` // gives Outfile + Interval
    // ...yaml-tagged fields matching the CLI sub-struct
}

func factory(config any) (providers.FeedProvider, error) {
    cfg, ok := config.(*Config)
    if !ok {
        return nil, fmt.Errorf("invalid config type for xname provider: expected *xname.Config")
    }
    return NewXNameProvider(/* fields from cfg */)
}

func init() {
    providers.MustRegister("xname", &providers.ProviderInfo{
        Name:        "xname",
        Description: "Generate RSS feeds from XName",
        Version:     "1.0.0",
        Factory:     factory,
        ConfigFactory: func() any { return &Config{/* defaults */} },
        Preview:     previewInfo,
    })
}

func NewXNameProvider(/* args */) (providers.FeedProvider, error) {
    base, err := providers.NewBaseProvider(providers.DatabaseConfig{
        UseContentDB:  false, // true + ContentDBName for stateful providers
        // ContentDBName: "xname.db",
    })
    if err != nil {
        slog.Error("Failed to create base provider for XName", "error", err)
        return nil, fmt.Errorf("initialize xname base provider: %w", err)
    }

    p := &Provider{BaseProvider: base /* , ... */}
    p.SetGenerateFeedFunc(providerfeed.BuildGenerator(
        p.FetchItems,
        previewInfo,
        nil,        // or p.feedConfig if config varies per-instance
        p.OgDB,     // pass nil if the template does not consume OpenGraph data
    ))
    return p, nil
}

func (p *Provider) FetchItems(limit int) ([]providers.FeedItem, error) {
    // 1. Fetch upstream (use pkg/api client — never net/http directly).
    // 2. Parse into your Item structs.
    // 3. Sort newest-first if upstream order is not reliable.
    // 4. Honor `limit` (0 = no cap) AFTER sorting.
    // 5. Return []providers.FeedItem (slice of *Item).
}

Key contracts:

• MustRegister panics on duplicate name — pick a unique kebab-case name.
• ConfigFactory returns *Config with field defaults pre-filled. YAML decoding writes on top of those defaults via loadProviderConfigFromYAML in cmd/feed-forge/main.go.
• Config MUST embed providers.GenerateConfig with yaml:",inline". That is the only thing the generate command sees for outfile + interval.
• providerfeed.BuildGenerator handles httpcache.ErrNotModified (touches mtime, returns nil), directory creation, template rendering, OpenGraph fetching, and logging. Don't reimplement any of that.

4. Implement Item (types.go)

Item must satisfy providers.FeedItem:

type FeedItem interface {
    Title() string
    Link() string         // external article URL (also used as OpenGraph fetch key)
    CommentsLink() string // discussion URL; equal to Link() if no separate discussion
    Author() string
    Score() int
    CommentCount() int
    CreatedAt() time.Time
    Categories() []string
    ImageURL() string
    Content() string      // HTML body for the entry
}

Optional duck-typed extras the template renderer detects:

• AuthorURI() string — populates Atom <author><uri> (see Tildes).
• VideoID(), channel helpers — anything your template references.

Conventions:

• HTML-unescape titles at the boundary: upstream Atom CDATA often hides " etc. See internal/tildes/types.go:62.
• CreatedAt() MUST return time.Time, never a string. Sort uses .After. This matches the time.Time/TIMESTAMP rule in CLAUDE.md.
• ImageURL() is consumed by templates for <media:thumbnail>. Return "" if none.
• Content() is wrapped in <![CDATA[…]]> by the template — return ready-to-render HTML, not plain text.

5. Implement fetch (api.go)

Always use pkg/api. Never net/http.Get directly. Picks:

• api.NewGenericClient() — almost everything (Tildes, YouTube, Oglaf, Feissarimokat).
• api.NewHackerNewsClient() / api.NewRedditClient() — only for those specific APIs.

For conditional GET / 304 handling on RSS-style sources, wrap with httpcache.CachedGet(ctx, client, store, url, headers) (see internal/oglaf/provider.go:329). The store comes from p.HTTPCache. Return httpcache.ErrNotModified from FetchItems to let providerfeed.BuildGenerator bump the output mtime and skip rewrite.

Atom/RSS parse pattern (Tildes-style):

client := api.NewGenericClient()
resp, err := client.Get(feedURL, nil)
if err != nil { return nil, fmt.Errorf("fetch xname feed: %w", err) }
defer func() { _ = resp.Body.Close() }()

body, err := io.ReadAll(resp.Body)
if err != nil { return nil, fmt.Errorf("read xname response: %w", err) }

var feed atomFeed
if err := xml.Unmarshal(body, &feed); err != nil {
    return nil, fmt.Errorf("parse xname atom: %w", err)
}

6. Add the Atom template (templates/-atom.tmpl)

Start by copying templates/tildes-atom.tmpl and trimming. The templates/embedded.go //go:embed *.tmpl line picks up new files automatically — no edit needed there. Template data shape lives in pkg/feed/template.go (TemplateData and TemplateItem):

• Feed-level: FeedTitle, FeedLink, FeedDescription, FeedAuthor, FeedID, Updated, Generator.
• Item-level: Title, Link, CommentsLink, ID, Updated, Published, Author, AuthorURI, Categories, Score, Comments, Content, Summary, ImageURL + provider-specific Subreddit, Domain.
• OpenGraphData map keyed by .Link is populated for you when you pass p.OgDB to BuildGenerator.

Always pipe interpolated strings through xmlEscape (already registered in the template funcs).

7. Wire into the CLI (cmd/feed-forge/main.go)

Four edits, all in cmd/feed-forge/main.go:

1. Import the new package in the import block (the comment there says "Import providers to trigger init() self-registration" — that is the whole reason the import exists).
2. CLI sub-struct under var CLI struct { … }. Mirror Config field-for-field. Each field needs help:"…", default:"…" where useful, and a yaml:"…" tag matching the Config tag. Add interval and outfile fields. cmd:"<name>" on the embedded struct registers the subcommand.
3. buildProviderConfig — add a case "<name>": that maps CLI fields into your *Config. CRITICAL (from CLAUDE.md): Kong only fills the active subcommand's CLI sub-struct, so generate does NOT use buildProviderConfig; it goes through loadProviderConfigFromYAML which is why every Config field needs a yaml tag.
4. switch ctx.Command() in main() — add a case "<name>": that builds the config, creates the provider via the registry, resolves the outfile via resolveOutfile, and calls provider.GenerateFeed.

8. Add an example to config_example.yaml

Add a block matching your YAML tags. Include interval: 30m (or whatever is sane for the source's update cadence — comics are usually 24h).

9. Test

• Fixture test: copy internal/tildes/provider_test.go and adapt. Drop a recorded response into testdata/. Verify counts/title/link mapping. Avoid spinning up NewBaseProvider in unit tests — it opens real SQLite files. Instead construct Item directly from parsed entries, as Tildes' test does.
• task build runs vet + tests + lint. Use it instead of go build. Run goimports -w . after Go edits.
• For preview during development: ./build/feed-forge preview <name> opens the TUI. ./build/feed-forge preview <name> --index 0 prints one item's rendered XML to stdout — handy for template iteration.
• Golden files: if you add golden-style tests, run task update-golden once results stabilize and review the diff before committing.

Pitfalls

• Forgetting the import in main.go. Without it, init() never runs and MustRegister is silently absent — generate will not find the provider.
• Missing yaml tags on Config fields. The generate command path bypasses Kong; only YAML tags are read. Fields without them stay zero-valued in generate runs but work fine for the single-provider subcommand. Always add both.
• Returning strings instead of time.Time for timestamps. Breaks chronological sort and stateful DB ordering (CLAUDE.md "Database Timestamps").
• Direct net/http calls. Bypasses rate limiting + retries — use pkg/api clients.
• Writing your own Atom serializer. Use providerfeed.BuildGenerator + a template. The shared generator handles OpenGraph fetching, escaping, time formatting, and ErrNotModified skips for you.
• Reading reddit-json for inspiration. It has proxy/auth code that no other provider needs and will lead you down dead ends. Read Tildes / YouTube / Oglaf instead.