external-client-add

name

external-client-add

description

Add a new typed HTTP client for an external data source (like a new route-lookup API, flight database, weather provider, photo service), wired into the snapshot enrichment pipeline with the standard CachedLookup + disk cache + 429 throttle pattern. Use when the user asks to integrate a new upstream data source — e.g. "add FlightRadar24 lookups", "pull winds aloft from NOAA", "fetch runway data from OurAirports". Don't use for local reference data bundled into the image (aircraft_db.csv.gz et al); those live under FlightJar.Core/ReferenceData.

Adding an external HTTP data source

Every external data source in FlightJar follows the same template: typed HttpClient via IHttpClientFactory + CachedLookup<TKey, TValue>

• gzipped on-disk JSON cache + HttpThrottle + per-key in-flight dedup
• stale-on-error fallback. The shared plumbing lives in FlightJar.Clients/Caching/; individual clients own their upstream URL shape, response parsing, and TTL policy.

Good references when starting: PlanespottersClient (single cache + single throttle, most minimal), AdsbdbClient (two caches sharing one throttle + one disk file via schema version), MetarClient (batched bulk fetch instead of per-key lookups).

Touchpoints

1. dotnet/src/FlightJar.Clients/<Name>/<Name>Client.cs — the client itself. Uses CachedLookup for memory caching + dedup, HttpThrottle for inter-request rate limiting, GzipJsonCache for /data/<name>.json.gz persistence.
2. dotnet/src/FlightJar.Clients/<Name>/<Record>.cs — the POCO returned from lookups (e.g. PhotoInfo, MetarEntry). Plain C# record; system-level serializer handles naming.
3. dotnet/src/FlightJar.Api/Program.cs — register the typed HttpClient + singleton + add the cache-load step to startup.
4. dotnet/src/FlightJar.Core/State/RegistrySnapshot.cs — if the new data is surfaced per aircraft, add fields to SnapshotAircraft.
5. dotnet/src/FlightJar.Api/Workers/RegistryWorker.cs (or the snapshot pusher if you're in a new codebase path) — add the per-tick enrichment loop that reads the cache.
6. Env var — add a FLIGHT_* / NEW_CLIENT_* toggle to FlightJar.Core/Configuration/AppOptions.cs + FlightJar.Api/Configuration/AppOptionsBinder.cs, and wire it through to the client's Enabled flag.
7. Tests — dotnet/tests/FlightJar.Clients.Tests/<Name>/... with a mocked HttpMessageHandler under Mocks/.
8. README.md — add to the "What you get" bullets and the configuration reference table.
9. CLAUDE.md — add to the External-API clients section.

Step 1 — Pick the caching shape

Three variants are in use. Pick whichever matches your upstream:

Variant	Example	Use when
Single cache	PlanespottersClient	One-in-one-out per key lookup, e.g. registration → photo.
Two caches, one disk file	AdsbdbClient	Two related lookups (routes by callsign + aircraft by ICAO) sharing a throttle so concurrent misses don't blow past the rate limit.
Batched	MetarClient	Upstream supports "give me N items in one call" — one request per tick covering every key in the current snapshot's working set.

Do not invent a fourth variant. If none of these fit, ask before coding — it means the upstream has unusual shape (pagination, per-tile fanout, multi-kind responses) that probably warrants a discussion. OpenAipClient is the existing exception (bbox-tile + multi-kind); study it before deviating further.

Step 2 — Write the client

Minimum skeleton (single-cache variant):

public sealed class FooClient : IAsyncDisposable
{
    public const string Url = "https://upstream.example/api/{0}";
    public const int CacheSchemaVersion = 1;

    public static readonly TimeSpan PositiveTtl = TimeSpan.FromHours(12);
    public static readonly TimeSpan NegativeTtl = TimeSpan.FromMinutes(30);
    public const int CacheMaxSize = 10_000;
    public static readonly TimeSpan MinRequestInterval = TimeSpan.FromSeconds(1.2);

    private readonly HttpClient _http;
    private readonly ILogger<FooClient> _logger;
    private readonly TimeProvider _time;
    private readonly string? _cachePath;
    private readonly GzipJsonCache _diskCache;
    private readonly CachedLookup<string, FooRecord> _cache;
    private readonly HttpThrottle _throttle;

    public bool Enabled { get; }

    // ... constructor sets the above fields and initialises the
    // CachedLookup with ("<name>", PositiveTtl, NegativeTtl,
    // CacheMaxSize, _throttle, _time, logger).
}

Always:

• Use IHttpClientFactory-created HttpClient (registered as typed client in Program.cs). Never new HttpClient().
• Inject TimeProvider so tests can run against FakeTimeProvider.
• Use CachedLookup.LookupAsync — it handles in-flight dedup, positive/negative TTL, and stale-on-error. Do not roll your own Dictionary<TKey, Task<TValue>>.
• Translate 429 responses into UpstreamRateLimitedException so CachedLookup honours the Retry-After cooldown.
• Persist only serialisable POCOs to the on-disk cache. Version the schema via CacheSchemaVersion; when you change the shape, bump the version so old caches are dropped cleanly.

Step 3 — Register in Program.cs

Follow the existing AddHttpClient + AddSingleton pattern:

builder.Services.AddHttpClient<FooClient>();
builder.Services.AddSingleton(sp => new FooClient(
    sp.GetRequiredService<IHttpClientFactory>().CreateClient(nameof(FooClient)),
    sp.GetRequiredService<ILogger<FooClient>>(),
    cachePath: Path.Combine(dataDir, "foo.json.gz"),
    enabled: appOptions.FooEnabled));

If the client needs a cache-load at startup, wire that via CreateAsync inside the IHostedService that kicks off workers.

Step 4 — Snapshot enrichment

In RegistryWorker's per-tick loop (BuildEnrichedSnapshotAsync or the nearest equivalent), add a synchronous-at-snapshot-time lookup:

var fooCached = fooClient.LookupCached(key);
// use cached value in the snapshot

Then kick off async warm-up for any cache misses so the next tick has the data ready, bounded by the worker's per-tick budget. Look at how MetarClient batches misses via RefreshMissingAsync(keys, ct) for the pattern.

Do not call LookupAsync synchronously on the snapshot path — that serialises the whole snapshot build behind the upstream's latency. Async warm-up + cached reads is the right discipline.

Step 5 — Env var

In AppOptions.cs add public bool FooEnabled { get; init; } = true; and in AppOptionsBinder.cs bind it from an env var like FOO_ENABLED. Default to enabled unless the upstream requires a key / has questionable ToS — then default to off (see the OPENAIP handling).

Step 6 — Tests

Mock the HttpMessageHandler using the harness in tests/FlightJar.Clients.Tests/Mocks/. Cover:

• Happy path (200 + parse + cache entry written).
• 404 / negative response populates the negative-TTL cache so we don't re-hit the upstream for the same key.
• 429 → UpstreamRateLimitedException → caller-visible retry cooldown. Use FakeTimeProvider to advance past the cooldown.
• Disk cache round-trip (write cache, instantiate a new client pointed at the same path, assert cached value returns from memory without HTTP traffic).

Step 7 — README + CLAUDE.md

Add one bullet to the "What you get" list explaining what the user sees, and one row to the configuration reference table describing the env var. Add the persistence file to the /data/ table in CLAUDE.md. Do not omit these — if a user forks the repo and runs it, the undocumented client is invisible.

Verify

cd dotnet && dotnet format FlightJar.slnx --verify-no-changes
dotnet test tests/FlightJar.Clients.Tests/
dotnet test FlightJar.slnx  # catches enrichment regressions

Do not

• Do not fetch from github.com/<owner>/<repo>/raw/<path>. Use raw.githubusercontent.com — the documented reason in CLAUDE.md is that the redirect path has a lower throughput ceiling on CI and will silently rate-limit builds.
• Do not call the upstream from the hot ingest path. Enrichment happens on the 1 Hz snapshot tick, not per BEAST frame.
• Do not store secrets in the on-disk cache. If the client needs an API key, it belongs in the env and only appears in request headers — never serialised into /data/*.json.gz.
• Do not add a new client if the data could come from an existing one. Adsbdb already exposes routes + tails + photo URLs; OurAirports is already bundled for airport + navaid lookups. Check before adding a parallel dependency.