architecture

name	architecture
description	SignalDB architecture reference - FDAP stack, write/query data flow, service components, deployment models, and dual catalog system. Use when understanding how components fit together, data flow, or system design.
user-invocable	false
sources	["docs/architecture/overview.md","src/signaldb-bin/src/**","Cargo.toml"]

SignalDB Architecture Reference

FDAP Stack

SignalDB is built on Flight, DataFusion, Arrow, Parquet:

• Flight: Apache Arrow Flight for zero-copy inter-service gRPC communication
• DataFusion: SQL query processing engine (used by Querier)
• Arrow: In-memory columnar format used throughout the entire pipeline
• Parquet: Persistent columnar storage via Iceberg table format

Critical rule: Always use Arrow/Parquet types re-exported by DataFusion to ensure version compatibility:

// CORRECT
use datafusion::arrow::array::StringArray;
use datafusion::parquet::arrow::ArrowWriter;

// WRONG - version mismatch risk
use arrow::array::StringArray;

Write Path

OTLP Client (gRPC :4317 / HTTP :4318)
    -> Acceptor (validates auth, enforces rate limits + storage quotas, converts OTLP->Arrow, writes to WAL)
    -> Writer via Flight do_put (transforms v1->v2 schema, writes to WAL)
    -> WalProcessor (background, 5s interval)
    -> Iceberg Tables (Parquet files in object store)

Key details:

1. Acceptor writes to WAL before acknowledging client
2. Acceptor converts OTLP protobuf -> Arrow RecordBatches using Flight schemas (v1)
3. Writer transforms v1 Flight schema -> v2 Iceberg schema (field renames, type conversions, computed partition fields)
4. Writer's WalProcessor reads WAL entries every 5s (exponential backoff on failure), writes Parquet via DataFusion
5. Processed WAL entries are marked and cleaned up

Query Path

HTTP Client (Tempo / Pyroscope / Loki APIs)
    -> Router (:3000 HTTP, :50053 Flight)
    -> Querier via Flight do_get (:50054)
    -> DataFusion SQL against Iceberg tables
    -> Parquet files in object store
    -> Results stream back as Arrow RecordBatches

Key details:

1. Router validates auth, discovers Queriers via QueryExecution capability
2. Flight tickets encode query type + tenant context: find_trace:{tenant}:{dataset}:{trace_id}[:{start}:{end}] (optional unix-second time hints)
3. Querier uses TenantCatalog to bridge DataFusion 3-level model to Iceberg 2-level namespace
4. Results stream back as Arrow RecordBatches via Flight (trace not found -> Flight not_found status)
5. Standalone querier also serves Tempo's tempopb.Querier gRPC protocol on the Flight port (see tempo-api skill)
6. Router also nests query APIs for every signal, each lowering a parsed query to a DataFusion Expr/aggregate over the signal's Iceberg tables (never a SQL string, matching the trace path):
   • /loki (LogQL, epic #366): LogsService executes log queries (streams) and metric queries (rate/count_over_time/sum by, date_bin bucketed matrix); LogsService also backs /loki labels/values/series.
   • /prometheus (PromQL, epic #328): MetricsService executes selectors, sum/avg/min/max/count [by], and rate/increase over metrics_gauge+metrics_sum (date_bin bucketed matrix), plus histogram_quantile(phi, metric) interpolated from metrics_histogram OTLP buckets; MetricsService also backs /prometheus labels/values/series. histogram_quantile over an inner rate(), binary ops, and topk remain pending (#335).
   • /pyroscope (profiles) via ProfileService.

Service Components

Service	Ports	Capability	Key Files
Acceptor	gRPC:4317, HTTP:4318	TraceIngestion	src/acceptor/
Writer	Flight:50061 (standalone), 50051 (mono)	TraceIngestion, Storage	src/writer/
Router	HTTP:3000, Flight:50053	Routing	src/router/
Querier	Flight:50054	QueryExecution	src/querier/
Compactor	None (background task)	Compaction	src/compactor/

Deployment Models

• Monolithic (cargo run --bin signaldb): All services in one process, shared SQLite catalog. Compactor included if enabled in config.
• Microservices: Independent binaries, shared catalog (PostgreSQL or SQLite)
• Hybrid: Mix of co-located and distributed services

Note: Monolithic mode integrates the Compactor service (Phases 1-3 complete) when [compactor].enabled = true. The compactor provides:

• Phase 1: Dry-run compaction planning
• Phase 2: Active Parquet file compaction for storage efficiency
• Phase 3: Retention enforcement, snapshot expiration, and orphan file cleanup

The compactor runs concurrent background loops for compaction, retention enforcement, and orphan cleanup, all using tokio::select! for non-blocking execution.

Dual Catalog System

1. Service Catalog (Catalog): PostgreSQL/SQLite for service discovery, tenant management, API keys, datasets
2. Iceberg Catalog (CatalogManager): SQLite-only SQL catalog named "signaldb" for Iceberg table metadata

The Iceberg catalog only supports SQLite (not PostgreSQL). This is distinct from the service catalog.