impulse-data-model

name

impulse-data-model

description

Understand and prepare the data Impulse reads and writes. Use when the user asks "what tables does Impulse need", how to land / ingest measurement data into the silver layer, what the gold-layer output looks like, how fact and dimension tables join, or how to point Impulse at existing tables whose column names differ (via SolverConfig column mappings). Covers the three required silver tables, the optional tag/mapping/unit tables, RLE vs RAW channel formats, the gold star schema, and the SolverConfig / custom-solver escape hatches.

Impulse — data model (silver input, gold output)

Impulse reads a silver layer of measurement tables and writes a gold layer star schema. It does not ship an ingestion component — producing the silver layer is your responsibility, and landing data in the shape below is the simplest path.

Silver layer — the input Impulse reads

DefaultSolver needs only three tables. The rest are optional add-ons, used only when configured in source (see impulse-config).

Table	Required?	Purpose
container_metrics	Yes	One row per recording — timestamps, duration, channel count, and any container-level columns.
channel_metrics	Yes	One row per (container_id, channel_id) — per-channel statistics; also holds channel-selection columns (e.g. channel_name) in the wide model.
channels	Yes	The time-series sample data (RLE or RAW — see below).
container_tags	Optional	EAV (container_id, key, value). Add for tag-based container filtering.
channel_tags	Optional	EAV (container_id, channel_id, key, value). Add for EAV channel selection.
channel_mapping	Optional	Logical→physical channel alias table (enables channel_with_alias()).
unit_conversion	Optional	Per-unit conversion factors (used with channel_mapping).

Key invariants when landing your own data

• container_id is the primary key on container_metrics and the foreign key everywhere else. One container = one recording. Pick a stable integer/long (or string) per recording; the engine adopts whatever type your tables use, as long as it is consistent across them.
• (container_id, channel_id) identifies a channel within a container. Channel IDs are local to their container.
• channels supports two formats:
  • RAW — one row per sample: (container_id, channel_id, timestamp, value).
  • RLE — one row per stable interval: (container_id, channel_id, tstart, tend, value). Run-length encoding collapses consecutive identical values into intervals and greatly reduces processing time. Set query_engine.data_type to "RAW" or "RLE" to match (see impulse-config).
  • An optional boolean is_plausible column lets the solver drop implausible samples when drop_implausible_data=True (requires RAW).
  • Extra columns on channels are ignored — the engine projects down to the columns above before solving, so it is safe to keep additional bookkeeping columns on the table.
• Tag tables are strict EAV. query.channel(channel_name="Engine RPM") looks up channel_tags.value where key = 'channel_name'. Without channel_tags, channel selectors match columns on channel_metrics instead.
• The remaining metric columns (durations, min/max/mean, …) are not fixed by the engine — add the columns your queries and gold dimensions need. You do not have to match the demo schema column-for-column.

Channel selection is metadata-driven

Channels are always selected by signal metadata (tags or channel_metrics columns), never by fixed column positions — so the same schema supports arbitrary signal sets across projects. How the solver resolves a selection depends on which optional tables you configured:

• With channel_tags configured → channels selected from its EAV rows (pivoted on the fly).
• Without it → channels selected from columns on channel_metrics.
• With container_tags configured → containers filtered from EAV rows; without it, from container_metrics columns (wide-only model).

Landing data (ingestion pattern)

If your CSVs already match the shape, loading is a few lines:

import os, pandas as pd
csv_dir = "/Volumes/my_catalog/silver/raw/reporting"
for t in ["container_metrics", "container_tags", "channel_metrics", "channel_tags", "channels"]:
    (spark.createDataFrame(pd.read_csv(f"{csv_dir}/{t}.csv"))
          .write.mode("overwrite")
          .saveAsTable(f"my_catalog.silver.{t}"))

For real ingestion of MDF4 / vendor binaries, the typical Databricks skeleton is: detect files with Auto Loader → decode per-file in a Spark UDF into a bronze samples table → collapse bronze into channels (RAW or RLE) and derive the *_metrics / *_tags tables → track run status → periodically OPTIMIZE (cluster/Z-order channels on container_id, channel_id, since it is by far the largest). Implement only the steps your situation needs.

Gold layer — the output Impulse writes

A star schema. Every table is prefixed with your configured table_prefix (e.g. my_report_histogram_fact).

Fact tables

Table	Grain
event_instance_fact	One row per event instance per container
histogram_fact	One row per bin per container
histogram2d_fact	One row per (x, y) bin per container
stats_aggregator_fact	One row per statistic label per signal per event instance
calculated_channel_fact	One row per sample interval per container (a derived signal, silver channels shape)
calculated_channel_metrics	One row per calculated channel per container (optional; silver channel_metrics shape). Written only when config.calculated_channels.emit_channel_metrics is set. See impulse-channels.

Dimension tables

Table	Holds
measurement_dimension	Container metadata selected via measurement_dimensions.
event_dimension	Event definitions (name, TSAL expression, required channels).
histogram_dimension	Histogram metadata (bins, signal info, units).
histogram2d_dimension	2D histogram metadata.
stats_aggregator_dimension	Statistics metadata (channel names, statistic labels incl. custom).
calculated_channel_dimension	Calculated-channel definitions (name, expression, identity). See impulse-channels.

Join pattern — key columns connect facts to dimensions:

• container_id → links every fact to measurement_dimension.
• event_id → links event_instance_fact, histogram_fact, histogram2d_fact to event_dimension.
• visual_id → links each aggregation fact to its own dimension table.
• channel_id → links calculated_channel_fact to calculated_channel_dimension.

stats_aggregator_fact additionally joins to event_instance_fact via event_instance_id for per-interval breakdowns.

The reporting mode writes all of these; see impulse-reporting for column-level fact/dimension schemas and impulse-config for how measurement_dimensions picks the container columns.

Adapting to an existing layout

Reshaping into the silver shape at ingest is recommended. If the data already lives in Delta with different names and rewriting is impractical, there are two escape hatches. The rule of thumb:

• SolverConfig for naming differences — same tables and relationships, different column names.
• Custom solver for structural differences — no EAV tags, alias lookups, composite keys.
• ETL into the standard shape for everything else.

Column-name remapping with SolverConfig

Declare a per-table mapping from your physical names to the engine's fixed internal names (container_id, channel_id, tstart, tend, value, key, …). The mapping is applied once, when the table is read; everything downstream uses the internal names. Set it under query_engine.solver_config in your report config:

"query_engine": {
    "solver": "DefaultSolver",
    "solver_config": {
        "container_metrics": {"column_name_mapping": {"my_measurement_id": "container_id"}},
        "channels": {"column_name_mapping": {"start_us": "tstart", "end_us": "tend"}}
    }
}

In RAW mode the channels internal names are timestamp and (optionally) is_plausible — remap them the same way (e.g. {"ts_raw": "timestamp"}).

The underlying tables must still follow the silver-layer relationships (EAV tag tables, per-(container_id, channel_id) channel rows). The full SolverConfig schema — per-table filters, project_id scoping, channel_mapping.join_keys, and unit conversion — is in impulse-config.

Custom solver

For layouts that don't match the relationships at all, subclass QuerySolver (from impulse_query_engine.analyze.query.solvers) and register it in config. You take on the solver pipeline stages (filter_container_tags, filter_container_metrics, filter_channel_tags, filter_channel_metrics, solve). This is a large investment — first check whether a one-time ETL into the standard shape is cheaper.