Impulse Data Model

Impulse — data model (silver input, gold output)

Impulse reads a silver layer of measurement tables and writes a gold layer star schema. It does not ship an ingestion component — producing the silver layer is your responsibility, and landing data in the shape below is the simplest path.

Silver layer — the input Impulse reads

DefaultSolver needs only three tables. The rest are optional add-ons, used only when configured in source (see impulse-config).

Key invariants when landing your own data

• container_id is the primary key on container_metrics and the foreign key everywhere else. One container = one recording. Pick a stable integer/long (or string) per recording; the engine adopts whatever type your tables use, as long as it is consistent across them.
• (container_id, channel_id) identifies a channel within a container. Channel IDs are local to their container.
• channels supports two formats:
  • RAW — one row per sample: (container_id, channel_id, timestamp, value).
  • RLE — one row per stable interval: (container_id, channel_id, tstart, tend, value). Run-length encoding collapses consecutive identical values into intervals and greatly reduces processing time. Set query_engine.data_type to "RAW" or "RLE" to match (see impulse-config).
  • An optional boolean is_plausible column lets the solver drop implausible samples when drop_implausible_data=True (requires RAW).
  • Extra columns on channels are ignored — the engine projects down to the columns above before solving, so it is safe to keep additional bookkeeping columns on the table.
• poi_channels holds Points-in-Time (POI) data — a value defined only at its timestamp. Schema: (container_id long, channel_id int, timestamp long [epoch µs], value_double double nullable, value_string string nullable).
• Tag tables are strict EAV. query.channel(channel_name="Engine RPM") looks up channel_tags.value where key = 'channel_name'. Without channel_tags, channel selectors match columns on channel_metrics instead.
• The remaining metric columns (durations, min/max/mean, …) are not fixed by the engine — add the columns your queries and gold dimensions need. You do not have to match the demo schema column-for-column.

Channel selection is metadata-driven

Channels are always selected by signal metadata (tags or channel_metrics columns), never by fixed column positions — so the same schema supports arbitrary signal sets across projects. How the solver resolves a selection depends on which optional tables you configured:

• With channel_tags configured → channels selected from its EAV rows (pivoted on the fly).
• Without it → channels selected from columns on channel_metrics.
• With container_tags configured → containers filtered from EAV rows; without it, from container_metrics columns (wide-only model).

Beyond filtering, these container tables also feed container-level metadata into UDFs at solve time: a UDF declaring container_metrics=[...] (columns) or container_tags=[...] (EAV keys, which require a container_tags_table) receives their per-container values as keyword arguments — see impulse-tsal.

Landing data (ingestion pattern)

If your CSVs already match the shape, loading is a few lines:

import os, pandas as pd
csv_dir = "/Volumes/my_catalog/silver/raw/reporting"
for t in ["container_metrics", "container_tags", "channel_metrics", "channel_tags", "channels"]:
    (spark.createDataFrame(pd.read_csv(f"{csv_dir}/{t}.csv"))
          .write.mode("overwrite")
          .saveAsTable(f"my_catalog.silver.{t}"))

For real ingestion of MDF4 / vendor binaries, the typical Databricks skeleton is: detect files with Auto Loader → decode per-file in a Spark UDF into a bronze samples table → collapse bronze into channels (RAW or RLE) and derive the *_metrics / *_tags tables → track run status → periodically OPTIMIZE (cluster/Z-order channels on container_id, channel_id, since it is by far the largest). Implement only the steps your situation needs.

Gold layer — the output Impulse writes

A star schema. Every table is prefixed with your configured table_prefix (e.g. my_report_histogram_fact).

Fact tables

Dimension tables

Join pattern — key columns connect facts to dimensions:

• container_id → links every fact to measurement_dimension.
• event_id → links event_instance_fact, histogram_fact, histogram2d_fact to event_dimension.
• visual_id → links each aggregation fact to its own dimension table.
• channel_id → links calculated_channel_fact to calculated_channel_dimension.

stats_aggregator_fact additionally joins to event_instance_fact via event_instance_id for per-interval breakdowns.

The reporting mode writes all of these; see impulse-reporting for column-level fact/dimension schemas and impulse-config for how measurement_dimensions picks the container columns.

Adapting to an existing layout

Reshaping into the silver shape at ingest is recommended. If the data already lives in Delta with different names and rewriting is impractical, there are two escape hatches. The rule of thumb:

• SolverConfig for naming differences — same tables and relationships, different column names.
• Custom solver for structural differences — no EAV tags, alias lookups, composite keys.
• ETL into the standard shape for everything else.

Column-name remapping with SolverConfig

Declare a per-table mapping from your physical names to the engine's fixed internal names (container_id, channel_id, tstart, tend, value, key, …). The mapping is applied once, when the table is read; everything downstream uses the internal names. Set it under query_engine.solver_config in your report config:

"query_engine": {
    "solver": "DefaultSolver",
    "solver_config": {
        "container_metrics": {"column_name_mapping": {"my_measurement_id": "container_id"}},
        "channels": {"column_name_mapping": {"start_us": "tstart", "end_us": "tend"}}
    }
}

In RAW mode the channels internal names are timestamp and (optionally) is_plausible — remap them the same way (e.g. {"ts_raw": "timestamp"}).

The underlying tables must still follow the silver-layer relationships (EAV tag tables, per-(container_id, channel_id) channel rows). The full SolverConfig schema — per-table filters, project_id scoping, channel_mapping.join_keys, and unit conversion — is in impulse-config.

Custom solver

For layouts that don't match the relationships at all, subclass QuerySolver (from impulse_query_engine.analyze.query.solvers) and register it in config. You take on the solver pipeline stages (filter_container_tags, filter_container_metrics, filter_channel_tags, filter_channel_metrics, solve). This is a large investment — first check whether a one-time ETL into the standard shape is cheaper.