Parquet Storage Schema #

Each Litmus run produces one Parquet file. The file has two layers: the at-rest format and the query projection.

At-rest — three row types. Every row carries an explicit record_type discriminator with one of three values:

• record_type = 'run' — exactly one row per file. Carries run-level identity, timing, outcome, plus UUT / station / project / git / environment context.
• record_type = 'step' — one row per (step_path, vector_index) execution. Step identity, timing, and rolled-up outcome. Conditions and observations are on the paired vector row, not here.
• record_type = 'vector' — one row per execution. Every step execution has at least one vector row: a synthesized scope vector for the step itself. Mode-2 (vectors-fixture / run_vector) loops add one vector row per iteration. The vector row carries the inputs / outputs lane columns and a nested measurements list (LIST<STRUCT>).

Measurements are nested, not rows. Measurements live inside the vector row's measurements column as a typed nested list (LIST<STRUCT>). Each struct holds name, value, unit, outcome, timestamp, limit_*, characteristic_id, spec_ref, and signal-path fields (uut_pin, fixture_connection, instrument_*). There is no at-rest record_type = 'measurement' row.

Query projection — four virtual types. The query projection UNNESTs the nested measurements from each vector row into a flat fact and presents a fourth virtual row type record_type = 'measurement' in query results. All WHERE record_type = 'measurement' queries target this projected view, not the at-rest file. The inputs / outputs lanes are also projected into the measurements_dynamic EAV table (keyed by role and name) for query-time access. Query output shape is byte-stable regardless of at-rest format changes.

This page mirrors the canonical at-rest schema; the column names and types here match what read_parquet returns.

File layout #

<data_dir>/runs/{date}/
├── {timestamp}_{run_id8}_{serial}.parquet  # Run + step + vector rows; measurements nested in vector rows
├── {timestamp}_{run_id8}.parquet           # Same shape, no UUT serial (dev runs)
└── {timestamp}_{run_id8}_{serial}_ref/     # Reference data (waveforms, images, files)
    ├── {vector_id}_scope_waveform.npz
    ├── {vector_id}_camera_image.png
    └── ...

Timestamps are UTC and sort naturally. The 8-char run_id sits right after the timestamp (the trailing serial is optional, so its absence never shifts the leading parts) and disambiguates two runs of the same serial that start in the same second. DuckDB / Spark / Polars / Pandas all read the file directly with read_parquet.

Discriminator #

At-rest row types (three):

• run — one row per run; run-level metadata (start/end timestamps, UUT serial, station, outcome).
• step — one row per (step_path, vector_index) execution; code identity, timing, rolled-up outcome.
• vector — one row per execution, keyed (step_path, vector_index, retry). Every step has at least one vector row: its scope vector. Mode-2 (vectors-fixture) loops add one iteration vector per pass. The vector row carries inputs / outputs lane columns and a nested measurements list.

Query projection (virtual fourth type):

• measurement — the query projection UNNESTs each vector's nested measurements list into a flat fact row stamped record_type = 'measurement'. These rows exist in query results but not in the at-rest file.

To list steps: WHERE record_type = 'step'. To list vectors: WHERE record_type = 'vector'. To list measurements: WHERE record_type = 'measurement'. All kinds: omit the filter.

Identity & timing #

Who — operator #

What — UUT #

What — part spec #

Where — station #

Where — fixture #

Where — instruments (dynamic step_instruments_*) #

Per-step instrument identity, captured from the pytest fixtures the test actually used. All columns are list[string] (one entry per instrument) and arrays stay in parallel order.

For real hardware, identity comes from *IDN? at session start. For mock instruments, identity comes from the instrument YAML configs.

-- DuckDB: unnest parallel arrays for per-instrument queries
SELECT
    step_name,
    unnest(step_instruments_name) AS instrument,
    unnest(step_instruments_serial) AS serial,
    unnest(step_instruments_cal_due) AS cal_due
FROM read_parquet('data/runs/**/*.parquet')
WHERE record_type = 'step';

Test context #

Input conditions (inputs lane — at-rest format) #

At rest, each vector's commanded conditions are stored in the inputs column as a typed nested list: LIST<STRUCT<name, value_type, value_int, value_double, value_bool, value_text, value_timestamp, value_json, unit, uut_pin>>. One struct per parameter; value_type selects which value_* field holds the actual value.

The Query API projects these lane structs into the measurements_dynamic EAV table (keyed by role='input' and name) for query-time access. See Query API for how to select input fields in analysis.

Entry structure (one item in the inputs list):

Naming convention (applies to name inside each entry):

Bare names are spec-relevant for condition matching; prefixed names are stimulus/settings. Convention is enforced by docs, not by the writer.

Stimulus signal-path sub-fields for each param (also stored in the inputs lane as separate entries with compound names):

Observations (outputs lane — at-rest format) #

Observations are measured context — readings captured during the test, not commanded values. Stored at rest in the outputs column with the same LIST<STRUCT> shape as inputs. The Query API projects these into the measurements_dynamic EAV table with role='output'.

Each struct entry encodes one observation under name. Non-scalar payloads route to the _ref/ sibling directory and are stored as file:// URIs with value_type = 'uri' in the value_text field:

from litmus.data.backends.parquet import load_file, is_file_reference
 
if is_file_reference(column_value):
    data = load_file(parquet_path, column_value)

Measurement fields (projected from nested struct) #

At rest, measurements live in the vector row's measurements column as a LIST<STRUCT>. The fields below are exposed as flat columns on the projected record_type = 'measurement' rows the Query API surfaces.

Limits (on projected record_type='measurement' rows) #

Spec traceability #

-- Yield by characteristic across all parts
SELECT characteristic_id, part_id,
       AVG(CASE WHEN measurement_outcome='passed' THEN 1.0 ELSE 0.0 END) AS yield
FROM read_parquet('data/runs/**/*.parquet')
WHERE record_type = 'measurement'
GROUP BY characteristic_id, part_id;

Measurement signal path #

Rollup outcomes #

Environment traceability #

Custom metadata #

Test code can add arbitrary run-level metadata via run_context.set():

def test_example(run_context, psu, dmm, verify):
    run_context.set("operator_badge", "EMP-12345")
    run_context.set("fixture_serial", "FIX-001")
    run_context.set("ambient_temp", 23.5)
    ...

At rest, custom metadata is stored as a JSON blob in the Parquet file-level metadata under the key custom_metadata — not as a column in the row data. It is run-scoped (one blob per file, not one entry per measurement).

measurements_dynamic EAV table (query projection) #

The Query API projects all inputs and outputs lane entries into a long EAV table named measurements_dynamic. This is what the Query API reads when you select inputs or outputs by name.

Querying this table directly is rarely needed — use the Query API (FieldRef.input("vin"), FieldRef.output("v_rail")) which joins it for you and handles type coherence (fails loud if a name carries mixed value_types in scope; auto-resolves when unambiguous).

Outcome values #

Comparator values #

Retries #

All retries are stored. Each retry produces a new step + vector pair at that execution's grain, so measurement-less retries are captured. Measurements stay nested in the paired vector row.

Mode-1 (parametrize / single / unswept): each attempt is a separate step row + scope vector row. vector_retry increments per attempt. In the query projection, the UNNESTed measurement rows carry the same vector_retry as their enclosing vector.

record_type | vector_index | vector_retry | step_outcome   | measurement_name | measurement_value
step        | 0            | 0            | failed         | —                | —                  ← first attempt (at-rest)
vector      | 0            | 0            | failed         | —                | —                  ← scope vector with nested measurements
measurement | 0            | 0            | —              | output_voltage   | 3.50               ← projected (query-time UNNEST)
step        | 0            | 1            | failed         | —                | —                  ← first retry
vector      | 0            | 1            | failed         | —                | —
measurement | 0            | 1            | —              | output_voltage   | 3.48               ← projected
step        | 0            | 2            | passed         | —                | —                  ← second retry
vector      | 0            | 2            | passed         | —                | —
measurement | 0            | 2            | —              | output_voltage   | 3.30               ← projected

Mode-2 (vectors fixture): each attempt at one in-body iteration is a separate vector row with the same vector_index and an incremented vector_retry.

Filter to the final attempt with WHERE vector_retry = (SELECT MAX(vector_retry) FROM … WHERE record_type IN ('step','vector') …), scoping by (run_id, step_path, vector_index).

File-level metadata #

Beyond columns, each Parquet file carries metadata:

import pyarrow.parquet as pq
from litmus.environment import EnvironmentSnapshot
 
pf = pq.ParquetFile("data/runs/2026-05-16/20260516T143025Z_SN001.parquet")
metadata = pf.schema_arrow.metadata
env = EnvironmentSnapshot.model_validate_json(metadata[b"environment_json"])
print(f"Python {env.python_version}, Litmus {env.litmus_version}")

Export column naming #

When exporting runs to CSV or HDF5, input and output lane entries use input_ / output_ prefixes on the field name:

Run-level custom_metadata keys are also exported, each as a custom_<key> column (CSV).

Querying examples #

Load a run with pandas #

Measurements are nested in vector rows at rest. Use DuckDB to UNNEST them first, then load into pandas:

import duckdb
import pandas as pd
 
# UNNEST the nested measurements from vector rows and join with run context
con = duckdb.connect()
df = con.execute("""
    SELECT
        v.run_id, v.uut_serial, v.station_hostname,
        v.step_name, v.step_path, v.vector_index, v.vector_retry,
        v.step_outcome, v.vector_outcome, v.run_outcome,
        m.name  AS measurement_name,
        m.value AS measurement_value,
        m.unit  AS measurement_unit,
        m.outcome AS measurement_outcome,
        m.limit_low, m.limit_high, m.limit_nominal,
        m.uut_pin, m.instrument_name
    FROM read_parquet('data/runs/2026-05-16/20260516T143025Z_SN001.parquet') AS v,
         UNNEST(v.measurements) AS t(m)
    WHERE v.record_type = 'vector'
""").df()
 
# Step rows (direct — no UNNEST needed)
steps = pd.read_parquet(
    "data/runs/2026-05-16/20260516T143025Z_SN001.parquet"
)
steps = steps[steps["record_type"] == "step"]
 
# Failures
failures = df[df["measurement_outcome"] == "failed"]
print(failures[["step_name", "measurement_name", "measurement_value",
                "limit_low", "limit_high", "uut_pin", "instrument_name"]])

When using Litmus's Query API, the UNNEST is handled automatically — WHERE record_type = 'measurement' works as expected.

Yield by station with DuckDB (direct file — UNNEST required) #

Measurements are nested in vector rows. UNNEST them to get the flat measurement fact:

SELECT
    v.part_id,
    v.station_hostname,
    m.name AS measurement_name,
    COUNT(*) AS total,
    SUM(CASE WHEN m.outcome = 'passed' THEN 1 ELSE 0 END) AS passed,
    ROUND(100.0 * SUM(CASE WHEN m.outcome = 'passed' THEN 1 ELSE 0 END) / COUNT(*), 2) AS yield_pct
FROM read_parquet('data/runs/**/*.parquet') AS v,
     UNNEST(v.measurements) AS t(m)
WHERE v.record_type = 'vector'
GROUP BY 1, 2, 3
ORDER BY yield_pct ASC;

When querying via the Litmus Query API, the UNNEST runs automatically and WHERE record_type = 'measurement' works as-is.

Cross-run instrument-failure correlation (direct file) #

SELECT
    m.instrument_name,
    m.instrument_resource,
    COUNT(*) AS failures
FROM read_parquet('data/runs/**/*.parquet') AS v,
     UNNEST(v.measurements) AS t(m)
WHERE v.record_type = 'vector'
  AND m.outcome = 'failed'
GROUP BY 1, 2
ORDER BY failures DESC;

Slowest steps across runs #

SELECT
    step_name,
    AVG(EPOCH(step_ended_at) - EPOCH(step_started_at)) AS avg_seconds,
    COUNT(*) AS runs
FROM read_parquet('data/runs/**/*.parquet')
WHERE record_type = 'step'
  AND step_started_at IS NOT NULL
GROUP BY step_name
ORDER BY avg_seconds DESC;

ATML / IEEE 1671 alignment #

See also #

• Models — Pydantic model index + ERD
• Event types — the event-log payloads that source these rows
• Query API — how to select inputs and outputs by role and name in analysis
• Measurement traceability — how the signal-path columns get populated