Parquet Storage Schema #

Each TesterKit run produces one Parquet file. The file has two layers: the at-rest format and the query projection.

At-rest — three row types. Every row carries an explicit record_type discriminator with one of three values:

• record_type = 'run' — exactly one row per file. Carries run-level identity, timing, outcome, plus UUT / station / project / git / environment context.
• record_type = 'step' — one row per step execution (per step_path, per retry). Carries the step's identity, timing, and rolled-up outcome — and, when the step measures directly, its own inputs / outputs / measurements. vector_index is always NULL on a step row; if the step is nested inside a parent loop, the iteration it ran under is carried by vector_outer_index.
• record_type = 'vector' — a condition point: one row per iteration of a sweep or in-body loop (the vectors fixture / run_vector). Present only when a step loops — a step that measures directly emits no vector rows. Each vector row carries that point's own inputs / outputs / measurements, with vector_index its 0-based position in the loop.

Measurements are nested, not rows. Measurements live in a measurements column — a typed nested list (LIST<STRUCT>) — on whichever row owns them: the step row for a directly-measured step, or the vector row for a swept one. Each struct holds name, value, unit, outcome, timestamp, limit_*, characteristic_id, spec_ref, and signal-path fields (uut_pin, fixture_connection, instrument_*). There is no at-rest record_type = 'measurement' row.

Query projection — four virtual types. The query projection UNNESTs the nested measurements from both step and vector rows into a flat fact and presents a fourth virtual row type record_type = 'measurement' in query results. All WHERE record_type = 'measurement' queries target this projected view, not the at-rest file. The inputs / outputs lanes (from either row type) are also projected into two separate EAV tables, inputs and outputs (the table name IS the role — no role column), for query-time access. Query output shape is byte-stable regardless of at-rest format changes.

Run identity lives once. The query projection's derived tables each carry only their own grain's columns plus a run_id foreign key — run identity (UUT / station / part / project / git / environment context) lives once, in the runs table, and every other query view (steps, measurements, instruments) joins it back in. Reads always see the same full column set as before; only the storage underneath is normalized.

This page mirrors the canonical at-rest schema; the column names and types here match what read_parquet returns.

File layout #

<data_dir>/runs/{date}/
├── {timestamp}_{run_id8}_{serial}.parquet  # Run + step + vector rows; measurements nested in the step or vector row that owns them
├── {timestamp}_{run_id8}.parquet           # Same shape, no UUT serial (dev runs)
└── {timestamp}_{run_id8}_{serial}_ref/     # Reference data (waveforms, images, files)
    ├── {vector_id}_scope_waveform.npz
    ├── {vector_id}_camera_image.png
    └── ...

Timestamps are UTC and sort naturally. The 8-char run_id sits right after the timestamp (the trailing serial is optional, so its absence never shifts the leading parts) and disambiguates two runs of the same serial that start in the same second. DuckDB / Spark / Polars / Pandas all read the file directly with read_parquet.

Discriminator #

At-rest row types (three):

• run — one row per run; run-level metadata (start/end timestamps, UUT serial, station, outcome).
• step — one row per step execution, keyed (step_path, step_retry, vector_outer_index); code identity, timing, rolled-up outcome, and the step's own inputs / outputs / measurements when it measures directly.
• vector — a condition point: one row per sweep / in-body-loop iteration, keyed (step_path, vector_outer_index, vector_index, vector_retry) (it also carries the enclosing step_retry). vector_outer_index is NULL for a top-level loop and set only when the loop is nested inside a parent loop. Present only when a step loops. Carries that point's inputs / outputs lane columns and a nested measurements list.

Query projection (virtual fourth type):

• measurement — the query projection UNNESTs the nested measurements list from both step and vector rows into flat fact rows stamped record_type = 'measurement'. These rows exist in query results but not in the at-rest file.

To list steps: WHERE record_type = 'step'. To list vectors: WHERE record_type = 'vector'. To list measurements: WHERE record_type = 'measurement'. All kinds: omit the filter.

Identity & timing #

Who — operator #

What — UUT #

What — part spec #

Where — station #

Where — fixture #

Where — instruments (instruments lane — at-rest format) #

Instrument identity and calibration traceability for the run, captured from the pytest fixtures the test used. Stored at rest in the instruments column as a typed nested list — LIST<STRUCT> — one struct per instrument. Every row carries the column; the run row holds the full inventory.

Entry structure (one item in the instruments list):

For real hardware, identity comes from *IDN? at session start. For mock instruments, identity comes from the instrument YAML configs.

The Query API materializes these structs into the instruments table (one row per instrument per run) at ingest, so per-instrument queries select named columns instead of unnesting the nested list:

-- DuckDB: one row per instrument per run, via the materialized table
SELECT name, serial_number, cal_due, count(DISTINCT run_id) AS runs
FROM instruments
GROUP BY name, serial_number, cal_due;

To read the nested list directly off the parquet (e.g. an external lakehouse ingest):

-- DuckDB: unnest the instruments struct off the run row
SELECT
    i.name AS instrument,
    i.serial_number AS serial,
    i.cal_due AS cal_due
FROM read_parquet('data/runs/**/*.parquet'), UNNEST(instruments) AS t(i)
WHERE record_type = 'run';

Test context #

Input conditions (inputs lane — at-rest format) #

At rest, a step's commanded conditions — or, for a swept iteration, a vector's — are stored in that row's inputs column as a typed nested list: LIST<STRUCT<name, value_type, value_int, value_double, value_bool, value_text, value_timestamp, value_json, unit, uut_pin>>. One struct per parameter; value_type selects which value_* field holds the actual value.

The Query API projects these lane structs into the inputs EAV table (keyed by name — the table IS the role) for query-time access. See Query API for how to select input fields in analysis.

Entry structure (one item in the inputs list):

Naming convention (applies to name inside each entry):

Bare names are spec-relevant for condition matching; prefixed names are stimulus/settings. Convention is enforced by docs, not by the writer.

Stimulus signal-path sub-fields for each param (also stored in the inputs lane as separate entries with compound names):

Observations (outputs lane — at-rest format) #

Observations are measured context — readings captured during the test, not commanded values. Stored at rest in the outputs column with the same LIST<STRUCT> shape as inputs. The Query API projects these into the outputs EAV table.

Each struct entry encodes one observation under name. Non-scalar payloads route to the _ref/ sibling directory and are stored as file:// URIs with value_type = 'uri' in the value_text field:

from testerkit.data.backends.parquet import load_file, is_file_reference
 
if is_file_reference(column_value):
    data = load_file(parquet_path, column_value)

Measurement fields (projected from nested struct) #

At rest, measurements live in the owning row's measurements column — the step row for a direct measurement, the vector row for a swept one — as a LIST<STRUCT>. The fields below are exposed as flat columns on the projected record_type = 'measurement' rows the Query API surfaces.

Limits (on projected record_type='measurement' rows) #

Spec traceability #

-- Yield by characteristic across all parts
SELECT characteristic_id, part_id,
       AVG(CASE WHEN measurement_outcome='passed' THEN 1.0 ELSE 0.0 END) AS yield
FROM read_parquet('data/runs/**/*.parquet')
WHERE record_type = 'measurement'
GROUP BY characteristic_id, part_id;

Measurement signal path #

Rollup outcomes #

Environment traceability #

Custom metadata #

Test code can add arbitrary run-level metadata via run_context.set():

def test_example(run_context, psu, dmm, verify):
    run_context.set("operator_badge", "EMP-12345")
    run_context.set("fixture_serial", "FIX-001")
    run_context.set("ambient_temp", 23.5)
    ...

At rest, custom metadata is stored as a JSON blob in the Parquet file-level metadata under the key custom_metadata — not as a column in the row data. It is run-scoped (one blob per file, not one entry per measurement).

inputs / outputs EAV tables (query projection) #

The Query API projects all inputs and outputs lane entries into two long EAV tables, named inputs and outputs — the table name IS the role, so there is no role column. This is what the Query API reads when you select inputs or outputs by name.

Querying these tables directly is rarely needed — use the Query API (FieldRef.input("vin"), FieldRef.output("v_rail")) which joins the right one for you and handles type coherence (fails loud if a name carries mixed value_types in scope; auto-resolves when unambiguous).

Projection tables are a normalized snowflake #

The query projection is a snowflake: each derived table (runs, steps, vectors, measurements, inputs, outputs, instruments) stores only its own grain's columns plus the foreign key to its parent. Run identity lives once in runs; step code/timing lives once in steps; the swept condition points live in vectors; the flat measurement fact and the inputs/outputs lanes carry only their payload plus the coordinate key that reaches their step/vector. The measurements / steps / step_vectors / instruments views JOIN back up the hierarchy to reconstruct the wide, denormalized row shape — so every query and its results are unchanged; only the storage underneath is normalized (no denormalized copy that can drift). This is a derived cache: it rebuilds from the parquet on boot and carries no schema_version of its own.

Outcome values #

Comparator values #

Retries #

All retries are stored. Each retry produces a new row at that execution's grain — a fresh step row for a directly-measured step, or a fresh vector row for a swept iteration — so measurement-less retries are captured too. Measurements stay nested on the retried row.

Mode-1 (parametrize / single / unswept): each attempt is a separate step row carrying its own nested measurements; step_retry increments per attempt. There are no vector rows. In the query projection, the UNNESTed measurement rows carry the same step_retry as their enclosing step.

record_type | step_retry | step_outcome | measurement_name | measurement_value
step        | 0          | failed       | —                | —                  ← first attempt (measurements nested on the step row)
measurement | 0          | —            | output_voltage   | 3.50               ← projected (query-time UNNEST)
step        | 1          | failed       | —                | —                  ← first retry
measurement | 1          | —            | output_voltage   | 3.48               ← projected
step        | 2          | passed       | —                | —                  ← second retry
measurement | 2          | —            | output_voltage   | 3.30               ← projected

Mode-2 (vectors fixture): each attempt at one in-body iteration is a separate vector row with the same vector_index and an incremented vector_retry.

Filter to the final attempt with a window over the retry axis — for step-grained rows, the max step_retry per (run_id, step_path); for swept rows, the max vector_retry per (run_id, step_path, vector_index).

File-level metadata #

Beyond columns, each Parquet file carries metadata:

import pyarrow.parquet as pq
from testerkit.environment import EnvironmentSnapshot
 
pf = pq.ParquetFile("data/runs/2026-05-16/20260516T143025Z_SN001.parquet")
metadata = pf.schema_arrow.metadata
env = EnvironmentSnapshot.model_validate_json(metadata[b"environment_json"])
print(f"Python {env.python_version}, TesterKit {env.testerkit_version}")

Export column naming #

When exporting runs to CSV or HDF5, input and output lane entries use input_ / output_ prefixes on the field name:

Run-level custom_metadata keys are also exported, each as a custom_<key> column (CSV).

Querying examples #

Load a run with pandas #

Measurements are nested in step (and vector) rows at rest. Use DuckDB to UNNEST them first, then load into pandas:

import duckdb
import pandas as pd
 
# Measurements are nested on step rows (direct) and vector rows (swept).
# UNNEST from both, joined with each row's run/step context.
con = duckdb.connect()
df = con.execute("""
    SELECT
        v.run_id, v.uut_serial_number, v.station_hostname,
        v.step_name, v.step_path, v.vector_index,
        v.step_retry, v.vector_retry,
        v.step_outcome, v.vector_outcome, v.run_outcome,
        m.name  AS measurement_name,
        m.value AS measurement_value,
        m.unit  AS measurement_unit,
        m.outcome AS measurement_outcome,
        m.limit_low, m.limit_high, m.limit_nominal,
        m.uut_pin, m.instrument_name
    FROM read_parquet('data/runs/2026-05-16/20260516T143025Z_SN001.parquet') AS v,
         UNNEST(v.measurements) AS t(m)
    WHERE v.record_type IN ('step', 'vector')
""").df()
 
# Failures
failures = df[df["measurement_outcome"] == "failed"]
print(failures[["step_name", "measurement_name", "measurement_value",
                "limit_low", "limit_high", "uut_pin", "instrument_name"]])

When using TesterKit's Query API, the UNNEST is handled automatically — WHERE record_type = 'measurement' works as expected.

Yield by station with DuckDB (direct file — UNNEST required) #

Measurements are nested in step (and vector) rows. UNNEST them to get the flat measurement fact:

SELECT
    v.part_id,
    v.station_hostname,
    m.name AS measurement_name,
    COUNT(*) AS total,
    SUM(CASE WHEN m.outcome = 'passed' THEN 1 ELSE 0 END) AS passed,
    ROUND(100.0 * SUM(CASE WHEN m.outcome = 'passed' THEN 1 ELSE 0 END) / COUNT(*), 2) AS yield_pct
FROM read_parquet('data/runs/**/*.parquet') AS v,
     UNNEST(v.measurements) AS t(m)
WHERE v.record_type IN ('step', 'vector')
GROUP BY 1, 2, 3
ORDER BY yield_pct ASC;

When querying via the TesterKit Query API, the UNNEST runs automatically and WHERE record_type = 'measurement' works as-is.

Cross-run instrument-failure correlation (direct file) #

SELECT
    m.instrument_name,
    m.instrument_resource,
    COUNT(*) AS failures
FROM read_parquet('data/runs/**/*.parquet') AS v,
     UNNEST(v.measurements) AS t(m)
WHERE v.record_type = 'vector'
  AND m.outcome = 'failed'
GROUP BY 1, 2
ORDER BY failures DESC;

Slowest steps across runs #

SELECT
    step_name,
    AVG(EPOCH(step_ended_at) - EPOCH(step_started_at)) AS avg_seconds,
    COUNT(*) AS runs
FROM read_parquet('data/runs/**/*.parquet')
WHERE record_type = 'step'
  AND step_started_at IS NOT NULL
GROUP BY step_name
ORDER BY avg_seconds DESC;

ATML / IEEE 1671 alignment #

See also #

• Models — Pydantic model index + ERD
• Event types — the event-log payloads that source these rows
• Query API — how to select inputs and outputs by role and name in analysis
• Measurement traceability — how the signal-path columns get populated