k4bench.regression.models

k4bench.regression.models

Dataclasses and enums for the nightly regression report.

The severity and direction axes are kept separate rather than cross-producted into one enum: severity says how much attention a metric deserves, direction says which way it moved. Neither carries a good/bad judgment — direction is a plain sign, not an evaluation, since a step in either direction can equally be an optimization, a deliberate change, or a bug.

Severity

Bases: str, Enum

How much attention a metric verdict deserves.

OK — inside the baseline's normal variation. WATCH — first night crossing both detection gates; shown in the report but not alerted on (see the two-strike rule in :mod:k4bench.regression.engine). CONFIRMED — crossed both gates on two consecutive reliable nights. FAILURE — hard job failure (non-zero returncode / missing run); bypasses confirmation and always alerts immediately. UNKNOWN — not enough reliable history to judge; never a flag, mirroring reliability.py's "no evidence ⇒ no verdict".

Direction

Bases: str, Enum

Which way a flagged metric moved. NONE for OK/UNKNOWN/FAILURE.

Purely the mechanical sign of the change — UP/DOWN make no claim about whether the move is desirable. A metric going down is not an "improvement" any more than one going up is a "regression" in the colloquial sense: either can be a deliberate change, an optimization, or a bug, and the report leaves that call to a human.

SeriesId dataclass

SeriesId(detector: str, platform: str, sample: str, label: str, metric_family: str, metric: str, sub_detector: str | None = None)

Identity of one metric history: the axes that must never be pooled.

A (detector, platform, sample) triple is one independent run group (different platforms/samples have independent baselines); label is the benchmark config within the run, metric the column evaluated, and sub_detector the region name for region-level metrics only. metric_family selects the practical-effect floor in the engine.

HostFact dataclass

HostFact(name: str, cpu_cores: int | None = None)

The machine a release's nights were benchmarked on.

Recorded alongside the measurement because a benchmark host is part of what produced the number: a step that lands exactly where the runs moved to a different machine — or to the same machine with a different core count — has an explanation that no code change competes with. The nightly reliability check already rejects a contended host, but a perfectly healthy host that is simply a different one is invisible to it, and that is the case this carries.

cpu_cores is the logical core count, None when the run recorded no machine info.

RegionDelta dataclass

RegionDelta(region: str, base: float | None, onset: float | None, delta: float)

How much one sub-detector region's per-event time moved across a change window.

A run-level step is a single number, and a single number names no mechanism: "ALLEGRO got 21% slower" and "the HCAL barrel got 14× slower while everything else stood still" are the same measurement, but only the second can be matched against a diff. The region timing that answers this is recorded by the k4BenchRegionTimingAction plugin on every run; this is that decomposition, computed across the two releases the change entered between.

base/onset are the per-event median times (seconds) on each end of the window, None when that end recorded nothing for the region — a region that only appears on one side is a real event (a detector added or removed) and must stay distinguishable from one that stayed flat. delta is the signed move, and is what the regions are ranked by.

ReleasePoint dataclass

ReleasePoint(run_date: str, value: float | None, n_runs: int = 0, n_judged: int = 0, severity: Severity = UNKNOWN, direction: Direction = NONE, hosts: tuple[HostFact, ...] = ())

One Key4hep release in a metric's recent history.

The unit is the release, not the night, because that is the unit the engine judges on: nights sharing a run_date are repeat measurements of one software state, and rendering them as separate data points would show a stable metric wobbling under a stack that never changed.

value is the median of the release's judged nights — the same values the engine's own release-median rule reads — falling back to whatever the release recorded when none of its nights could be judged. n_runs and n_judged are what makes that fallback readable rather than misleading: n_judged=0 says the level shown was never assessed (an unreliable host, or a series still warming up), which is a different fact from a release that was measured and found flat. A release that recorded nothing at all is not a point here — a gap in the tail is a gap, never a zero.

severity/direction are the engine's own read of the release (its worst night, and which way that night moved), so a history tail carries not just the levels but which of them were flagged at the time — the difference between a series that has stepped once and one that trips every other week.

hosts names the machines that produced the release's nights (see :class:HostFact) — normally one, and more only when a release was measured on several.

MetricVerdict dataclass

MetricVerdict(detector: str, platform: str, sample: str, label: str, metric_family: str, metric: str, sub_detector: str | None, run_id: str, run_date: str, value: float | None, baseline_median: float | None, baseline_mad: float | None, pct_change: float | None, z_score: float | None, severity: Severity, direction: Direction, reason: str, onset_run_id: str | None = None, onset_run_date: str | None = None, last_accepted_run_id: str | None = None, last_accepted_run_date: str | None = None, first_confirmed_run_id: str | None = None, history: tuple[ReleasePoint, ...] = (), region_deltas: tuple[RegionDelta, ...] = ())

The engine's judgement of one metric on one night.

run_id is the nightly run directory name; run_date is the Key4hep release the run measured, not the date it ran. The two differ often — the nightly build does not publish every day, so consecutive runs frequently re-measure one release. Anything correlating a verdict with the software stack must key on run_date.

onset_* and last_accepted_* bound the window a CONFIRMED change entered in, and are None on every other severity. Confirmation is a two-strike rule, so the night a change is reported is one reliable night after the night it first appeared: onset_* identifies that first night, and last_accepted_* the newest night before it observed at the then-accepted level. The change therefore landed in (last_accepted, onset] — the interval to search for a cause. last_accepted_* is None when no such night exists (a change confirmed before the series ever settled), which makes the window open-ended rather than empty.

first_confirmed_run_id names the night a CONFIRMED change was first confirmed for its release (None on every other severity). It equals run_id on that first night; on a later night of the same release re-confirming the change it points back — letting the report and email render a repeat as a repeat rather than fresh news.

flagged property

flagged: bool

True for anything worth a row in the report (not OK/UNKNOWN).

is_reconfirmed property

is_reconfirmed: bool

True when a later nightly measurement of the same Key4hep release re-confirms a change already confirmed for that release.

The engine resets its confirmation state at a release boundary, so a first_confirmed_run_id that both exists and differs from this night's run_id can only mean the same release was measured again — never a different release still tripping the old baseline. A legacy verdict with no first_confirmed_run_id is therefore not reconfirmed (it is treated as New).

is_new_confirmation property

is_new_confirmation: bool

True for the first confirmation of a change for the release currently being measured — a confirmed verdict that is not a same-release reconfirmation (see :attr:is_reconfirmed).

RunGroupReport dataclass

RunGroupReport(detector: str, platform: str, sample: str, k4h_release: str, run_date: str, run_id: str, verdicts: list[MetricVerdict] = list(), job_failures: list[str] = list(), notes: list[str] = list(), reliable: bool | None = None, github_run_url: str | None = None, geometry_path: str = '')

All verdicts for one (detector, platform, sample) triple for the night.

job_failures carries hard, group-level problems that have no metric series to attach to (e.g. no run uploaded for tonight at all, or a config that produced no results). notes carries non-alertable context (e.g. tonight's run failed the reliability check, so metrics were not judged). reliable is tonight's host-reliability tri-state (the same per-run verdict as :func:k4bench.results.reliability_evidence.run_reliability_map; None = no evidence), persisted so report consumers — e.g. the dashboard's Overview tab — can apply the standard unreliable-run filter without re-downloading run data. github_run_url is tonight's benchmarking job's own CI run (from run_info.json), distinct from whatever CI run generated the report itself.

regressions property

regressions: list[MetricVerdict]

Confirmed regressions — a "regression" here means any confirmed step beyond the baseline, either direction; nothing is judged good or bad, only that it moved beyond the baseline twice in a row.

new_regressions property

new_regressions: list[MetricVerdict]

Confirmed regressions first confirmed for the release being measured (see :attr:MetricVerdict.is_new_confirmation).

reconfirmed_regressions property

reconfirmed_regressions: list[MetricVerdict]

Confirmed regressions that are a later measurement of the same release re-confirming an earlier confirmation (see :attr:MetricVerdict.is_reconfirmed).

NightlyReport dataclass

NightlyReport(generated_at: str, groups: list[RunGroupReport] = list())

One night's verdicts across every run group found on EOS.

regressions property

regressions: list[MetricVerdict]

Confirmed regressions across all groups, either direction.

new_regressions property

new_regressions: list[MetricVerdict]

New confirmations across all groups (first confirmation for the release being measured).

reconfirmed_regressions property

reconfirmed_regressions: list[MetricVerdict]

Same-release reconfirmations across all groups.

failures property

failures: list[MetricVerdict]

Per-config hard failures across all groups.

job_failures property

job_failures: list[tuple[RunGroupReport, str]]

Group-level hard failures (e.g. missing run), with their group.

has_alertable property

has_alertable: bool

True when the night warrants an alert email: any confirmed regression or any hard failure. WATCHes never alert.

report_night property

report_night: str

The nightly date this report covers (newest run across groups).

by_detector

by_detector() -> dict[str, list[RunGroupReport]]

Group the run groups by detector (a detector can have several (platform, sample) groups), preserving insertion order.

Source code in k4bench/regression/models.py

def by_detector(self) -> dict[str, list[RunGroupReport]]:
    """Group the run groups by detector (a detector can have several
    ``(platform, sample)`` groups), preserving insertion order."""
    out: dict[str, list[RunGroupReport]] = {}
    for g in self.groups:
        out.setdefault(g.detector, []).append(g)
    return out