Skip to content

k4bench.regression.regions

k4bench.regression.regions

Where inside the detector a timing step landed.

A confirmed run-level regression is one number, and one number names no mechanism: "ALLEGRO got 21% slower" and "the HCAL barrel got fourteen times slower while everything else stood still" are the same measurement, but only the second can be matched against a diff. The k4BenchRegionTimingAction plugin records per-event time per top-level detector region on every run, so the second form is already measured — it has simply never been read by anything that attributes a regression.

This module reads it: for one benchmark configuration and one change window, how each region's per-event time differs between the two ends the change entered between — the two releases, or the window's two runs when one release holds both. It judges nothing (the engine has already decided that the metric stepped) and it introduces no thresholds of its own; it reports the decomposition, largest movement first, and leaves the reading to whoever asked.

Two costs shape the implementation. Region files are per configuration and hold per-event arrays, so loading a whole trend window across every label is expensive — this loads exactly the two ends of one window, for one label, and only when something actually regressed there. And a release that recorded no region file is absent, never zero: a region that appears on one side of a window only is a real event (a detector added, removed or renamed) and must stay distinguishable from one that stood still.

The same files also carry each event's wall time and how much of it was spent in stepping, which is the other half of the question: whether the step is in the typical event or in a handful of long ones (:func:region_evidence). Both readings come from one read of each file.

dirs_by_release

dirs_by_release(run_dirs: Sequence[str]) -> dict[str, list[Path]]

Group run directories by the release they measured, keyed exactly as the engine keys releases (:func:~k4bench.regression.engine.release_key), so a window's ends match the verdict that named them.

Source code in k4bench/regression/regions.py
def dirs_by_release(run_dirs: Sequence[str]) -> dict[str, list[Path]]:
    """Group run directories by the release they measured, keyed exactly as the
    engine keys releases (:func:`~k4bench.regression.engine.release_key`), so a
    window's ends match the verdict that named them."""
    grouped: dict[str, list[Path]] = {}
    for path in run_dirs:
        run_dir = Path(path)
        if not run_dir.is_dir():
            continue
        meta = parse_run_dir(run_dir)
        # `pd.NaT` is *truthy*, so an `or` here would keep the missing release
        # date instead of falling back to the run date — the same fallback
        # `x_date` makes for the frame the engine walked. Test for absence
        # explicitly, or a run predating release-date capture keys on something
        # the verdict's window never names, and its regions read as unmeasured.
        release_date = meta.get("k4h_release_date")
        if release_date is None or pd.isna(release_date):
            release_date = meta.get("run_date")
        key = release_key(release_date, run_dir.name)
        grouped.setdefault(key, []).append(run_dir)
    return grouped

window_end_dirs

window_end_dirs(run_dirs: Sequence[str], *, base_release: str, onset_release: str, base_run_id: str | None = None, onset_run_id: str | None = None) -> tuple[list[Path], list[Path]] | None

The run directories of each end of the window (base, onset], or None when either end has none.

An end is every night of its release, the nights being repeat measurements of one software state. A window whose ends name one release is two runs of it, and base_run_id / onset_run_id are what tell them apart — the same pair the verdict, the email's window token and the blame range are identified by. The ends are then those two runs rather than the release's pool, because one pool measured against itself is not a comparison. Without a resolvable run on each side there is again nothing to compare.

Source code in k4bench/regression/regions.py
def window_end_dirs(
    run_dirs: Sequence[str],
    *,
    base_release: str,
    onset_release: str,
    base_run_id: str | None = None,
    onset_run_id: str | None = None,
) -> tuple[list[Path], list[Path]] | None:
    """The run directories of each end of the window ``(base, onset]``, or
    ``None`` when either end has none.

    An end is every night of its release, the nights being repeat measurements
    of one software state. A window whose ends name one release is two runs of
    it, and *base_run_id* / *onset_run_id* are what tell them apart — the same
    pair the verdict, the email's window token and the blame range are
    identified by. The ends are then those two runs rather than the release's
    pool, because one pool measured against itself is not a comparison. Without
    a resolvable run on each side there is again nothing to compare.
    """
    grouped = dirs_by_release(run_dirs)
    base_dirs, onset_dirs = grouped.get(base_release, []), grouped.get(onset_release, [])
    if not base_dirs or not onset_dirs:
        return None
    if base_release == onset_release:
        base_dir = _run_dir_named(base_dirs, base_run_id)
        onset_dir = _run_dir_named(onset_dirs, onset_run_id)
        if base_dir is None or onset_dir is None or base_dir == onset_dir:
            return None
        return [base_dir], [onset_dir]
    return base_dirs, onset_dirs

region_evidence

region_evidence(run_dirs: Sequence[str], *, label: str, base_release: str, onset_release: str, base_run_id: str | None = None, onset_run_id: str | None = None, limit: int = MAX_REGIONS, judgeable_configs: set[tuple[str, str]] | None = None) -> tuple[tuple[RegionDelta, ...], EventProfile | None]

How each region's per-event time moved across (base, onset], largest movement first, and the per-event wall times at both ends (:class:~k4bench.regression.models.EventProfile).

Returns ((), None) when either end recorded no region timing at all — with only one side measured there is no comparison to make, and inventing one (treating the missing side as zero) would report every region of the detector as newly appearing. The profile is likewise None unless both ends recorded per-event wall times. When judgeable_configs is supplied, failed or orphaned config-nights absent from that set are gaps and do not enter either end.

The ends are those of :func:window_end_dirs: for a window inside one release its two runs, whose pool measured against itself would report every region as having stood still.

Source code in k4bench/regression/regions.py
def region_evidence(
    run_dirs: Sequence[str],
    *,
    label: str,
    base_release: str,
    onset_release: str,
    base_run_id: str | None = None,
    onset_run_id: str | None = None,
    limit: int = MAX_REGIONS,
    judgeable_configs: set[tuple[str, str]] | None = None,
) -> tuple[tuple[RegionDelta, ...], EventProfile | None]:
    """How each region's per-event time moved across ``(base, onset]``, largest
    movement first, and the per-event wall times at both ends
    (:class:`~k4bench.regression.models.EventProfile`).

    Returns ``((), None)`` when either end recorded no region timing at all —
    with only one side measured there is no comparison to make, and inventing
    one (treating the missing side as zero) would report every region of the
    detector as newly appearing. The profile is likewise ``None`` unless both
    ends recorded per-event wall times. When *judgeable_configs* is supplied,
    failed or orphaned config-nights absent from that set are gaps and do not
    enter either end.

    The ends are those of :func:`window_end_dirs`: for a window inside one
    release its two runs, whose pool measured against itself would report every
    region as having stood still.
    """
    ends = window_end_dirs(
        run_dirs, base_release=base_release, onset_release=onset_release,
        base_run_id=base_run_id, onset_run_id=onset_run_id,
    )
    if ends is None:
        return (), None
    base_dirs, onset_dirs = ends

    base_nights = _read_nights(base_dirs, label, judgeable_configs)
    onset_nights = _read_nights(onset_dirs, label, judgeable_configs)
    base = _release_medians(base_nights)
    onset = _release_medians(onset_nights)
    if not base or not onset:
        return (), None

    deltas = []
    for region in sorted(set(base) | set(onset)):
        before, after = base.get(region), onset.get(region)
        # A region present on one side only genuinely appeared or disappeared;
        # its whole time is the movement, and saying so is the point.
        delta = (after or 0.0) - (before or 0.0)
        if not math.isfinite(delta):
            continue
        deltas.append(RegionDelta(region=region, base=before, onset=after, delta=delta))
    deltas.sort(key=lambda d: (-abs(d.delta), d.region))

    base_sample, onset_sample = _event_sample(base_nights), _event_sample(onset_nights)
    profile = None
    if base_sample is not None and onset_sample is not None:
        events = dict.fromkeys(
            e.event for e in (*base_sample.longest, *onset_sample.longest)
        )
        matched = sorted(
            (
                MatchedEvent(
                    event=event,
                    base=_event_time(base_nights, event),
                    onset=_event_time(onset_nights, event),
                )
                for event in events
            ),
            key=lambda m: (-max(m.base or 0.0, m.onset or 0.0), m.event),
        )
        profile = EventProfile(
            base=base_sample, onset=onset_sample, matched=tuple(matched),
        )
    return tuple(deltas[:limit]), profile

region_deltas

region_deltas(run_dirs: Sequence[str], *, label: str, base_release: str, onset_release: str, base_run_id: str | None = None, onset_run_id: str | None = None, limit: int = MAX_REGIONS, judgeable_configs: set[tuple[str, str]] | None = None) -> tuple[RegionDelta, ...]

The region half of :func:region_evidence alone.

Source code in k4bench/regression/regions.py
def region_deltas(
    run_dirs: Sequence[str],
    *,
    label: str,
    base_release: str,
    onset_release: str,
    base_run_id: str | None = None,
    onset_run_id: str | None = None,
    limit: int = MAX_REGIONS,
    judgeable_configs: set[tuple[str, str]] | None = None,
) -> tuple[RegionDelta, ...]:
    """The region half of :func:`region_evidence` alone."""
    deltas, _profile = region_evidence(
        run_dirs, label=label,
        base_release=base_release, onset_release=onset_release,
        base_run_id=base_run_id, onset_run_id=onset_run_id,
        limit=limit, judgeable_configs=judgeable_configs,
    )
    return deltas