Skip to content

k4bench.regression.common_mode

k4bench.regression.common_mode

Split a run group's nightly movement into a group-wide part and a per-config part.

A sweep benchmarks one detector as many configurations of itself, all on one host, in one job, against one software stack. When that shared context moves, every configuration moves with it, and the step detector — which judges each configuration independently — reports the same event once per configuration. The finding is right and the arithmetic is right; the presentation is wrong, and the volume buries whatever else the night found.

This module answers the question the run group can answer and a single series cannot: how much of tonight's move was shared? For each metric it computes one number per run, the group's common-mode shift, and divides it out of each configuration's series. What remains is that configuration's own movement.

The shift is judged as its own series rather than discarded. That distinction is the whole design:

  • Subtracting it would erase a stack-wide regression — the case where every configuration really did get slower — which is a genuine loss of sensitivity.
  • Judging it makes the detector more sensitive to that case, not less: the shift is a median over the whole group, so its noise falls by roughly sqrt(n_configs), and a shared change clears the gates sooner than it does when it has to clear them separately in every configuration.

What the reader gets is one finding for a shared move and one finding for a configuration that moved on its own, instead of the two being indistinguishable piles of the same row.

The decomposition only ever happens when the group really did move together: one configuration holding its historical level vetoes it outright (see :data:MIN_UNANIMITY). A run group is not a poll, and a majority that moved is not a common mode — on a removal sweep the majority is exactly what a detector-local regression produces.

is_common_mode 

is_common_mode(label: str) -> bool

Whether label names the run group's common mode rather than a config.

Source code in k4bench/regression/common_mode.py 
89
90
91
def is_common_mode(label: str) -> bool:
    """Whether *label* names the run group's common mode rather than a config."""
    return label == COMMON_MODE_LABEL

pretty_config 

pretty_config(label: str) -> str

Human-readable configuration label, for the one label that is not a configuration. Anything else is returned unchanged.

Source code in k4bench/regression/common_mode.py 
94
95
96
97
def pretty_config(label: str) -> str:
    """Human-readable configuration label, for the one label that is not a
    configuration. Anything else is returned unchanged."""
    return "all configs (common mode)" if is_common_mode(label) else label

format_shift 

format_shift(value: float | None) -> str

A common-mode factor as display text (×1.204), never a unit.

Source code in k4bench/regression/common_mode.py 
100
101
102
103
104
105
106
107
108
def format_shift(value: float | None) -> str:
    """A common-mode factor as display text (``×1.204``), never a unit."""
    if value is None:
        return "—"
    try:
        value = float(value)
    except (TypeError, ValueError):
        return "—"
    return "—" if value != value else f"×{value:.4g}"  # NaN-safe

common_mode_shifts 

common_mode_shifts(df: DataFrame, metric: str) -> dict[str, float]

Per-run group-wide shift factors for metric, as {run_id: factor}.

df is a run group's long-form trend frame (run_id, label, and metric), already restricted to the rows that may be judged.

Each configuration is first divided by its own median over the window, which removes the fact that configurations sit at wildly different absolute levels and leaves a series centred on 1.0. The shift for a run is then the median of those ratios across configurations — a robust estimate that a handful of configurations moving on their own cannot drag.

Normalising by a window median uses runs from both sides of each night, which would be illegitimate if this were a detection statistic. It is not: it removes a per-configuration scale, a quantity that does not depend on time, and the comparison that decides anything is always across configurations within one run.

A run appears in the result when enough configurations contributed (:data:MIN_COMMON_MODE_CONFIGS) and they agree about what happened: a shift worth speaking of has to carry every configuration with it (:data:MIN_UNANIMITY), while a run that simply sat at its usual level is recorded as the ~1.0 it measured.

A missing run is not a measured 1.0. It is the absence of a common-mode measurement — too few configurations to ask, or configurations that disagreed — and the two must not be confused, because 1.0 is a claim that the group held still. Callers divide by 1.0 for such runs (:func:residuals) and judge every series exactly as measured.

Source code in k4bench/regression/common_mode.py 
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
def common_mode_shifts(df: pd.DataFrame, metric: str) -> dict[str, float]:
    """Per-run group-wide shift factors for *metric*, as ``{run_id: factor}``.

    *df* is a run group's long-form trend frame (``run_id``, ``label``, and
    *metric*), already restricted to the rows that may be judged.

    Each configuration is first divided by its own median over the window,
    which removes the fact that configurations sit at wildly different absolute
    levels and leaves a series centred on 1.0. The shift for a run is then the
    median of those ratios across configurations — a robust estimate that a
    handful of configurations moving on their own cannot drag.

    Normalising by a window median uses runs from both sides of each night,
    which would be illegitimate if this were a detection statistic. It is not:
    it removes a per-configuration *scale*, a quantity that does not depend on
    time, and the comparison that decides anything is always across
    configurations within one run.

    A run appears in the result when enough configurations contributed
    (:data:`MIN_COMMON_MODE_CONFIGS`) and they agree about what happened: a
    shift worth speaking of has to carry every configuration with it
    (:data:`MIN_UNANIMITY`), while a run that simply sat at its usual level is
    recorded as the ~1.0 it measured.

    A missing run is not a measured 1.0. It is the absence of a common-mode
    measurement — too few configurations to ask, or configurations that
    disagreed — and the two must not be confused, because 1.0 is a claim that
    the group held still. Callers divide by 1.0 for such runs
    (:func:`residuals`) and judge every series exactly as measured.
    """
    if df is None or df.empty or metric not in df.columns:
        return {}
    sub = df[["run_id", "label", metric]].dropna()
    if sub.empty:
        return {}

    ratios = sub.copy()
    ref = ratios.groupby("label")[metric].transform("median")
    # A configuration with a zero or absent reference level carries no scale to
    # divide out, so it abstains rather than contributing an infinity.
    usable = ref.notna() & (ref > 0)
    ratios = ratios[usable]
    if ratios.empty:
        return {}
    ratios = ratios.assign(ratio=ratios[metric] / ref[usable])

    out: dict[str, float] = {}
    for run_id, group in ratios.groupby("run_id"):
        values = group["ratio"].to_numpy(dtype=float)
        values = values[np.isfinite(values)]
        if len(values) < MIN_COMMON_MODE_CONFIGS:
            continue
        shift = float(np.median(values))
        if not np.isfinite(shift) or shift <= 0:
            continue
        if abs(shift - 1.0) >= NEGLIGIBLE_SHIFT:
            # Every configuration must have moved at least MIN_UNANIMITY of the
            # way, in the same direction. One configuration still at its own
            # historical level vetoes the whole run: whatever moved the others
            # was not shared with it, so it is not common mode, and calling it
            # one would invert which configuration looks guilty.
            agreement = (values - 1.0) / (shift - 1.0)
            if agreement.min() < MIN_UNANIMITY:
                continue
        out[str(run_id)] = shift
    return out

residuals 

residuals(values: Series, run_ids: Series, shifts: dict[str, float]) -> Series

values with each run's common-mode shift divided out.

Division rather than subtraction because the shift is a ratio: a host or stack that costs 3 % more costs it in proportion, and the configurations of one sweep span a wide range of absolute levels, over which a single additive offset would mean nothing.

Source code in k4bench/regression/common_mode.py 
179
180
181
182
183
184
185
186
187
188
def residuals(values: pd.Series, run_ids: pd.Series, shifts: dict[str, float]) -> pd.Series:
    """*values* with each run's common-mode shift divided out.

    Division rather than subtraction because the shift is a *ratio*: a host or
    stack that costs 3 % more costs it in proportion, and the configurations of
    one sweep span a wide range of absolute levels, over which a single
    additive offset would mean nothing.
    """
    factors = run_ids.map(lambda rid: shifts.get(str(rid), 1.0)).astype(float)
    return values / factors

shift_history 

shift_history(shifts: dict[str, float], run_dates: dict[str, object], reliability: dict[str, bool | None], workloads: dict[str, int] | None = None) -> DataFrame

The group-wide shift as a history frame the engine can walk.

Shaped exactly like a configuration's own history — run_id, run_date, value, reliable, and the workload each run simulated — because it is one: a series centred on 1.0 whose steps are shared moves of the whole run group, judged by the same gates as everything else.

Only the runs :func:common_mode_shifts actually measured appear. A run it left out is a gap in this series, never a 1.0: "the group moved by nothing" is a measurement, and that run did not make it.

Source code in k4bench/regression/common_mode.py 
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
def shift_history(
    shifts: dict[str, float],
    run_dates: dict[str, object],
    reliability: dict[str, bool | None],
    workloads: dict[str, int] | None = None,
) -> pd.DataFrame:
    """The group-wide shift as a history frame the engine can walk.

    Shaped exactly like a configuration's own history — ``run_id``,
    ``run_date``, ``value``, ``reliable``, and the workload each run simulated
    — because it *is* one: a series centred on 1.0 whose steps are shared moves
    of the whole run group, judged by the same gates as everything else.

    Only the runs :func:`common_mode_shifts` actually measured appear. A run it
    left out is a gap in this series, never a 1.0: "the group moved by nothing"
    is a measurement, and that run did not make it.
    """
    run_ids = [rid for rid in shifts if rid in run_dates]
    history = pd.DataFrame({
        "run_id":   run_ids,
        "run_date": [run_dates[rid] for rid in run_ids],
        "value":    [shifts[rid] for rid in run_ids],
        "reliable": [reliability.get(rid) for rid in run_ids],
    })
    if workloads:
        history["workload"] = [workloads.get(rid) for rid in run_ids]
    return history