k4bench.analysis.event_drivers

k4bench.analysis.event_drivers

What an event's cost goes with: per-event costs set against what each event holds.

An event's cost (its time, an algorithm's time, instructions or allocations, the time and steps simulation spent in a detector) is a :class:Measure, one value per event. What the event holds is its row of an :class:EventTable: the canned quantities of :data:FEATURES, computed from its generator particles (:mod:k4bench.analysis.event_source), and what each collection holds in it (:mod:k4bench.analysis.event_size), one column each under a name an expression (:mod:k4bench.analysis.expression) can use:

• n_<collection>, E_<collection>: the elements and calorimeter energy of the event itself, as the job read it: simulation's output, or reconstruction's input;
• out_n_<collection>, out_E_<collection>: reconstruction's outputs.

A name that would collide with another gets its file's name appended, then a number; :attr:Column.collection and :attr:Column.file say what it is.

Matching events

A table's rows are the entries of the files it was read from; a measure's values are keyed by the event number the job's profiling counted. The two match when the profiling counted exactly the entries 0 .. n-1, which is what :func:align checks; when they do not (a filtered output, an event range not from the start), it says so and nothing is matched. The files' rows are matched entry by entry: :func:event_table refuses files whose EventHeaders name different events at the same entry (a reordered output), as far as they have EventHeaders. The warm-up event 0 is left out of every measure, as in the other per-event views.

What the numbers do and do not say

:func:associations ranks the columns by Spearman's rank correlation with a measure, so a cost that rises with a quantity in any monotonic way ranks high and one with a symmetric or periodic dependence (on phi, or on theta for a forward-backward symmetric cost) ranks low; the latter take a scatter, or abs_cos_theta. The top of a ranking picked from dozens of columns is an association to look at, not a significance: no threshold is drawn. :func:trend fits one straight line, robustly, and the events well above it are above that line, not necessarily expensive for their content as a whole.

Feature dataclass

Feature(name: str, label: str, expression: str, ranked: bool = True)

A canned per-event quantity of the generator particles.

Column dataclass

Column(label: str, kind: str, ranked: bool = True, collection: str | None = None, file: str | None = None, expression: str | None = None)

One column of an :class:EventTable.

EventTable dataclass

EventTable(frame: DataFrame, columns: Mapping[str, Column], source: EventSource | None = None, left_out: Mapping[str, str] = dict(), _particles: dict = dict())

What each event holds: one row per entry, one column per quantity.

evaluate

evaluate(expression: str) -> Series

expression in each event (:func:k4bench.analysis.expression.evaluate).

Source code in k4bench/analysis/event_drivers.py

def evaluate(self, expression: str) -> pd.Series:
    """*expression* in each event (:func:`k4bench.analysis.expression.evaluate`)."""
    particles = self.particles()
    return evaluate(
        expression, self.frame, particles,
        self.source.event_index if particles is not None else None,
    )

Measure dataclass

Measure(name: str, metric: str, unit: str, values: Series, profiled: tuple[int, ...])

One per-event cost of one configuration.

Trend dataclass

Trend(slope: float, intercept: float)

A straight line through a measure against one quantity: Theil–Sen's slope, and the median intercept.

event_table

event_table(source: EventSource | None, content: Sequence[OutputFile] = (), outputs: Sequence[OutputFile] = ()) -> EventTable | None

The table of a run's events: :data:FEATURES from source, and each collection's elements and energy from content (simulation's output file) or the source's input content, and from outputs (reconstruction's files), each measured file for as many events as the others; None when there is nothing to tabulate.

Each output is checked against source on its own and left out, with why (:attr:EventTable.left_out), when it holds other events: another number of them, other EventHeaders, or an entry without one, its or the input's, which leaves its order unchecked.

Raises ValueError for outputs without source, since only the input's EventHeaders show that the outputs hold the events in the order the job read them, and when the source and content disagree on the number of events or their EventHeaders on which event an entry is.

Source code in k4bench/analysis/event_drivers.py

def event_table(
    source: EventSource | None,
    content: Sequence[OutputFile] = (),
    outputs: Sequence[OutputFile] = (),
) -> EventTable | None:
    """The table of a run's events: :data:`FEATURES` from *source*, and each
    collection's elements and energy from *content* (simulation's output
    file) or the source's input content, and from *outputs*
    (reconstruction's files), each measured file for as many events as the
    others; ``None`` when there is nothing to tabulate.

    Each output is checked against *source* on its own and left out, with
    why (:attr:`EventTable.left_out`), when it holds other events: another
    number of them, other EventHeaders, or an entry without one, its or the
    input's, which leaves its order unchecked.

    Raises ``ValueError`` for *outputs* without *source*, since only the
    input's EventHeaders show that the outputs hold the events in the order
    the job read them, and when the source and *content* disagree on the
    number of events or their EventHeaders on which event an entry is.
    """
    if source is None and any(_measured(f) for f in outputs):
        raise ValueError(
            "the job's input was not recorded, so nothing shows that its outputs "
            "hold the events in the order it read them"
        )
    left_out = {
        f.name: why for f in outputs if _measured(f) and (why := _unmatched(f, source))
    }
    outputs = [f for f in outputs if f.name not in left_out]
    measured = [f for f in (*content, *outputs) if _measured(f)]
    sizes = {
        n for n in (
            source.events if source is not None else None,
            *(f.events for f in measured),
        ) if n is not None
    }
    if not sizes:
        return None
    if len(sizes) > 1:
        raise ValueError(f"the event files disagree on the number of events: {sorted(sizes)}")
    _check_same_events([
        *([(source.file, source.event_numbers, source.run_numbers)] if source is not None else []),
        *((f.name, f.event_numbers, f.run_numbers) for f in measured),
    ])
    index = pd.RangeIndex(sizes.pop(), name="entry")
    # Gathered first and framed at once: a frame grown a column at a time
    # fragments.
    values: dict[str, np.ndarray] = {}
    columns: dict[str, Column] = {}
    taken = {f.name for f in (*FEATURES, *PRIMARY_FEATURES)}

    def add(name: str, data, column: Column) -> None:
        values[name] = np.asarray(data, dtype=float)
        columns[name] = column
        taken.add(name)

    if source is not None:
        _add_content(add, taken, "", source.input_counts, source.input_energies, source.file,
                     "content")
    for output in content:
        if _measured(output):
            _add_content(add, taken, "", output.event_counts(), output.event_energies(),
                         output.name, "content")
    for output in outputs:
        if _measured(output):
            _add_content(add, taken, "out_", output.event_counts(), output.event_energies(),
                         output.name, "output")

    frame = pd.DataFrame(values, index=index)
    table = EventTable(frame, columns, source, left_out)
    if source is not None:
        features = list(FEATURES)
        if _one_primary(table):
            features += PRIMARY_FEATURES
        truth = pd.DataFrame(
            {f.name: table.evaluate(f.expression).to_numpy() for f in features}, index=index,
        )
        frame = pd.concat([frame, truth], axis=1)
        columns = {
            **{f.name: Column(f.label, "truth", f.ranked, expression=f.expression) for f in features},
            **columns,
        }
        table = EventTable(frame, columns, source, left_out)
    return table

alias

alias(prefix: str, collection: str, file: str, taken: set[str]) -> str

The expression name of collection of file under prefix: its name with every character that cannot be in a name made _, then, if that is taken, the file's name appended, then a number.

Source code in k4bench/analysis/event_drivers.py

def alias(prefix: str, collection: str, file: str, taken: set[str]) -> str:
    """The expression name of *collection* of *file* under *prefix*: its
    name with every character that cannot be in a name made ``_``, then, if
    that is *taken*, the file's name appended, then a number."""
    name = _identifier(prefix + collection)
    if name in taken:
        name = _identifier(f"{name}_{PurePath(file).name.split('.')[0]}")
    base, number = name, 2
    while name in taken:
        name, number = f"{base}_{number}", number + 1
    return name

simulation_measures

simulation_measures(event_data: DataFrame | None, regions: Mapping | None) -> list[Measure]

The per-event costs of one simulation configuration: its event time (event_data, :func:~k4bench.analysis.loader.load_event_timing's frame) and the time and steps spent in each detector (regions, its :func:~k4bench.analysis.loader.load_region_timing entry), the costliest detectors of each first.

Source code in k4bench/analysis/event_drivers.py

def simulation_measures(event_data: pd.DataFrame | None, regions: Mapping | None) -> list[Measure]:
    """The per-event costs of one simulation configuration: its event time
    (*event_data*, :func:`~k4bench.analysis.loader.load_event_timing`'s frame)
    and the time and steps spent in each detector (*regions*, its
    :func:`~k4bench.analysis.loader.load_region_timing` entry), the
    costliest detectors of each first."""
    measures: list[Measure] = []
    if event_data is not None and not event_data.empty:
        times = event_data.set_index("event_number")["event_time_s"]
        measures.append(Measure(WHOLE_EVENT, "time", "s", _without_warm_up(times),
                                tuple(int(e) for e in times.index)))
    if regions:
        at_location = regions.get("at_location")
        steps = regions.get("steps")
        if at_location is not None and not at_location.empty:
            profiled = tuple(int(e) for e in at_location.index)
            for detector in _costliest_first(at_location):
                measures.append(Measure(str(detector), "time", "s",
                                        _without_warm_up(at_location[detector]), profiled))
            if steps is not None:
                for detector in _costliest_first(steps):
                    measures.append(Measure(str(detector), "steps", "",
                                            _without_warm_up(steps[detector]),
                                            tuple(int(e) for e in steps.index)))
    return measures

reconstruction_measures

reconstruction_measures(event_data: DataFrame | None, timing: ComponentTiming | None) -> list[Measure]

The per-event costs of one reconstruction configuration: its event time (event_data) and, per algorithm, each of :data:MEASURE_METRICS its auditor file (timing) recorded, the costliest algorithms of each first.

Source code in k4bench/analysis/event_drivers.py

def reconstruction_measures(
    event_data: pd.DataFrame | None, timing: ComponentTiming | None,
) -> list[Measure]:
    """The per-event costs of one reconstruction configuration: its event
    time (*event_data*) and, per algorithm, each of
    :data:`MEASURE_METRICS` its auditor file (*timing*) recorded, the
    costliest algorithms of each first."""
    measures: list[Measure] = []
    if event_data is not None and not event_data.empty:
        times = event_data.set_index("event_number")["event_time_s"]
        measures.append(Measure(WHOLE_EVENT, "time", "s", _without_warm_up(times),
                                tuple(int(e) for e in times.index)))
    if timing is None:
        return measures
    wall = timing.execute["wall_s"]
    algorithms = [a for a in timing.algorithms() if a in wall.columns]
    profiled = tuple(int(e) for e in wall.index)
    for metric, (what, unit) in MEASURE_METRICS.items():
        table = timing.execute.get(metric)
        if table is None:
            continue
        for algorithm in _costliest_first(table[[a for a in algorithms if a in table.columns]]):
            values = _without_warm_up(table[algorithm])
            if len(values):
                measures.append(Measure(str(algorithm), what, unit, values, profiled))
    return measures

align

align(measure: Measure, table: EventTable) -> str | None

Why measure's events cannot be matched to table's rows, or None when they can: the job's profiling must have counted exactly the entries 0 .. n-1 of the table's files.

Source code in k4bench/analysis/event_drivers.py

def align(measure: Measure, table: EventTable) -> str | None:
    """Why *measure*'s events cannot be matched to *table*'s rows, or
    ``None`` when they can: the job's profiling must have counted exactly
    the entries ``0 .. n-1`` of the table's files."""
    profiled = measure.profiled
    if len(set(profiled)) != len(profiled):
        return "the job's profiling counted some events twice"
    if sorted(profiled) != list(range(table.events)):
        return (
            f"the job's profiling counted {len(profiled)} events, numbered "
            f"{min(profiled, default='–')}–{max(profiled, default='–')}, but its event "
            f"files hold the entries 0–{table.events - 1}"
        )
    return None

paired

paired(measure: Measure, x: Series) -> DataFrame

measure's events beside x (a table column, indexed by entry), the events either lacks dropped: event, x and y.

Source code in k4bench/analysis/event_drivers.py

def paired(measure: Measure, x: pd.Series) -> pd.DataFrame:
    """*measure*'s events beside *x* (a table column, indexed by entry), the
    events either lacks dropped: ``event``, ``x`` and ``y``."""
    y = measure.values
    joined = pd.DataFrame({"x": x.reindex(y.index).to_numpy(), "y": y.to_numpy()}, index=y.index)
    joined.index.name = "event"
    return joined.dropna()

spearman

spearman(x: Series, y: Series) -> float | None

Spearman's rank correlation of x and y over the events both have, or None with fewer than :data:MIN_PAIRS of them or either constant.

Source code in k4bench/analysis/event_drivers.py

def spearman(x: pd.Series, y: pd.Series) -> float | None:
    """Spearman's rank correlation of *x* and *y* over the events both have,
    or ``None`` with fewer than :data:`MIN_PAIRS` of them or either constant."""
    both = pd.DataFrame({"x": x, "y": y}).dropna()
    if len(both) < MIN_PAIRS or both["x"].nunique() < 2 or both["y"].nunique() < 2:
        return None
    return float(both["x"].rank().corr(both["y"].rank()))

associations

associations(measure: Measure, table: EventTable, columns: Sequence[str] | None = None) -> DataFrame

Every ranked column of table (or columns) by the size of its rank correlation with measure, largest first: rho with its sign, and n, the events paired. Columns without a correlation (constant, too few events) are left out.

The columns with a value in every event of measure are ranked and correlated together; each of the others over the events it has (:func:spearman).

Source code in k4bench/analysis/event_drivers.py

def associations(measure: Measure, table: EventTable,
                 columns: Sequence[str] | None = None) -> pd.DataFrame:
    """Every ranked column of *table* (or *columns*) by the size of its rank
    correlation with *measure*, largest first: ``rho`` with its sign, and
    ``n``, the events paired. Columns without a correlation (constant,
    too few events) are left out.

    The columns with a value in every event of *measure* are ranked and
    correlated together; each of the others over the events it has
    (:func:`spearman`)."""
    names = list(columns if columns is not None else table.ranked())
    y = measure.values.dropna()
    x = table.frame.reindex(y.index)[names]
    complete = x.columns[x.notna().all()]
    rows = []
    if len(y) >= MIN_PAIRS and len(complete):
        rho = _rank_correlations(x[complete], y)
        rows += [(name, r, len(y)) for name, r in rho.items() if np.isfinite(r)]
    for name in x.columns.difference(complete, sort=False):
        pairs = paired(measure, table.frame[name])
        if (r := spearman(pairs["x"], pairs["y"])) is not None:
            rows.append((name, r, len(pairs)))
    frame = pd.DataFrame(rows, columns=["column", "rho", "n"]).set_index("column")
    return frame.reindex(frame["rho"].abs().sort_values(ascending=False, kind="stable").index)

trend

trend(x: Series, y: Series) -> Trend | None

The robust line of y against x; None with fewer than :data:MIN_SLOPE_EVENTS events or no two distinct x.

Source code in k4bench/analysis/event_drivers.py

def trend(x: pd.Series, y: pd.Series) -> Trend | None:
    """The robust line of *y* against *x*; ``None`` with fewer than
    :data:`MIN_SLOPE_EVENTS` events or no two distinct *x*."""
    both = pd.DataFrame({"x": x, "y": y}).replace([np.inf, -np.inf], np.nan).dropna()
    slope = theil_sen_slope(both["x"], both["y"])
    if slope is None:
        return None
    return Trend(slope, float(np.median(both["y"] - slope * both["x"])))

above_trend

above_trend(pairs: DataFrame, line: Trend) -> DataFrame

The events of pairs (:func:paired) more than :data:OUTLIER_SIGMAS robust standard deviations above line, furthest first: x, y, expected, residual and ratio (NaN where the line is too near zero, :data:MIN_RATIO_SHARE). When most events lie exactly on the line, as a count that rarely changes does, every event above it beyond rounding (:data:_ON_LINE).

Source code in k4bench/analysis/event_drivers.py

def above_trend(pairs: pd.DataFrame, line: Trend) -> pd.DataFrame:
    """The events of *pairs* (:func:`paired`) more than
    :data:`OUTLIER_SIGMAS` robust standard deviations above *line*, furthest
    first: ``x``, ``y``, ``expected``, ``residual`` and ``ratio`` (NaN where
    the line is too near zero, :data:`MIN_RATIO_SHARE`). When most events lie
    exactly on the line, as a count that rarely changes does, every event
    above it beyond rounding (:data:`_ON_LINE`)."""
    expected = line.at(pairs["x"])
    residual = pairs["y"].to_numpy() - expected
    spread = _MAD_SIGMA * float(np.median(np.abs(residual - np.median(residual))))
    if not spread > 0:
        spread = _ON_LINE * max(float(np.median(np.abs(pairs["y"]))), 1.0) / OUTLIER_SIGMAS
    floor = MIN_RATIO_SHARE * abs(float(np.median(pairs["y"])))
    with np.errstate(divide="ignore", invalid="ignore"):
        ratio = np.where(expected > floor, pairs["y"].to_numpy() / expected, np.nan)
    out = pairs.assign(expected=expected, residual=residual, ratio=ratio)
    return out[out["residual"] > OUTLIER_SIGMAS * spread].sort_values("residual", ascending=False)

association_matrix

association_matrix(measures: Sequence[Measure], table: EventTable, *, rows: int = 15, columns: int = 12, per_row: int = 5) -> tuple[DataFrame, DataFrame, dict[str, list[str]]]

The rank correlations of the first rows of measures that match table with the columns most associated with them: the per_row strongest of each, the strongest overall first, at most columns of them, a column that moves with one already chosen (:data:SIMILAR_RHO) folded into it. (rho, n, similar): the first two indexed by measure name with one column per table column, NaN where a pair has no correlation; the last, each chosen column's folded ones.

Source code in k4bench/analysis/event_drivers.py

def association_matrix(
    measures: Sequence[Measure], table: EventTable, *,
    rows: int = 15, columns: int = 12, per_row: int = 5,
) -> tuple[pd.DataFrame, pd.DataFrame, dict[str, list[str]]]:
    """The rank correlations of the first *rows* of *measures* that match
    *table* with the columns most associated with them: the *per_row*
    strongest of each, the strongest overall first, at most *columns* of
    them, a column that moves with one already chosen (:data:`SIMILAR_RHO`)
    folded into it. ``(rho, n, similar)``: the first two indexed by measure
    name with one column per table column, NaN where a pair has no
    correlation; the last, each chosen column's folded ones."""
    ranked = {
        m.name: associations(m, table)
        for m in [m for m in measures if align(m, table) is None][:rows]
    }
    strength: dict[str, float] = {}
    for frame in ranked.values():
        for name, rho in frame["rho"].head(per_row).items():
            strength[name] = max(strength.get(name, 0.0), abs(rho))
    candidates = sorted(strength, key=lambda name: -strength[name])
    alike = table.frame[candidates].rank().corr().abs()
    similar: dict[str, list[str]] = {}
    for name in candidates:
        twin = next((c for c in similar if alike.loc[name, c] > SIMILAR_RHO), None)
        if twin is not None:
            similar[twin].append(name)
        elif len(similar) < columns:
            similar[name] = []
    chosen = list(similar)
    rho = pd.DataFrame(
        {name: frame["rho"].reindex(chosen) for name, frame in ranked.items()},
    ).T.reindex(columns=chosen)
    n = pd.DataFrame(
        {name: frame["n"].reindex(chosen) for name, frame in ranked.items()},
    ).T.reindex(columns=chosen)
    return rho, n, similar