k4bench.analysis.event_size

k4bench.analysis.event_size

What each output file a job wrote holds on disk: {label}_event_size.json.

The runners measure every file a successful run wrote, right after it, while the files still exist (:func:record_event_size): ddsim's output file, and every file a k4run job created in its working directory. Each file has its size on disk; a podio file, an EDM4hep ROOT file whose events tree podio wrote, also has its events, its ROOT compression setting and, for every collection of its events, the bytes its branches take compressed, as they lie in the file, and uncompressed. :func:load_event_size reads the files back as :class:OutputFile records, whose :meth:OutputFile.per_event gives each collection's bytes per event.

How a collection's bytes are counted

podio writes each collection of the events tree as a branch of its name (<name>_objIdx for a subset collection) and one _<name>_<member> branch for each of its relations and vector members; split, each of those has a sub-branch per data member. A collection's bytes are those of all its branches and their sub-branches together, TKey headers included, as uproot counts them per branch (compressed_bytes, uncompressed_bytes). The collections are those podio's own metadata lists for the events category, with their types. The branches of no collection, podio's frame parameters, are :data:FRAME_PARAMETERS. The collections and the frame parameters together are the events tree; what the file holds besides, the metadata and run trees and ROOT's own records, is in the file's size only.

What the numbers do not say:

• what a production file of the same events would take: the benchmark writes tens to a few thousand events with the job's own compression setting, and a file with more events fills ROOT's baskets further, which can compress a little differently;
• the size of a file in any other format: an LCIO (.slcio) or RNTuple file has its size on disk only.

What each event holds

Measured with per_event, a podio file also has, entry by entry, each collection's entries (its elements, read from the count its branch keeps per entry), each calorimeter hit collection's energy_gev (:data:CALORIMETER_TYPES, the sum of its hits' energies) and the event_headers (the EventHeader's event and run number, None for an entry without one). Each collection is read on its own: one whose counts or energies cannot be read has none, which never reads as zero, and the bytes of every collection are kept whatever its per-event reading did. :meth:OutputFile.event_counts and :meth:OutputFile.event_energies give them back as frames indexed by entry.

CollectionSize dataclass

CollectionSize(name: str, type: str, subset: bool, compressed_bytes: int, uncompressed_bytes: int, entries: tuple[int, ...] | None = None, energy_gev: tuple[float, ...] | None = None)

One collection's branches in a file's events tree.

OutputFile dataclass

OutputFile(name: str, bytes: int, events: int | None = None, compression: int | None = None, collections: tuple[CollectionSize, ...] = (), event_numbers: tuple[int | None, ...] | None = None, run_numbers: tuple[int | None, ...] | None = None)

One file a job wrote.

has_events property

has_events: bool

Whether the file is a podio file holding events.

event_bytes property

event_bytes: int

The compressed bytes of its events tree.

per_event

per_event() -> DataFrame

Each collection's bytes per event, largest first: type, subset, compressed_kb and uncompressed_kb per event, and its share of the event's compressed bytes, indexed by collection. Empty for a file without events.

Source code in k4bench/analysis/event_size.py

def per_event(self) -> pd.DataFrame:
    """Each collection's bytes per event, largest first: ``type``,
    ``subset``, ``compressed_kb`` and ``uncompressed_kb`` per event, and
    its ``share`` of the event's compressed bytes, indexed by
    ``collection``. Empty for a file without events."""
    if not self.has_events:
        return pd.DataFrame(
            columns=["type", "subset", "compressed_kb", "uncompressed_kb", "share"],
            index=pd.Index([], name="collection"),
        )
    df = pd.DataFrame(
        [(c.name, c.type, c.subset, c.compressed_bytes, c.uncompressed_bytes)
         for c in self.collections],
        columns=["collection", "type", "subset", "compressed_kb", "uncompressed_kb"],
    ).set_index("collection")
    df[["compressed_kb", "uncompressed_kb"]] /= KB * self.events
    total = df["compressed_kb"].sum()
    df["share"] = df["compressed_kb"] / total if total > 0 else 0.0
    return df.sort_values("compressed_kb", ascending=False, kind="stable")

event_counts

event_counts() -> DataFrame

Each measured collection's elements per entry: one column per collection, indexed by entry. Empty when none was measured.

Source code in k4bench/analysis/event_size.py

def event_counts(self) -> pd.DataFrame:
    """Each measured collection's elements per entry: one column per
    collection, indexed by ``entry``. Empty when none was measured."""
    return self._event_frame({c.name: c.entries for c in self.collections})

event_energies

event_energies() -> DataFrame

Each measured calorimeter hit collection's summed energy (GeV) per entry: one column per collection, indexed by entry.

Source code in k4bench/analysis/event_size.py

def event_energies(self) -> pd.DataFrame:
    """Each measured calorimeter hit collection's summed energy (GeV)
    per entry: one column per collection, indexed by ``entry``."""
    return self._event_frame({c.name: c.energy_gev for c in self.collections})

describe_compression

describe_compression(setting: int | None) -> str

A ROOT compression setting in words: 101 is "ZLIB level 1".

Source code in k4bench/analysis/event_size.py

def describe_compression(setting: int | None) -> str:
    """A ROOT compression setting in words: 101 is "ZLIB level 1"."""
    if setting is None:
        return "unknown"
    algorithm, level = divmod(setting, 100)
    if level == 0:
        return "uncompressed"
    return f"{_ALGORITHMS.get(algorithm, f'algorithm {algorithm}')} level {level}"

load_event_size

load_event_size(run_dir: str | Path) -> tuple[dict[str, tuple[OutputFile, ...]], dict[str, str]]

Every label's output files in run_dir that could be read, and why each other could not: each record is read on its own, so one that is malformed hides no other configuration. Both empty when it has none.

Source code in k4bench/analysis/event_size.py

def load_event_size(
    run_dir: str | Path,
) -> tuple[dict[str, tuple[OutputFile, ...]], dict[str, str]]:
    """Every label's output files in *run_dir* that could be read, and why
    each other could not: each record is read on its own, so one that is
    malformed hides no other configuration. Both empty when it has none."""
    files: dict[str, tuple[OutputFile, ...]] = {}
    unreadable: dict[str, str] = {}
    for path, label in labelled_files(Path(run_dir), EVENT_SIZE_SUFFIX):
        try:
            files[label] = read_event_size(path)
        except (OSError, ValueError) as exc:
            unreadable[label] = str(exc)
    return files, unreadable

read_event_size

read_event_size(path: Path) -> tuple[OutputFile, ...]

The output files one _event_size.json records.

Raises ValueError for an unsupported schema_version (:func:~k4bench.plugin.schema.validate_event_size_schema) or a file that is not an event-size record.

Source code in k4bench/analysis/event_size.py

def read_event_size(path: Path) -> tuple[OutputFile, ...]:
    """The output files one ``_event_size.json`` records.

    Raises ``ValueError`` for an unsupported ``schema_version``
    (:func:`~k4bench.plugin.schema.validate_event_size_schema`) or a file
    that is not an event-size record.
    """
    raw = json.loads(path.read_text())
    validate_event_size_schema(raw, source=path)
    try:
        return tuple(OutputFile.from_json(f) for f in raw["files"])
    except (KeyError, TypeError, ValueError) as exc:
        raise ValueError(f"{path}: malformed event size record ({exc!r})") from exc

record_event_size

record_event_size(json_path: Path, files: Mapping[str, Path], *, per_event: bool = False) -> None

Write what each of files (name to path) holds to json_path, and what each event holds when per_event, warning about a file that could not be read. A failure costs only this record, never the run.

Source code in k4bench/analysis/event_size.py

def record_event_size(
    json_path: Path, files: Mapping[str, Path], *, per_event: bool = False,
) -> None:
    """Write what each of *files* (name to path) holds to *json_path*, and
    what each event holds when *per_event*, warning about a file that could
    not be read. A failure costs only this record, never the run."""
    try:
        record = measure_outputs(files, per_event=per_event)
        json_path.write_text(json.dumps(record, separators=(",", ":")))
    except Exception:  # noqa: BLE001 - the event size is an extra, never the run's failure
        print(f"  WARNING: no event size recorded:\n{traceback.format_exc()}")

measure_outputs

measure_outputs(files: Mapping[str, Path], *, per_event: bool = False) -> dict

The _event_size.json record of files, name to path: each one's size on disk and, for a podio file, what its events hold, event by event when per_event.

Source code in k4bench/analysis/event_size.py

def measure_outputs(files: Mapping[str, Path], *, per_event: bool = False) -> dict:
    """The ``_event_size.json`` record of *files*, name to path: each one's
    size on disk and, for a podio file, what its events hold, event by event
    when *per_event*."""
    open_root = _root_opener() if any(p.suffix == ".root" for p in files.values()) else None
    records = []
    for name, path in files.items():
        record: dict = {"name": name, "bytes": path.stat().st_size}
        if open_root is not None and path.suffix == ".root":
            try:
                with open_root(path) as root_file:
                    record.update(_podio_events(root_file, per_event=per_event))
            except Exception as exc:  # noqa: BLE001 - one unreadable file hides no other
                print(f"  WARNING: could not read the events of {name}: {exc}")
        records.append(record)
    return {"schema_version": EVENT_SIZE_SCHEMA_VERSION, "files": records}

event_content

event_content(root_file, *, entry_stop: int | None = None) -> dict

What each event of an open uproot root_file holds, up to entry_stop: collections, each collection's measured entries and energy_gev by name (a collection that could not be read is left out), and event_headers. Empty for a file that is not a podio file.

Source code in k4bench/analysis/event_size.py

def event_content(root_file, *, entry_stop: int | None = None) -> dict:
    """What each event of an open uproot *root_file* holds, up to
    *entry_stop*: ``collections``, each collection's measured ``entries`` and
    ``energy_gev`` by name (a collection that could not be read is left out),
    and ``event_headers``. Empty for a file that is not a podio file."""
    collections = podio_collections(root_file)
    if collections is None:
        return {}
    tree = root_file["events"]
    measured: dict[str, dict] = {}
    for name, (type_, subset) in collections.items():
        content = _collection_content(tree, name, type_, subset, entry_stop)
        if content:
            measured[name] = content
    headers = read_event_headers(tree, collections, entry_stop)
    return {"collections": measured, **({"event_headers": headers} if headers else {})}

collection_sizes

collection_sizes(branches: Iterable, collections: Mapping[str, tuple[str, bool]]) -> tuple[list[dict], dict]

The bytes of each of collections (name to type and whether it is a subset) among an events tree's top-level branches (uproot's), and of the branches of none, as (collections, parameters). Collections are in collections' order.

Source code in k4bench/analysis/event_size.py

def collection_sizes(
    branches: Iterable, collections: Mapping[str, tuple[str, bool]],
) -> tuple[list[dict], dict]:
    """The bytes of each of *collections* (name to type and whether it is a
    subset) among an events tree's top-level *branches* (uproot's), and of the
    branches of none, as ``(collections, parameters)``. Collections are in
    *collections*' order."""
    names = set(collections)
    sizes = {name: [0, 0] for name in collections}
    rest = [0, 0]
    for branch in branches:
        owner = _owner(branch.name, names)
        total = sizes[owner] if owner is not None else rest
        compressed, uncompressed = _branch_bytes(branch)
        total[0] += compressed
        total[1] += uncompressed
    return (
        [
            {"name": name, "type": type_, "subset": subset,
             "compressed_bytes": sizes[name][0], "uncompressed_bytes": sizes[name][1]}
            for name, (type_, subset) in collections.items()
        ],
        {"compressed_bytes": rest[0], "uncompressed_bytes": rest[1]},
    )

podio_collections

podio_collections(root_file) -> dict[str, tuple[str, bool]] | None

The events category's collections podio's metadata lists in an open uproot file, name to type and whether it is a subset; None for a file that is not a podio file.

Source code in k4bench/analysis/event_size.py

def podio_collections(root_file) -> dict[str, tuple[str, bool]] | None:
    """The ``events`` category's collections podio's metadata lists in an open
    uproot file, name to type and whether it is a subset; ``None`` for a file
    that is not a podio file."""
    kinds = root_file.classnames(recursive=False, cycle=False)
    if kinds.get("events") != "TTree" or kinds.get("podio_metadata") != "TTree":
        return None
    metadata = root_file["podio_metadata"]
    names, types, subsets = (
        metadata[_TYPE_INFO + leaf].array(library="np")[0]
        for leaf in ("name", "dataType", "isSubset")
    )
    return {str(n): (str(t), bool(s)) for n, t, s in zip(names, types, subsets, strict=True)}

read_event_headers

read_event_headers(tree, collections: Mapping[str, tuple[str, bool]], entry_stop) -> dict | None

Each entry's EventHeader event and run number up to entry_stop, None for an entry without one; None when the file has no EventHeader collection or it could not be read.

Source code in k4bench/analysis/event_size.py

def read_event_headers(tree, collections: Mapping[str, tuple[str, bool]], entry_stop) -> dict | None:
    """Each entry's EventHeader event and run number up to *entry_stop*,
    ``None`` for an entry without one; ``None`` when the file has no
    EventHeader collection or it could not be read."""
    name = next((n for n, (t, subset) in collections.items()
                 if t == _EVENT_HEADER_TYPE and not subset), None)
    if name is None:
        return None
    try:
        numbers = {
            key: [int(v[0]) if len(v) else None for v in
                  tree[f"{name}/{name}.{member}"].array(entry_stop=entry_stop, library="np")]
            for key, member in (("event_number", "eventNumber"), ("run_number", "runNumber"))
        }
    except Exception as exc:  # noqa: BLE001 - the headers are an extra of the record
        print(f"  WARNING: no event headers read from {name}: {exc}")
        return None
    return numbers

round_significant

round_significant(value: float) -> float

value to :data:_DIGITS significant digits.

Source code in k4bench/analysis/event_size.py

def round_significant(value: float) -> float:
    """*value* to :data:`_DIGITS` significant digits."""
    return float(f"{value:.{_DIGITS}g}")