k4bench.benchmark.k4run

k4bench.benchmark.k4run

Benchmark a k4run job and its variants.

The k4run counterpart of :mod:k4bench.benchmark.ddsim. k4Bench knows nothing about the job: the options files, the directory to stage and the job's own arguments come from the caller, and everything measured comes from the Gaudi layer (see :func:k4bench.runner.k4run.run_k4run).

A benchmark is the baseline job plus one run per variant, a named set of arguments appended to the job's own (label variant_<name>). Runs are sequential, for the same reason as the ddsim sweep: parallel runs would skew each other's wall time and memory.

Every run gets a fresh working directory, removed after the run: a writable copy of the stage directory when one is given (reconstruction configurations such as CLDConfig must run from their own directory), otherwise an empty one. A job's outputs therefore never land in the caller's directory, and one run's outputs are never read by the next. Paths in the job's arguments are read from that working directory, so give them as absolute paths.

Each run is preceded by a warm-up: the same job for one event, unmeasured and discarded. A job loads its libraries, geometry and conditions from CVMFS, and on a host whose CVMFS cache does not hold them yet, fetching them dominates its initialization (tens of seconds for CLD reconstruction, against a few seconds from a warm cache), which would measure the host's cache rather than the job. The warm-up event also reads in the code the first measured event runs.

With memory_events set, each run that succeeds is followed by a memory pass: the same job again for that many events, recording which component allocated each heap block, into <label>_heap.json (:mod:k4bench.analysis.heap). Recording owners makes every allocation several times dearer, so the pass is never timed; it runs from a fresh working directory, and everything it writes but the heap file is discarded. A run that failed gets none, as its job would most likely fail again, slower.

After the last run, each run's data flow (:mod:k4bench.analysis.dataflow) is built from what its job recorded, with Marlin -x of the stack saved beside it when a job ran Marlin processors.

Given the job's input file (source_input), what went into the events, the file's generator particles and collections event by event, is recorded once before the first run as event_source.json.gz (:mod:k4bench.analysis.event_source); each measured run records what each event of its own outputs holds.

K4runConfig dataclass

K4runConfig(options: list[str], n_events: int, log_dir: Path, stage_dir: Path | None = None, extra_args: list[str] = list(), variants: dict[str, list[str]] = dict(), setup_script: Path | None = None, verbose: bool = False, warm_up: bool = True, memory_events: int | None = None, source_input: Path | None = None)

All parameters needed to benchmark a k4run job.

Parameters:

Name	Type	Description	Default
options	list[str]	Gaudi options files, loaded in order. Relative paths are relative to stage_dir when it is given, otherwise to the current directory.	required
n_events	int	Number of events per run.	required
log_dir	Path	Directory where per-run logs and the auditor's files are written.	required
stage_dir	Path | None	Directory copied into every run's working directory, for jobs that read files relative to where they run.	None
extra_args	list[str]	Job arguments passed verbatim to every run.	list()
variants	dict[str, list[str]]	{name: arguments}. Each variant is one more run, with its arguments appended to extra_args.	dict()
setup_script	Path | None	Optional shell script sourced before each k4run invocation. A relative path is relative to the current directory.	None
verbose	bool	If True, print k4run output in real time instead of only logging it.	False
warm_up	bool	If True, precede each run with the same job for :data:WARM_UP_EVENTS events, unmeasured, so the measured run finds what the job reads from CVMFS already cached.	True
memory_events	int | None	Events of the untimed memory pass after each run that succeeds, which records the heap each component held; None skips it.	None
source_input	Path | None	The job's input file, from whose first n_events entries what went into the events is recorded; None records none. Dropped when a run's job starts past the first entry. A relative path is relative to the current directory.	None

planned_k4run_labels

planned_k4run_labels(variants: list[str]) -> list[str]

The result labels a k4run benchmark with these variants produces, in run order: baseline, then variant_<name> for each variant.

Source code in k4bench/benchmark/k4run.py

def planned_k4run_labels(variants: list[str]) -> list[str]:
    """The result labels a k4run benchmark with these *variants* produces, in
    run order: ``baseline``, then ``variant_<name>`` for each variant."""
    return [BASELINE_LABEL, *(f"{VARIANT_PREFIX}{name}" for name in variants)]

run_k4run_benchmark

run_k4run_benchmark(config: K4runConfig) -> list[RunResult]

Run the baseline job and every variant, and return all results.

A run that fails is marked with a non-zero return code and included in the results; the benchmark always continues to the last variant.

Source code in k4bench/benchmark/k4run.py

def run_k4run_benchmark(config: K4runConfig) -> list[RunResult]:
    """Run the baseline job and every variant, and return all results.

    A run that fails is marked with a non-zero return code and included in
    the results; the benchmark always continues to the last variant.
    """
    config.log_dir.mkdir(parents=True, exist_ok=True)
    labels = planned_k4run_labels(list(config.variants))
    job_args = [config.extra_args, *([*config.extra_args, *v] for v in config.variants.values())]
    runs = list(zip(labels, job_args, strict=True))

    # Removed first, so a run is never credited with a previous benchmark's.
    (config.log_dir / MARLIN_PROCESSORS).unlink(missing_ok=True)
    (config.log_dir / EVENT_SOURCE).unlink(missing_ok=True)
    for label in labels:
        label_path(config.log_dir, label, DATAFLOW_SUFFIX).unlink(missing_ok=True)

    if config.source_input is not None and record_event_source(
        config.log_dir / EVENT_SOURCE, config.source_input,
        n_events=config.n_events, input_content=True, checksum=True,
    ):
        print(f"Event source: {config.n_events} events of {config.source_input.name}")

    results: list[RunResult] = []
    for index, (label, args) in enumerate(runs, start=1):
        print(f"[{index}/{len(runs)}] {label}")
        print(f"         Options: {' '.join(config.options)}")
        # Removed first, so a run is never marked by a previous run's warm-up
        # or credited with a previous memory pass.
        for suffix in (WARM_UP_LOG_SUFFIX, HEAP_SUFFIX, HEAP_LOG_SUFFIX):
            label_path(config.log_dir, label, suffix).unlink(missing_ok=True)
        if config.warm_up:
            _warm_up(config, label, args)
        try:
            work_dir = _stage(config.stage_dir, label)
        except OSError:
            print(f"  ERROR staging {label}:\n{traceback.format_exc()}")
            result = RunResult(label=label, returncode=1, n_events=config.n_events)
        else:
            try:
                result = run_k4run(
                    options=config.options,
                    label=label,
                    n_events=config.n_events,
                    work_dir=work_dir,
                    log_dir=config.log_dir,
                    setup_script=config.setup_script,
                    extra_args=args,
                    verbose=config.verbose,
                    per_event_content=True,
                )
            finally:
                _remove(work_dir)
        print_run_result(result)
        results.append(result)
        if config.memory_events is not None and result.succeeded:
            try:
                _memory_pass(config, label, args)
            except Exception:  # noqa: BLE001 - the heap file is an extra, never the benchmark's failure
                print(f"  WARNING: no heap file for {label}:\n{traceback.format_exc()}")
    _drop_offset_event_source(config, labels)
    try:
        _record_dataflow(config, labels)
    except Exception:  # noqa: BLE001 - the data flow is an extra, never the benchmark's failure
        print(f"  WARNING: no data flow recorded:\n{traceback.format_exc()}")
    return results