File formats¶

The artifacts k4Bench reads and writes. These are the contracts between the runner, the analysis layer, the CI pipeline, and the dashboard.

The loaders are the source of truth

The JSON schemas below are described at a useful level of detail, but they evolve with the code. The analysis loaders (k4bench.analysis.loader) are what actually parse these files — when in doubt about a field, check the loader.

Output directory layout¶

A single k4bench run populates --output-dir (default logs/<xml-stem>/):

logs/<geometry>/
├── <label>.log              # full ddsim stdout/stderr incl. the time -v block
├── <label>_results.csv      # one row: run-level metrics
├── <label>_events.json      # per-event timing/RSS   (event plugin only)
├── <label>_regions.json     # per-detector timing    (region plugin only)
├── <label>_event_size.json  # what the output file holds on disk, collection by collection
└── event_source.json.gz     # the events' generator particles, from the first run's output file

A k4bench k4run run (default logs/<options-stem>/) writes the same log, results CSV and events JSON, the last one by the k4BenchAuditor in the same format, and instead of the regions JSON:

├── <label>_components.json  # per-event, per-component (algorithm/service) time, instructions, memory and allocations (auditor only)
├── <label>_joboptions.opts  # the fully resolved job options, as JobOptionsSvc dumps them
├── <label>_dataflow.json    # what each algorithm read and wrote (built after the last run)
├── <label>_heap.json        # the heap each component held, from the untimed memory pass (--memory-events only)
├── <label>_event_size.json  # what each file the job created holds on disk, collection by collection
├── <label>_warm-up.log      # the log of the unmeasured warm-up, only when it failed
├── <label>_heap.log         # the log of the memory pass, only when it failed
├── marlin_processors.xml.gz # `Marlin -x` of the stack, only when a job ran Marlin processors
└── event_source.json.gz     # the input's generator particles and collections, event by event (--source-input only)

Its labels are baseline and variant_<name> for each variant. A <label>_warm-up.log marks a run whose warm-up failed, so its measurement may include fetching the software from CVMFS. A <label>_heap.log marks a memory pass that failed or wrote no heap file, and says why. A k4run output_size_mb is the size of every file the job created in its working directory.

In the nightly CI each run directory also gets run_info.json and machine_info.json, and the whole directory is uploaded to EOS.

results CSV¶

<label>_results.csv is one header row plus one data row, whose columns are exactly the fields of RunResult — see that page for the authoritative list. The headline columns:

Column	Unit	Meaning
`label`, `returncode`, `n_events`	—	identity + ddsim exit code
`wall_time_s`	s	elapsed wall clock
`user_cpu_s`, `sys_cpu_s`	s	CPU times
`peak_rss_mb`	MB	peak resident memory
`peak_vmem_mb`	MB	virtual-size high-water mark (`VmPeak`), copied from the event plugin JSON; empty when unavailable
`output_size_mb`	MB	size of the EDM4hep ROOT output
`events_per_sec`	ev/s	throughput

Metric fields can be empty when /usr/bin/time -v output can't be parsed (e.g. a crashed run). load_results reads these into a DataFrame with appropriate numeric types.

peak_rss_mb and the event summary mean_rss_mb include file-backed pages that a CVMFS publish can evict mid-run. They therefore depend on page residency.

events JSON¶

Written by the event timing plugin: parallel arrays, one entry per completed event — event numbers, per-event wall time (seconds), and RSS (MB) sampled at the start and end of each event. load_event_timing returns a DataFrame per run label and adds an RSS-delta column. Event 0 is a warmup outlier.

Key	Shape	Meaning
`schema_version`	integer, optional	version of this JSON format (currently `1`); absent in files written before versioning
`event_numbers`	array	event IDs
`event_times_s`	array	event wall time in seconds
`event_rss_begin_mb`, `event_rss_end_mb`	arrays	total RSS before/after each event
`peak_vmem_mb`	scalar, optional	kernel virtual-size high-water mark up to the plugin shutdown read (includes initialisation)
`event_rss_anon_begin_mb`, `event_rss_anon_end_mb`	arrays, optional	anonymous resident memory before/after each event (`RssAnon`)
`event_rss_file_end_mb`	array, optional	file-backed resident memory after each event (`RssFile`)

All memory values use kB / 1024, following the existing MB convention. Failed /proc reads produce negative values. The runner leaves an unavailable virtual peak empty in the CSV; new event summaries exclude negative samples. Every array present must have the same length. Historical files without the new keys retain the original five DataFrame columns; optional arrays add rss_anon_begin_mb, rss_anon_end_mb and rss_file_end_mb only when present. The scalar is a run-level result, not an event column. Normal shutdown writes it even when the event arrays are empty.

Schema version. The plugin writes "schema_version": 1 as the first key, including when no event completed. The number describes the JSON format, not the software release. A file without the key is the legacy unversioned event format and loads exactly as before. A present value must be a plain integer: a string, boolean, float or value below 1 is refused as malformed, and a version newer than the reading k4bench supports is refused rather than parsed as a format it may not be. Both refusals raise ValueError, which trend builds treat as an unreadable run and skip; the runner leaves peak_vmem_mb empty for such a file. Additive, optional keys do not require a new version; a change that alters the meaning or shape of an existing key does.

Anonymous RSS growth. The trend summary derives rss_anon_slope_mb_per_event from event_rss_anon_end_mb: the Theil–Sen slope (median of all pairwise slopes) of the valid samples against their event numbers, in MB/event. Event 0 is excluded as warmup, negative (failed /proc) and missing samples are ignored, and at least 5 valid samples are required — otherwise the value is absent, not zero. Above 1500 samples the estimate is taken over 1500 evenly spaced samples, keeping cost bounded and the result deterministic. A positive slope indicates that anonymous memory grew while events were processed, which a leak or other accumulating state can cause; it is not proof of either. The metric is reported-only until enough history exists to choose a judgement policy for it.

peak_vmem_mb and mean_rss_anon_mb enter regression judging automatically once the engine has sufficient baseline history for each metric. Total RSS (peak_rss_mb, mean_rss_mb) and file-backed RSS (mean_rss_file_mb) remain reported-only diagnostics, as does the growth rate rss_anon_slope_mb_per_event.

regions JSON¶

Written by the region timing plugin (schema_version: 1). The version follows the same rules as the events JSON: an absent key is the legacy format, and a malformed or newer version is refused with ValueError. It attributes Geant4 stepping time to top-level DD4hep detectors under two views, at_location and by_birth, plus per-event totals, step counts, and metadata (timer used, the list of attributed detectors, measured timer overhead). load_region_timing parses it into per-run DataFrames keyed by detector. The special bucket unattributed collects steps outside any detector (vacuum/world).

dataflow JSON¶

Written by the k4run benchmark after its last run (schema_version: 1, same version rules as the events JSON), one per configuration, from three records:

Gaudi algorithms, k4FWCore's included: the keys of the data handles each declared. The k4BenchAuditor records them in _components.json as each algorithm's inputs and outputs (null for a component that holds no handles, such as a service). The k4FWCore Reader declares one output per collection of its input file.
Marlin processors in a MarlinProcessorWrapper: the parameters Marlin marks as LCIO input or output collections, with their LCIO type, from marlin_processors.xml.gz (Marlin -x of the stack). The collection names are the parameter's value in the job options, or its default where the job leaves it unset (explicit: false). The wrapper's EDM4hep→LCIO and LCIO→EDM4hep converters (convertAll, collNameMapping) come from the job options.
The k4FWCore Writer: its keep/drop rules, IOSvc.outputCommands.

Each algorithm (sequencers and services left out, in the order the auditor first saw them) lists its inputs and outputs as collections with a name, a store (gaudi for Gaudi's event store, lcio for the LCIO event Marlin processors share) and, for Marlin, type, parameter and explicit. handles_recorded and marlin_recorded say whether each source was available. DataFlow.graph replays the job to link each read to the algorithm that wrote it, carrying collections across the converters, and the Reconstruction page's Data Flow draws the result.

What it cannot know:

A Marlin processor that takes a collection name as a plain string parameter, rather than registering it as an input or output collection, declares nothing. In CLD reconstruction that is AIDAProcessor, OverlayTimingGeneric, MergeCollections, Statusmonitor and PatchCollections, so a collection only one of them writes (CLD's MCPhysicsParticles) has no recorded writer.
A declared input need not be read: a Marlin input left at its default is listed all the same.
The converters are modelled, not observed: one converting everything is taken to convert every collection the other store holds at that point, under its mapped name, unless the target already holds one of that name.
Gaudi handles carry a key, not a type.
Runs before the auditor recorded data handles have no Gaudi reads and writes (null); the October 2026 CLD runs were backfilled with the Reader's outputs taken from their input file.

run_info.json & machine_info.json¶

Written by the nightly CI (run_info.json by .github/scripts/run_info.py, machine_info.json by .github/scripts/machine_info.py) and read by the dashboard.

run_info.json — describes one run directory: date, platform, Key4hep release, detector, sample, the GitHub run link and commit, event count, the list of run labels (configs), the ddsim flags the run was invoked with (ddsim_args) and the Monte-Carlo seed parsed out of them (random_seed, null when the run fixed none), the configured source input_files (before HepMC files are copied to /tmp), and both configured and resolved paths for geometry (configured_xml_path, xml_path) and steering (steering_file, resolved_steering_file). Keeping both forms lets a reproducer compare the logical workload while executing the exact path each release used. The record also carries the stack's git provenance (k4h_stack_setup, k4h_stack_manifest, k4h_packages; see below). tool says what the run benchmarked: ddsim, or k4run (absent on runs older than k4run benchmarks, which were all ddsim). A k4run record also carries options, stage_dir and resolved_stage_dir, k4run_args (with $VARs expanded, i.e. the local input paths the job read), variants ({name: k4run_args}) and data_files (as configured; absent on runs older than the key); its configured_labels are baseline and variant_<name>. suite names the benchmark suite the run belongs to (sim or reco) and so its run tree (see EOS layout); reco runs land in runs/reco/, which the dashboard's Reconstruction page reads; the regression report and blame read only runs/sim/.
machine_info.json — the benchmark host and its state around the run: CPU model/cores, RAM/swap totals, and _start/_end snapshots of load, available memory, CPU frequency, and thermal throttling. The pairs let the dashboard show whether the machine was loaded or throttling — context for trusting a number.

Stack provenance (`k4h_packages`)¶

run_info.json records the upstream commit of every HEAD package in the Key4hep LCG view, read off CVMFS as the benchmark runs:

"k4h_stack_setup": "/cvmfs/sft-nightlies.cern.ch/lcg/views/devkey-head/Thu/x86_64-el9-gcc16-opt/setup.sh",
"k4h_stack_manifest": "/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/devkey-head/Thu/LCG_externals_x86_64-el9-gcc16-opt.txt",
"k4h_packages": {
  "k4geo":      {"commit": "9e2047a", "version": "HEAD", "repo_url": "https://github.com/key4hep/k4geo.git"},
  "fcc_config": {"commit": "1312733", "version": "HEAD", "repo_url": "https://github.com/HEP-FCC/FCC-config.git"}
}

This is what lets a regression be traced to the commits that could have caused it: diffing two nights' maps gives the exact set of upstream changes between them. It is captured at run time because it cannot be recovered later. LCG's weekday slots rotate after roughly a week, so the setup and manifest paths are identifiers only while their generated date still matches the recorded release. Legacy Spack records use k4h_stack_root instead of k4h_stack_manifest.

LCG build metadata does not carry source URLs, so repo_url is read from the exact LCGCMake toolchain revision that built the view (including its inherited heptools-* files). Because a nightly is incremental, installs record several revisions and each package's own build revision is used. That lookup is best-effort: if CERN GitLab is unavailable the commits are still recorded, but their repository links are null. k4h_packages is absent for runs predating provenance capture, and an empty map means unknown, never unchanged.

Measured stack and externals¶

Each captured run also contains LCG_externals.txt.gz and LCG_contrib.txt.gz, byte-for-byte copies of that view's manifests compressed with gzip. They record the full stack's versions, build hashes, install paths and dependencies. Header lines describe the platform, toolchain and compiler. Relative contrib install paths are rooted at /cvmfs/sft.cern.ch/lcg/contrib/. These files live beside run_info.json; readers never need the weekday slot to remain on CVMFS.

The small additions to run_info.json are:

"k4h_stack_capture": 1,
"k4h_loaded_packages": {
  "Geant4": {"version": "11.4.3", "githash": "e411040b8"},
  "gcc": {"version": "16.1.0", "githash": "8ef9376e8"},
  "k4geo": {"version": "HEAD", "commit": "e1ba7bc", "githash": "e411040b8", "repo_url": "https://github.com/key4hep/k4geo.git"}
},
"k4h_unmappable_packages": ["cldconfig", "fcc_config"],
"g4_datasets": {"G4LEDATA": "G4EMLOW8.8"}

k4h_stack_capture: 1 means both compressed manifests were saved. The loaded map is the union across every measured config that produced a result CSV; geometry sweeps and reconstruction variants can load different code. Missing instrumentation or a missing handoff makes this map absent, never an empty set. Version and available upstream/build revisions come from each matched manifest install. k4h_packages retains its existing HEAD-only contract.

The timing action's destructor and the auditor's finalize() read unique file paths from /proc/self/maps. The k4run options file also appends imported Python module paths at exit: reconstruction imports packages without mapped libraries. Executed staged options files are mapped through their recorded source paths, and a ddsim run's resolved steering file is matched the same way. Paths match resolved manifest install directories by longest prefix, including stable releases and the contrib compiler. Unmatched CVMFS paths are logged. A loaded package whose install path resolves elsewhere also records that directory as install_realpath, so the libraries the auditor names by their resolved paths are attributed to it after the symlink has moved on. Private .LABEL.loaded handoffs are consumed and deleted before writing run_info.json, and hidden files are excluded by the upload glob even if cleanup fails. Event and component JSON retain schema version 1.

Each component entry also names impl_library, the shared object defining its impl class, symlinks resolved: the type's library, or for a MarlinProcessorWrapper the library of the Marlin processor it runs. It is empty where that library could not be found. Files written by k4bench 0.0.49 and earlier lack the key; readers then take library where impl is the type itself, and leave it empty for a wrapped processor.

g4_datasets records directory names for the job's G4*DATA environment. With instrumentation these come from the actual child process, including any sourced setup. These are configured datasets, not a claim that every dataset was opened. They can override the toolkit’s default dataset lookup independently of its version (Geant4 installation guide). Without instrumentation the job environment is still recorded. When measured configs report different dataset environments, g4_datasets is absent. os in machine_info.json describes /etc/os-release inside the container; the host's kernel is not used as an OS version.

Stack Changes uses the selected ddsim detector/sample's latest run at each endpoint, with a union of their measured loaded sets. The whole-platform toggle only widens regression results. HEAD rows retain their forge compare links. Loaded pinned rows link to package-filtered lcgcmake history at the head build, for example -/commits/e411040b8?search=Geant4; unloaded pins have no link. Hash-only rebuilds are counted separately. An uncaptured endpoint falls back to the HEAD table with an explicit note. The k4run page remains separate and its captures live in the reco suite's run tree, runs/reco/.

A cold comparison adds one run-date listing and four GETs per endpoint (run_info, two manifests, machine_info), or two requests per legacy endpoint. The existing platform release discovery and regression-report reads are unchanged. The scope-specific snapshot is cached for an hour; no per-config remote reads are needed.

Blame reads only the two exact runs' small metadata files. Loaded pinned version changes, compiler, configured Geant4 data and OS changes are stored as an optional external_changes list on each BlameEntry, with scope, name, before, after and kind (pin, dataset, os). They are facts, never PR candidates. The existing real_change assessment can name one as the explanation, including a fixed-seed sample's re-rolled long events. No new assessment value is introduced. With no external changes the field and prompt block are omitted, and the comment facts digest and rendered comments are unchanged.

The same two runs limit the candidates: a changed HEAD package that neither run loaded is dropped before PR discovery, so its pull requests are never ranked, shown or commented on. ddsim does not record the Python modules its steering file imports, so a ddsim capture also lists k4h_unmappable_packages: HEAD packages it did not load that install no shared library under lib*/. These stay candidates. A k4run capture records an empty list. When either run has no k4h_unmappable_packages, or has no or an empty k4h_loaded_packages, every changed package stays a candidate.

Reading across the Spack-to-LCG boundary¶

Runs recorded from a Spack release and runs recorded from an LCG view are not directly comparable, and history keeps both:

Platform. The EOS path segment moved from x86_64-almalinux9-gcc14.2.0-opt to x86_64-el9-gcc16-opt. Results are filed under {detector}/{platform}/... in their suite's run tree, so the LCG platform has its own tree, metadata and report; the old tree stays readable and simply stops growing.

For regression detection the LCG platform's metric series continue the Spack platform's (k4bench/regression/lineage.py): the Spack nights come first in the history each series is judged against, as ordinary baseline points. A shift caused by the migration is therefore reported once, like any other step (watch, then regression), with a window from the last Spack run to the first LCG one, and the baseline re-anchors on the new level. Spack releases dated on or after the first LCG release are left out, since the two stacks publish different builds under the same date.

Once the LCG platform has run, the Spack platform is no longer expected: it drops out of the report instead of failing with no run uploaded every night until the grace period expires. A backfill of a night before the first LCG run still reports a night the Spack platform really did miss. - Package names. LCG spells them as upstream does (DD4hep, fcc_config) where Spack lower-cased and hyphenated them (dd4hep, fcc-config). - Commit length. LCG records the abbreviated sha (9e2047a) where Spack recorded all 40 characters. Both forges resolve either in a compare link.

A stack diff that spans the boundary therefore reports every package as removed and re-added. That is cosmetically noisy but inert for attribution: an added package has no base commit, so it yields no blame window.

Benchmark YAML¶

.github/benchmarks/{suite}/{concept}/{detector}.yml configures the nightly matrix; expanded by list_benchmarks.py. A config's path is the run tree its jobs upload to (see EOS layout): sim/CLD/CLD_o2_v09.yml uploads to runs/sim/CLD/CLD_o2_v09/. Keys and location rules are tabulated in the Configuration reference. A sim example (sim/IDEA/IDEA_o1_v03.yml):

xml: FCCee/IDEA/compact/IDEA_o1_v03/IDEA_o1_v03.xml
steering_file: $FCCCONFIG/FullSim/IDEA/IDEA_o1_v03/SteeringFile_IDEA_o1_v03.py
sweep: false
samples:
  - name: single_e-_10GeV
    n_events: 100
    ddsim_args: >-
      --enableGun --gun.particle e- --gun.distribution uniform --gun.energy 10*GeV
  - name: p8_ee_Zbb_ecm91
    n_events: 100
    input_files: root://eospublic.cern.ch//eos/.../events_noVtxSmear.hepmc

A reco benchmark (reco/CLD/CLD_o2_v09.yml) runs k4run on a frozen input instead, and uploads to the reco suite's own run tree, so its samples may share their names with the sim samples on the same xml:

xml: FCCee/CLD/compact/CLD_o2_v09/CLD_o2_v09.xml
stage_dir: $CLDCONFIG/share/CLDConfig
options: CLDReconstruction.py
k4run_args: >-
  --compactFile $DETECTOR_XML --inputFiles $LOCAL_INPUT_FILES --outputBasename reco
variants:
  - name: truth_tracking
    k4run_args: --truthTracking
samples:
  - name: p8_ee_Zbb_ecm91
    n_events: 100
    input_files: https://k4bench-data.web.cern.ch/inputs/reco/CLD/CLD_o2_v09/p8_ee_Zbb_ecm91_100ev_seed42.edm4hep.root
    k4run_args: --cms 91

Automated detector bumps cover the sim suite only: a reco benchmark's frozen input was simulated with the geometry xml names.

Automated detector version bumps¶

detector-bump.yml runs daily at 05:00 UTC and on manual dispatch. It looks in the CVMFS devkey-head/latest/x86_64-el9-gcc16-opt view used by the nightly, not k4geo main. Only already benchmarked families are tracked; when versions coexist, only the highest is replaced. A candidate must contain its matching XML.

The bot opens one PR per family on bump/<family>, one at a time: every bump edits the shared lineage blocks, so while any bump/ PR is open, later runs propose nothing and leave its branch untouched, keeping maintainer commits on it. Only this repository's bump/ branches count; PRs from forks are ignored, whatever their branch name. Once it is merged, the next run proposes the next family from the updated base. Closing a bump PR unmerged rejects that exact version: later runs skip it and move on to the next family, and propose the family again only when a newer version appears. All runs share one concurrency group, so two runs never both find no open PR and open one each. It renames the benchmark YAML, updates geometry references and examples, and replaces the one-hop regression lineage entry, recording the dropped entry's predecessor and the config that replaced it in RETIRED_DETECTORS, so it is still treated as replaced. Versioned steering paths change only when the replacement exists in the same nightly's FCC-config. Otherwise the PR records that the existing steering file was retained. Review header prose, steering compatibility, sweep choices, timeouts, and the baseline transition. The PR opens as a draft; CI (including real geometry patching) and PR-Agent run after a maintainer clicks Ready for review; a human merges it. There is no auto-merge. The action is invoked only for actual bumps, so a no-op run does not close existing PRs or clean up merged branches.

Steering selection checks file existence. On PRs whose branch starts with bump/, the required run job also smoke-tests the changed benchmark YAMLs. It resolves sample overrides and runs one 10 GeV electron gun event per distinct geometry/steering pair in the latest nightly stack, with a five-minute timeout per pair unless the benchmark YAML sets a top-level smoke_timeout in minutes (IDEA_o2's DR tube geometry needs longer to build). A simulation failure, timeout, or missing/empty output fails CI. Ordinary PRs and pushes to main skip this additional step. The smoke test does not run benchmark sweeps or download physics samples; physics correctness and sample-specific simulation options still need review.

To repeat a smoke test locally, source the nightly view and run:

python3 .github/scripts/smoke_benchmarks.py .github/benchmarks/sim/IDEA/IDEA_o1_v04.yml

No GitHub App, PAT, or additional repository secrets are needed. The workflow uses the built-in GITHUB_TOKEN, with Contents and Pull requests write access only in the PR job, and signs commits as github-actions[bot]. Workflow examples are version-independent because this token cannot update workflow files.

One-time repository setup:

In Settings → Actions → General → Workflow permissions, enable Allow GitHub Actions to create and approve pull requests. The workflow creates PRs; it does not approve or merge them. If the setting is unavailable, an organization administrator must allow it first.
Create the detector-bump label if it does not already exist.

Events caused by GITHUB_TOKEN start no workflows, so neither the bot's opening of the PR nor its push runs CI or PR-Agent. See GitHub's workflow triggering documentation. The bot therefore opens a draft, and a maintainer clicking Ready for review starts both: the Pipeline and AI PR Review workflows trigger on ready_for_review. Later maintainer pushes trigger them as usual. This is separate from approving or merging the PR.

Local inspection (no files written unless --apply --family NAME is supplied):

py-venv/bin/python .github/scripts/bump_detectors.py \
  --k4geo "$K4GEO" --fcc-config "$FCCCONFIG"

After merging, dispatch gh workflow run detector-bump.yml to check the no-op case. Open and rejected bump PRs are tracked repository-wide, so the workflow proposes bumps only when run on the default branch; a dispatch on any other branch does nothing. To test PR creation and CI, use a disposable fork whose default branch has one config deliberately behind: dispatch there, mark the resulting draft ready for review, and check the run result.

heap JSON¶

Written by the k4BenchAuditor in the k4run benchmark's memory pass (--memory-events, or memory_events in a nightly config; schema_version: 1, same version rules as the events JSON): after each timed run, the same job runs again for that many events with K4BENCH_ALLOC_OWNERS=1, and the preloaded allocation counter records which component allocated every heap block. While an audited call runs, its component is its thread's owner, so whatever the call allocates, at any depth, is charged to the innermost running component, and stays charged to it until it is freed, wherever and whenever that happens. A service initialized from within an algorithm's call is a call of its own and keeps its own memory, so the geometry lands on the geometry service. What no component allocated (the Python configuration, ROOT's start-up, the framework between calls) belongs to (no component). load_component_heap reads the file into a ComponentHeap, whose frames join the timed run's _components.json by component name.

Key	Shape	Meaning
`components`	array	`"(no component)"`, then the auditor's components in its order
`before_first_event`	object, or `null` without events	at the first event's start: what initialize and start left
`event_numbers`	array	the events of the pass
`execute`	object	at each event's end, a row per event
`at_end`	object	at the auditor's finalize: what survived the job's own finalize
`table`	object	`peak_entries`, the most blocks the counter's table held at once, and `mapped_bytes`, the memory it mapped for them

Each moment holds, per component (an array with one value per entry of components; in execute, one such array per event):

held_bytes — the heap it allocated and has not freed yet;
peak_held_bytes — the most it held at once: since the auditor started for before_first_event, and from the event's first top-level call to its last in execute. Not at at_end. A job that had several events in flight at once writes the per-event peaks as null: the owners' peaks are the process's, so no event has one of its own;
start_held_bytes — in execute only, what it held as the event opened. The peak beyond it is the component's working set, the memory the event needed of it (ComponentHeap.working_set).

and for the process (in execute, one value per event):

malloc_in_use_bytes — the heap as glibc's mallinfo2 counts it: chunks in use, those it mapped on their own included, and the free ones it caches per thread. The components' held_bytes add up to nearly that; the rest is the blocks allocated while the process was still loading, before the counter started, and the auditor's own memory (0.02–0.06% in CLD reconstruction);
rss_anon_mb — anonymous RSS, -1 when it could not be read. It also holds what libraries map themselves, the free heap malloc has not returned, and the counter's own table (table.mapped_bytes, all of it resident), but not heap that was allocated and never touched. In CLD reconstruction that is about a sixth of the heap: held bytes are memory allocated, not memory resident.

Heap bytes count as _components.json's allocation metrics do: chunk sizes, malloc's 8-byte header and the rounding to 16 bytes included.

What the numbers do not say:

Who uses memory. A block is charged to the component that created it. The event data the input reader creates stays the reader's until the output writer frees it, and the memory a library takes as it loads (static constructors, ROOT dictionaries) is the component's whose call loaded it.
What lies outside malloc's heap. Memory a library maps itself (pool allocators, CPython's arenas, large buffers outside malloc) and file-backed memory (code, CVMFS), which the events JSON's event_rss_file_end_mb shows.
The timed run's values. The pass is another, shorter run of the same job: it reads the same input, so its events are the timed run's first ones, but recording owners slows every allocation (a lookup in a table that outgrows the caches), so its numbers are comparable in kind, not identical in value. Its timing, components, events and job options are discarded.
What a service first used during an event keeps before the first event. Such a service is initialized then, and what it keeps is held from that event on.

In a job running calls concurrently (threads > 1 in _components.json), attribution still follows each thread's running component, but a snapshot taken while other threads run catches them mid-call. The reconstruction jobs benchmarked so far run on one thread, one event at a time.

event size JSON¶

Written by the runners right after each successful run (schema_version: 1, same version rules as the events JSON), one per configuration, while the output files still exist: ddsim's --outputFile, or every file a k4run job created in its working directory. Its files list each one with its name (relative to that directory) and its bytes on disk. A podio file, an EDM4hep ROOT file whose events tree podio wrote, also has:

Field	Meaning
`events`	entries of the `events` tree
`compression`	the file's ROOT compression setting: 100 × algorithm + level, so `101` is ZLIB level 1 and `505` ZSTD level 5
`collections`	each collection podio's metadata lists for the `events` category: `name`, `type` (e.g. `edm4hep::MCParticleCollection`), `subset`, and the `compressed_bytes` and `uncompressed_bytes` of its branches
`parameters`	the same two for the branches of no collection: podio's frame parameters

A collection's branches are the one named after it (<name>_objIdx for a subset collection) and its _<name>_<member> relations and vector members, each with its sub-branches, counted as uproot counts them (compressed_bytes, TKey headers included). They match ROOT's TBranch::GetZipBytes("*") to the byte. The collections and the parameters together are the events tree; the metadata and run trees are in the file's bytes only. LCIO and RNTuple files, and a run without uproot (it ships with the Key4hep stack), have bytes only. OutputFile.per_event gives each collection's kB per event, and the Event Size section of both dashboard pages draws it.

The numbers are this run's: the benchmark writes tens to a few thousand events with the job's own compression setting, and a production file of many more events fills ROOT's baskets further, which can compress a little differently.

What each event holds is recorded, as additive optional keys, for the runs that measure it: a ddsim benchmark's first run (its baseline, or the single run of an include or exclude benchmark) and every measured run of a k4run benchmark, never a warm-up or memory pass. Each list has one value per entry of the file:

Field	Meaning
`collections[].entries`	the collection's elements in each entry, from the count its branch keeps per entry
`collections[].energy_gev`	for `edm4hep::SimCalorimeterHitCollection` and `edm4hep::CalorimeterHitCollection`, the summed energy of its hits in each entry, to six significant digits
`event_headers`	`event_number` and `run_number`: each entry's EventHeader numbers, `null` for an entry without one

A collection whose counts or energies could not be read has none, which never means zero; its bytes are kept. A file without them reads with no per-event data, and a reader that predates them ignores them. OutputFile.event_counts and OutputFile.event_energies give them as frames indexed by entry.

event source JSON¶

event_source.json.gz, gzipped JSON (schema_version: 1), one per run and shared by its configurations: what went into the events. A ddsim benchmark writes it from its first run's output file; a k4run benchmark given --source-input (the nightly passes the job's input when it reads exactly one) writes it from that file before the first run, and removes it after the runs when any run's job options set FirstEventEntry to other than 0, since it describes the file's first entries. A failure to write it never fails the run.

Field	Meaning
`origin`	`recorded` by the benchmark; `backfill-verified`, added later from the file the run read, matched by checksum; `backfill-inferred`, added later from a file that should hold the same generator particles
`source`	`file` (its name), `first_entry` and `events` (the entries read), `sha256` (k4run inputs), `label` (the ddsim configuration whose output it is)
`event_headers`	as in the event size JSON, for the entries read
`generator`	`offsets` (`events + 1` values: where each event's particles begin) and one list per column for the particles of the generator record (`generatorStatus` not 0): `pdg`, `gen_status`, `charge`, `mass`, `px`, `py`, `pz`, `vx`, `vy`, `vz` (GeV, mm), floats to six significant digits
`input_content`	k4run only: `collections`, each with `name`, `type` and, as in the event size JSON, `entries` and `energy_gev` for the entries read

Entry i is the i-th event the job's profiling counted, since every job reads its input from its first entry: the key that joins this file to the per-event timing. The EventHeader numbers are kept beside it, not used as the key. What transport made of the particles (simulatorStatus, end points) is left out, since it depends on the detector and so differs between the configurations sharing the file. The record is still simulation's, not the generator's alone: it carries the crossing-angle boost the steering sets, and the products of decays Geant4 performs start where transport left their parent, so in Z→bb̄ about 1% of the particles, all born off the interaction point, differ slightly between detectors. load_event_source reads it as an EventSource, whose particles() adds p, pt, E, theta, phi, eta, abs_cos_theta, vr and the masks final and charged.

EOS layout¶

Nightly results live under EOS_ROOT = /eos/user/j/jbeirer/k4bench, encoding every browse dimension in the path so discovery is just directory listing. The root holds one directory per kind of data, and each kind is split by suite first (k4bench/layout.py spells every path):

runs/{suite}/{concept}/{detector}/{platform}/key4hep-{release}/{sample}/{YYYY-MM-DD}/
    sim:   run_info.json  machine_info.json  [event_source.json.gz]
           {config}_results.csv  {config}_events.json  {config}_regions.json  {config}.log
           [{config}_event_size.json]
    reco:  run_info.json  machine_info.json  [marlin_processors.xml.gz]  [event_source.json.gz]
           {config}_results.csv  {config}_events.json  {config}_components.json
           {config}_joboptions.opts  {config}_dataflow.json  {config}.log
           [{config}_heap.json]  [{config}_event_size.json]  [{config}_warm-up.log]
           [{config}_heap.log]
reports/{suite}/{YYYY-MM-DD}/
    report.json
    blame.json   (only on nights with an attributable confirmed regression)
reproducers/{suite}/
    {detector}-{sample}-{config}-{metric}-{base}-{onset}-{digest}.txt
inputs/{suite}/{concept}/{detector}/
    frozen inputs the suite's benchmarks fetch over WebEOS, each with its
    provenance and producing script

A suite is an independent benchmark pipeline with its own configs (.github/benchmarks/{suite}/), jobs, run tree, report and readers: sim (ddsim) and reco (k4run on a frozen input). No suite reads another suite's runs, and every reader names the suite it reads. A new suite is a new name at the second level of each kind; nothing else about the layout changes. The concept is the detector concept a detector config belongs to — its name up to the first underscore (ALLEGRO_o1_v04 → ALLEGRO, ILD_FCCee_v01 → ILD, SiD → SiD) — so the versions of one detector sit side by side.

This is the integration contract between CI and the dashboard (data flow). The regression report and blame read the sim suite; the dashboard reads both, one page per suite. No report judges the reco suite yet, so there is no reports/reco/ and the Reconstruction page reads its runs directly. reports/sim/ holds the nightly regression report (written by the regression-report CI job, rendered by the dashboard's Regressions tab), and reproducers/sim/ holds the runnable recipes the blame pull-request comments link to (written by the same job's comment step, read directly by whoever follows the link). A recipe is named for the measurement and change window it reproduces rather than for a night, so re-publishing the same window replaces the same file and a standing comment's link keeps working — see PR comments → Reproducing the measurement.

The night a report covers (`report.json`)¶

reports/sim/{YYYY-MM-DD}/ is named for the night the report covers, and summary.report_night repeats it. That is normally the newest run date the report holds. A night whose benchmarking uploaded nothing — every job of the fan-out failed — has no run to be named after, so its report carries the night in a top-level night key instead and reports one no run uploaded for {night} job failure per triple. The key is absent on every other night, where the newest run date is the answer.

Metric history on confirmed verdicts (`report.json`)¶

Every confirmed verdict in report.json carries a bounded history: a tail of up to twelve releases (never nights — nights sharing a release are repeat measurements of one software state), oldest first, ending at the release that verdict judged. Each point records the release date, its level, how many nights it aggregates and how many of those the detector could actually judge, the severity and direction it was flagged with, the machine(s) that ran it, and each machine's own level:

"history": [
  {"run_date": "2026-07-03", "value": 100.2, "n_runs": 2, "n_judged": 2,
   "severity": "OK", "direction": "NONE",
   "hosts": [{"name": "bench01", "cpu_cores": 64}]},
  {"run_date": "2026-07-04", "value": 120.4, "n_runs": 2, "n_judged": 2,
   "severity": "CONFIRMED", "direction": "UP",
   "hosts": [{"name": "bench02", "cpu_cores": 64}, {"name": "bench01", "cpu_cores": 64}],
   "host_levels": [
     {"host": {"name": "bench02", "cpu_cores": 64}, "value": 120.3},
     {"host": {"name": "bench01", "cpu_cores": 64}, "value": 120.5}]}
],
"region_deltas": [
  {"region": "HCAL_barrel", "base": 0.31, "onset": 4.52, "delta": 4.21},
  {"region": "ECAL_barrel", "base": 1.02, "onset": 1.03, "delta": 0.01}
]

n_judged: 0 means the release was measured but never assessed — an unreliable host, or a series still warming up — so its level must not be read as a flat night. A release that recorded nothing at all is simply absent: a gap is a gap, never a zero.

host_levels is the median each machine measured for the release, by the same rule as value: when any night was judged, only judged nights count, so a machine whose nights were all unjudged has no entry. It is what lets attribution tell a step that a machine measuring both sides reproduced (above, bench01) from one only a newly added machine measured. Points written before the field existed carry none, and readers treat that as "unknown".

A series that continues a replaced geometry version's history (DETECTOR_SUCCESSORS in k4bench/regression/lineage.py) carries the replaced version's releases too. Such a point has a detector key naming the detector config that measured it; absent or null means the verdict's own detector. A release both versions measured is two points, in order: the replaced version's first, then the successor's. The verdict records which config measured each end of its window in last_accepted_detector and onset_detector, using the same convention: absent or null means the verdict's own detector, and that is also what reports written before these fields existed hold. When the two ends differ, the window spans the version switch, and the geometry change alone can explain the step. The window can lie within one release: the base is that release's replaced-version point and the onset is its successor point.

The field exists for attribution. A step is only evidence that something changed if the series it came out of does not move that much by itself, and one number cannot say which. Only confirmed verdicts carry it (they are the only ones anything attributes), and older reports carry none — every reader treats an empty history as "no history recorded", never as a quiet series.

region_deltas answers the other half: where inside the detector a timing step landed, from the per-region timing the k4BenchRegionTimingAction plugin records on every run ({config}_regions.json). Each entry is one top-level detector region's per-event median time on each end of the change window, largest movement first — so "ALLEGRO got 21% slower" becomes "the HCAL barrel went from 0.31 to 4.52 s/event and nothing else moved", which is a claim a code diff can be checked against. Carried on confirmed timing verdicts only (region data is per-event time and says nothing about a memory step), and empty when either end of the window recorded no region file — with only one side measured there is no comparison, and treating the missing side as zero would report the whole detector as newly appearing. A region present on one end only keeps null on the other: it genuinely appeared or disappeared.

Region times are per-event medians: they describe the typical event, and they cannot see a step carried by a handful of long events. event_profile, read from the same files, says which of the two a timing step is:

"event_profile": {
  "base":  {"nights": 1, "n_events": 999, "mean": 0.5277, "median": 0.4302,
            "stepping_mean": 0.5248, "mean_without_longest": 0.4927,
            "longest": [{"event": 37, "seconds": 35.53, "region": "SET",
                         "region_seconds": 30.08}]},
  "onset": {"nights": 2, "n_events": 999, "mean": 0.4939, "median": 0.4279,
            "stepping_mean": 0.4909, "mean_without_longest": 0.4855,
            "longest": [{"event": 949, "seconds": 8.81, "region": "unattributed",
                         "region_seconds": 5.31}]},
  "matched": [{"event": 37, "base": 35.53, "onset": 0.38},
              {"event": 949, "base": 3.14, "onset": 8.81},
              {"event": 240, "base": 3.71, "onset": 3.80}]
}

Each end summarises its per-event wall times, warm-up event excluded: the mean and median, stepping_mean (time inside Geant4 stepping, summed over every region including unattributed, so mean - stepping_mean is the time outside it), the mean with the single longest event left out, and the longest events with the region that took most of each. An end measured on several nights is summarised night by night and the nights combined by their median. matched follows every one of those longest events to the other end: with a fixed random seed the same event number is the same event until the geometry or physics it meets changes, so an event whose time moved (37 above) is one whose simulation changed, and one that held (240) shows the rest of the sample did not. The example is ILD_FCCee_v02's baseline on 2026-09-25: its mean fell 6.4% while its median held, because one 35.5-second event left the sample.

Both fields ride only on confirmed timing verdicts, event_profile only when both ends recorded per-event times, and reports written before it existed carry none — every reader treats a missing profile as unknown.

Why a metric was not judged (`report.json`)¶

Every verdict with severity: "UNKNOWN" may carry an unjudged discriminator that explains why no judgement was made:

insufficient_history — too few settled baseline runs were available;
unreliable_host — the run failed the host-reliability check; or
reported_only — the metric is never judged by design, either because it duplicates a measurement another metric already judges (user_cpu_s tracks wall_time_s) or because its value is not reproducible enough to judge (peak_rss_mb, mean_rss_mb and mean_rss_file_mb count file-backed pages, which a CVMFS publish can evict from a running process), or because it is still collecting the history a judgement policy needs (rss_anon_slope_mb_per_event).

The field is null on judged verdicts. Older reports may omit it entirely, and readers must also tolerate values introduced by newer writers that they do not yet recognise. In either case, readers may use the verdict's reason text as a compatibility fallback, but must not assume every UNKNOWN means insufficient history.

Blame sidecar (`blame.json`)¶

For each confirmed regression whose blame window spans two different releases, blame.json records the repositories that moved across that window and the pull requests that could have caused it, ranked by how well each matches the regression. It is a sidecar, deliberately separate from report.json: blame needs GitHub, and a GitHub outage, a rate limit, or a force-pushed develop must never degrade or fail the nightly report and its email. Different failure domain, different file — written best-effort by blame_report.py after report.json is uploaded, and absent entirely on most nights (most nights have no confirmed, attributable regression).

{
  "generated_at": "2026-07-05T06:10:00+00:00",
  "report_night": "2026-07-05",
  "entries": [
    {
      "detector": "ALLEGRO_o1_v03", "platform": "…", "sample": "single_e-_10GeV",
      "label": "baseline", "metric": "wall_time_s", "sub_detector": null,
      "base_release": "2026-07-03", "onset_release": "2026-07-04",
      "n_unchanged": 60,
      "boundary_changes": {"2026-07-03": 0, "2026-07-04": 2},
      "assessment": {
        "verdict": "real_change",
        "reason": "flat within ±0.4% for six releases, and the new level held"
      },
      "repos": [
        {
          "package": "k4geo", "repo": "key4hep/k4geo",
          "base_commit": "0f226a98…", "head_commit": "21647280…",
          "compare_url": "https://github.com/key4hep/k4geo/compare/0f226a98…...21647280…",
          "status": "changed", "commits_unavailable": false, "truncated": false,
          "candidates": [
            {
              "repo": "key4hep/k4geo", "number": 1234,
              "title": "Lower the tracker step limit", "author": "…",
              "url": "https://github.com/key4hep/k4geo/pull/1234",
              "merged_at": "2026-07-04T…", "files": ["FCCee/ALLEGRO/…"],
              "additions": 20, "deletions": 4,
              "score": 72, "description": "raises the tracker step count, plausibly slower",
              "against": "the no_Tracker run stepped by the same amount"
            }
          ]
        }
      ]
    }
  ]
}

Each entry's first seven fields are a report.json verdict's identity; the dashboard joins an entry back to the confirmed regression it explains by that identity and the base_release/onset_release window, so a sidecar left over from an earlier build of the same night (the CI job also deletes the remote sidecar on a rerun that produces none) can never attach to a regression whose window it did not examine. The pipeline collects every PR in each changed repo's commit range; a separate ranking stage then scores each candidate for that group — score is a 0–100 likelihood it is the cause, description a one-line reason and against (optional) what the model said argues against it, judged once per detector/platform/sample/window (every metric — and every benchmark-config label, e.g. a removal sweep's baseline vs. no_<detector> — sharing that group and window shares one ranking, applied to all of them). A different detector or sample sharing the same release dates never shares a ranking. commits_unavailable marks a repo whose range could not be enumerated at all; truncated marks a candidate list known to be incomplete (compare/PR caps, or a PR that failed to fetch) — a regression touching either is left unranked, since "most likely" over a partial candidate set would overclaim. The ranking is a lead for a human, never a claim of cause. Readers drop unknown keys, so the schema can gain fields without breaking an older dashboard; structurally malformed sidecars are hidden, never fatal.

boundary_changes maps a release in the metric's history tail to the number of tracked packages that moved entering it. It is the one piece of the ranker's evidence the cross-configuration pass cannot recompute — that pass runs from the report and this sidecar with no provenance access — so it is persisted rather than derived twice. A release absent from the map is unread, never unchanged: 0 says the software was identical across that boundary and the metric moved anyway (the sharpest measurement of a series' own noise this suite produces), while a missing key says nobody looked.

assessment is the ranker's judgement of the movement itself, before any question of who caused it: real_change, likely_noise, or insufficient_evidence, with a one-line reason. It is shared by every entry of a rank group, and absent (null) when no model was configured, when the reply gave none, or on a sidecar written before the field existed — absent means not assessed, never real_change. A likely_noise window is still ranked, written and rendered on the dashboard and in the email, each carrying one line of the verdict beside the candidates; what it does not do is produce a pull-request comment, since an accusation in someone else's repository about a wobble is the most expensive mistake this pipeline can make.

The ranking stage is a language model that reads the metric that moved, its recent release-by-release history, where inside the detector the time went, the configurations that measured the same window without moving, how much of the tracked stack stood still, and each candidate PR's own description and code diff (descriptions and diffs both arrive fenced as untrusted data). A diff is sampled with the run's own compact directory first, then its detector's geometry tree, and a candidate touching that directory is served its diff budget first. It is configured entirely by environment — K4BENCH_LLM_URL, K4BENCH_LLM_MODEL, K4BENCH_LLM_API_KEY and optional K4BENCH_LLM_MAX_TOKENS (any OpenAI-compatible /chat/completions endpoint; the model is a config value, not pinned in code). Transient connection, timeout, HTTP 429 and HTTP 5xx failures retry with bounded backoff; length-truncated responses grow the output allowance up to a fixed ceiling. The complete blame stage also has a CI wall-clock limit. None of these failures can fail or delay the already-uploaded nightly report beyond that bound.

Ranking is optional: with endpoint/model unset, candidates are still written but left unranked (score 0, description ""), the dashboard shows the package diff without the candidate ledger, and the email omits the "most likely" line. When ranking is configured, CI publishes blame.json only if every candidate of every fully-discovered regression has an explanation (a score of zero remains valid; a likelihood that is not a number rejects that row rather than becoming a fake 0%); an empty or partial model response is logged and the sidecar is skipped rather than silently publishing a ranking the dashboard would hide. The diffs the model reads are transient input, never stored here (they are re-fetchable from GitHub); the sidecar keeps only the file paths plus the ranker's score/description. The model may only score the candidates it is given — a PR number it did not receive is dropped, so blame.json can never surface an invented PR.

On-demand historical evidence¶

historical_evidence records the pull requests from older release boundaries that the ranker asked to read before judging this window. Each entry is a reference — boundary_id, the boundary's base_release/onset_release, the package, the repo slug, the pr number, its title, changed files and churn — and never a patch or a description: those are re-fetchable from GitHub forever, and the cross-configuration pass fetches them again from exactly this reference. Absent on every sidecar written before the field existed, and empty on every ranking that used none.

The retrieval is a two-stage protocol, not a browsing tool:

The ranker's prompt carries a lightweight index of the older boundaries in the metric's own history tail — package names, repository slugs, add/change/remove status — under application-generated opaque ids (h1, h2, …). Building it costs no GitHub call. Boundaries whose release diff could not be read are listed as unreadable rather than omitted, because a gap in a list of dates would read as a boundary where nothing changed. Both listing caps — 8 boundaries, 25 packages per boundary — are stated when they bite ("showing 25 of 37 changed packages; the other 12 are not listed and cannot be requested"), for the same reason: a shortened list that does not admit to being short is read as a complete one, and would let a display bound exculpate a package nobody measured.
The model may answer with a historical_evidence_request naming ids and package names from that index only. An invented id, a package nobody offered, a boundary the index called unreadable, a request with no stated reason, or a selection past a cap is a decline: the window is left unranked, and any preliminary rankings written alongside the request are discarded — the model said it wanted the code before judging, so its judgement without the code is not the one to publish.
The application retrieves the validated selection through the same authenticated GitHub client and the same (repo, base, head) resolution cache the current window's candidates used, then asks once more with the code attached. That answer is the authoritative ranking — unless it asks again, which is a decline: there is no round left to honour it with, so a reply that says it is still not ready to judge does not get its scores published beside the statement. (A historical PR body is attacker-reachable prose and can try to induce that member; all it buys is a refusal.)

Bounds: at most 2 boundaries and 2 packages per boundary per request, 4 pull requests in total, one retrieval round, and one extra model call per rank group. The historical diffs get their own rendering budget (10 000 chars, waterfilled) so they can never take a character from the current window's candidates. Any way the retrieval falls short — a rate limit, a 404, an incomplete PR discovery or changed-file pagination, more pull requests than the cap can read completely — leaves the window unranked rather than ranked on a partial view of evidence the model itself said mattered.

Historical pull requests are analogues, never candidates. They shipped before the window opened, so they cannot have caused it: they never enter RepoBlame.candidates, the ranking-coverage gate, the dashboard's candidate ledger, or a comment target, and one echoed back as a ranking is dropped by the existing only-reorder rule. Their descriptions and diffs are fenced as untrusted input exactly like a current candidate's.

The feature is on by default wherever ranking and GITHUB_TOKEN are configured. K4BENCH_LLM_HISTORICAL_DIFFS=0 (or false/no/off) turns it off, and off means the prompts, model-call count, GitHub-call count and artifacts are exactly what they were before the feature existed.

When a comment is written about such a window, the cross-configuration review is handed the same analogues, re-fetched from these references and rendered under the same "historical, not a candidate" label — a review that revised the first pass without the evidence the first pass rested on would not be a second opinion. If a reference yields neither a diff nor a description it is unreadable and no comment is posted that night, the same fail-closed rule an unusable review already follows. (An empty patch alone is not a failure: a binary-only or pure-rename pull request has no textual hunk, and the first pass accepted it on its paths and prose.) One comment carries at most 12 analogues in total — MAX_PRS bounds one rank group, but a comment window unions every rank group inside it — and exceeding that suppresses the comment before any fetch rather than dropping analogues, since dropping some would silently leave the two passes weighing different evidence. The references are part of the comment's facts digest, so a materially different evidence set produces a replaceable comment rather than a frozen one; the analogues themselves are never rendered as accused pull requests in the public body.