Data flow¶
How data moves through the system at runtime — first a single sweep, then the nightly CI → EOS → dashboard pipeline.
A sweep, end to end¶
sequenceDiagram
autonumber
actor User
participant CLI as cli
participant B as benchmark
participant G as geometry
participant E as runner
participant DD as ddsim (child proc)
participant R as results
User->>CLI: k4bench --xml … --sweep --ddsim-args="…"
CLI->>B: run_sweep(config)
B->>G: scan detectors
B->>E: baseline run (original XML)
loop per detector
B->>G: patch geometry (temp XML, detector removed)
B->>E: run (patched XML)
E->>DD: /usr/bin/time -v ddsim …
DD-->>E: stdout → log; optional plugin JSON
E-->>B: RunResult
B->>G: patch directory cleaned up
end
B-->>CLI: list[RunResult]
CLI->>R: print summary + write CSV
CLI-->>User: table + exit code
Key points along the path:
- The temporary EDM4hep output file is reused across runs — only its size is recorded.
- Run-level metrics are scraped from the
/usr/bin/time -vblock in the log; optional per-event / per-detector JSON is written directly by the timing plugins. - Each patched geometry lives in one private temp directory only for the duration of its run.
The instrumentation/physics split that makes this possible is described in the architecture overview.
Nightly CI → EOS → dashboard¶
sequenceDiagram
autonumber
participant Cron as nightly (cron)
participant Gate as resolve-release job
participant Job as benchmark job
participant K as k4bench
participant EOS as CERN EOS
participant Reg as regression-report job
participant Mail as CERN e-group
participant Dash as dashboard
Cron->>Job: expand .github/benchmarks/{suite}/{concept}/*.yml → matrix
Cron->>Gate: wait for today's Key4hep stack on CVMFS
Gate-->>Job: release every job of this night sources
Job->>K: run (per config/sample, from benchmark_job.py)
K-->>Job: logs/benchmark/ (CSV + JSON + log)
Job->>Job: write run_info.json + machine_info.json
Job->>EOS: upload to runs/{suite}/{concept}/{detector}/{platform}/key4hep-{release}/{sample}/{date}/
Reg->>EOS: pull trailing sim run window per (detector, platform, sample)
Reg->>Reg: k4bench.regression: reliability-filtered step detection
Reg->>EOS: upload reports/sim/{date}/report.json
Reg-->>Mail: email on confirmed regressions/failures only
Dash->>EOS: list runs, download the files a view reads over HTTPS (cached on disk)
Dash->>Dash: load via k4bench.analysis; render tabs
Dash->>EOS: fetch reports/sim/{date}/report.json (Regressions tab)
The resolve-release job (.github/scripts/resolve_release.sh) names one
Key4hep release per night and the whole fan-out sources exactly that release, so
samples of the same night can never be filed under two releases. A view counts
only once its build has finished: LCG can publish a view before the packages it
links are built, so the gate also requires the slot's
nightlies/…/isDone-<platform> marker, written no later than the view. When no
complete stack is published for the day it falls back to the newest complete one
and says so in the job summary; the night still runs, because a repeat
measurement of a release is what k4bench/regression/engine.py uses to confirm a
WATCH.
The EOS directory layout is the integration contract between CI and the
dashboard — see File formats → EOS layout.
Because historical runs are immutable, the dashboard downloads each file of a
run at most once, and only when a view reads it (k4bench.remote.ensure_run_artifacts):
each file is published into its on-disk cache atomically, under a per-run lock
and manifest, so concurrent reruns never see a half-written file and two views
asking for different files of one run both get theirs. The nightly report and
the CLI download whole runs (ensure_run_cached), published with a single
atomic rename of the run directory and marked .complete.
The regression-report job (.github/scripts/regression_report.{py,sh}) runs
after the benchmark fan-out, so a crashed detector job still surfaces in the
report — its missing upload is the failure signal. It judges the suite its
suite input names, reading that suite's run tree alone and writing that
suite's report; the nightly runs it once for each suite its fan-out
benchmarked. A fan-out where every
job failed, or that never started, uploads nothing at all and would otherwise
republish the previous night, so nightly.yml passes its own Actions run id:
a report naming none of that run's measurements is reported as tonight's
outage instead (see
File formats → The night a report covers).
It reuses the same pure building blocks as the dashboard (k4bench.remote for
discovery/download, k4bench.analysis.trend for aggregation,
k4bench.results.reliability_evidence for the per-run reliability verdict) and
adds the step detector in k4bench/regression/engine.py. The precomputed
reports/sim/{date}/report.json is what the dashboard's Regressions tab renders —
it never recomputes verdicts live.
Deployment¶
The dashboard is containerised (dashboard/Dockerfile), pushed to ghcr.io,
and rolled out on CERN's OpenShift PaaS via the openshift/ manifests by the
deploy-dashboard.yml workflow. It serves at
k4bench-dashboard.app.cern.ch.
See also¶
- Component diagrams — the static structure.
- File formats — every artifact's schema.