Skip to content

k4bench.staging

k4bench.staging

Where the nightly copies a k4run job's inputs and data files before it runs.

The nightly job (.github/scripts/benchmark_job.py) copies every input and data file of a k4run job to a local path and passes that path to the job, so the k4run_args a run records name these paths. A reproducer that fetches the same sources to the same paths (:mod:k4bench.blame.reproduce) can therefore run the recorded arguments as they are. Like :mod:k4bench.artifacts this is a leaf module, so the nightly and the blame tooling share one spelling without importing each other.

input_key

input_key(source: str) -> str

The directory below :data:LOCAL_INPUT_DIR one input is copied into.

Named by a hash of the URL, so two inputs that share a file name never overwrite each other.

Source code in k4bench/staging.py
def input_key(source: str) -> str:
    """The directory below :data:`LOCAL_INPUT_DIR` one input is copied into.

    Named by a hash of the URL, so two inputs that share a file name never
    overwrite each other."""
    return hashlib.sha256(source.encode()).hexdigest()[:16]

data_key

data_key(sources: list[str]) -> str

The directory below :data:LOCAL_DATA_DIR a job's data files share, named by a hash of all of them.

Source code in k4bench/staging.py
def data_key(sources: list[str]) -> str:
    """The directory below :data:`LOCAL_DATA_DIR` a job's data files share,
    named by a hash of all of them."""
    return hashlib.sha256(" ".join(sources).encode()).hexdigest()[:16]

download_argv

download_argv(source: str, local: str) -> list[str]

The command copying the root:// (xrdcp) or http(s):// (curl) source to local.

Source code in k4bench/staging.py
def download_argv(source: str, local: str) -> list[str]:
    """The command copying the ``root://`` (xrdcp) or ``http(s)://`` (curl)
    *source* to *local*."""
    if source.startswith("root://"):
        return ["xrdcp", "--force", source, local]
    return ["curl", "-fsSL", "--retry", "3", "--retry-delay", "10", "-o", local, source]