Running a benchmark suite

Use benchmarkpkg to run benchmarks defined in a suite as defined in the previous section.

PkgBenchmark.benchmarkpkg — Function.

benchmarkpkg(pkg, [target]::Union{String, BenchmarkConfig}; kwargs...)

Run a benchmark on the package pkg using the BenchmarkConfig or git identifier target. Examples of git identifiers are commit shas, branch names, or e.g. "HEAD~1". Return a BenchmarkResults.

The argument pkg can be a name of a package or a path to a directory to a package.

Keyword arguments:

script - The script with the benchmarks, if not given, defaults to benchmark/benchmarks.jl in the package folder.
resultfile - If set, saves the output to resultfile
retune - Force a re-tune, saving the new tuning to the tune file.

The result can be used by functions such as judge. If you choose to, you can save the results manually using writeresults where results is the return value of this function. It can be read back with readresults.

If a REQUIRE file exists in the same folder as script, load package requirements from that file before benchmarking.

Example invocations:

using PkgBenchmark

benchmarkpkg("MyPkg") # run the benchmarks at the current state of the repository
benchmarkpkg("MyPkg", "my-feature") # run the benchmarks for a particular branch/commit/tag
benchmarkpkg("MyPkg", "my-feature"; script="/home/me/mycustombenchmark.jl")
benchmarkpkg("MyPkg", BenchmarkConfig(id = "my-feature",
                                      env = Dict("JULIA_NUM_THREADS" => 4),
                                      juliacmd = `julia -O3`))

source

The results of a benchmark is returned as a BenchmarkResult

PkgBenchmark.BenchmarkResults — Type.

Stores the results from running the benchmarks on a package.

The following (unexported) methods are defined on a BenchmarkResults (written below as results):

name(results)::String - The commit of the package benchmarked
commit(results)::String - The commit of the package benchmarked. If the package repository was dirty, the string "dirty" is returned.
juliacommit(results)::String - The commit of the Julia executable that ran the benchmarks
benchmarkgroup(results)::BenchmarkGroup - a BenchmarkGroup contaning the results of the benchmark.
date(results)::DateTime - Tthe time when the benchmarks were executed
benchmarkconfig(results)::BenchmarkConfig - The BenchmarkConfig used for the benchmarks.

BenchmarkResults can be exported to markdown using the function export_markdown.

source

More advanced customization

Instead of passing a commit, branch etc. as a String to benchmarkpkg, a BenchmarkConfig can be passed. This object contains the package commit, julia command, and what environment variables will be used when benchmarking. The default values can be seen by using the default constructor

julia> BenchmarkConfig()
BenchmarkConfig:
    id: nothing
    juliacmd: `/home/user/julia/julia`
    env:

The id is a commit, branch etc as described in the previous section. An id with value nothing means that the current state of the package will be benchmarked. The default value of juliacmd is joinpath(JULIA_HOME, Base.julia_exename() which is the command to run the julia executable without any command line arguments.

To instead benchmark the branch PR, using the julia command julia -O3 with the environment variable JULIA_NUM_THREADS set to 4, the config would be created as

julia> config = BenchmarkConfig(id = "PR",
                                juliacmd = `julia -O3`,
                                env = Dict("JULIA_NUM_THREADS" => 4))
BenchmarkConfig:
    id: PR
    juliacmd: `julia -O3`
    env: JULIA_NUM_THREADS => 4

To benchmark the package with the config, call benchmarkpkg as e.g.

benchmark("Tensors", config)

Info

The id keyword to the BenchmarkConfig does not have to be a branch, it can be most things that git can understand, for example a commit id or a tag.