Running a benchmark suite

benchmarkpkg(pkg, [target]::Union{String, BenchmarkConfig}; kwargs...)

Run a benchmark on the package pkg using the BenchmarkConfig or git identifier target. Examples of git identifiers are commit shas, branch names, or e.g. "HEAD~1". Return a BenchmarkResults.

The argument pkg can be the module of a package, a package name, or the path to the package's root directory.

Keyword arguments:

• script - The script with the benchmarks, if not given, defaults to benchmark/benchmarks.jl in the package folder.
• postprocess - A function to post-process results. Will be passed the BenchmarkGroup, which it can modify, or return a new one.
• resultfile - If set, saves the output to resultfile
• retune - Force a re-tune, saving the new tuning to the tune file.
• verbose::Bool = true - Print currently running benchmark.
• logger_factory - Specify the logger used during benchmark. It is a callable object (typically a type) with no argument that creates a logger. It must exist as a constant in some package (e.g., an anonymous function does not work).
• progressoptions - Deprecated.

The result can be used by functions such as judge. If you choose to, you can save the results manually using writeresults where results is the return value of this function. It can be read back with readresults.

Example invocations:

using PkgBenchmark

import MyPkg
benchmarkpkg(MyPkg) # run the benchmarks at the current state of the repository
benchmarkpkg(MyPkg, "my-feature") # run the benchmarks for a particular branch/commit/tag
benchmarkpkg(MyPkg, "my-feature"; script="/home/me/mycustombenchmark.jl")
benchmarkpkg(MyPkg, BenchmarkConfig(id = "my-feature",
                                            env = Dict("JULIA_NUM_THREADS" => 4),
                                            juliacmd = `julia -O3`))
benchmarkpkg(MyPkg,  # Run the benchmarks and divide the (median of) results by 1000
    postprocess=(results)->(results["g"] = median(results["g"])/1_000)

Stores the results from running the benchmarks on a package.

The following (unexported) methods are defined on a BenchmarkResults (written below as results):

• name(results)::String - The commit of the package benchmarked
• commit(results)::String - The commit of the package benchmarked. If the package repository was dirty, the string "dirty" is returned.
• juliacommit(results)::String - The commit of the Julia executable that ran the benchmarks
• benchmarkgroup(results)::BenchmarkGroup - a BenchmarkGroup contaning the results of the benchmark.
• date(results)::DateTime - The time when the benchmarks were executed
• benchmarkconfig(results)::BenchmarkConfig - The BenchmarkConfig used for the benchmarks.

BenchmarkResults can be exported to markdown using the function export_markdown.

BenchmarkConfig(;id::Union{String, Nothing} = nothing,
                 juliacmd::Cmd = `joinpath(Sys.BINDIR, Base.julia_exename())`,
                 env::Dict{String, Any} = Dict{String, Any}())

A BenchmarkConfig contains the configuration for the benchmarks to be executed by benchmarkpkg.

This includes the following:

• The commit of the package the benchmarks are run on.
• What julia command should be run, i.e. the path to the Julia executable and

the command flags used (e.g. optimization level with -O).

• Custom environment variables (e.g. JULIA_NUM_THREADS).

The constructor takes the following keyword arguments:

• id - A git identifier like a commit, branch, tag, "HEAD", "HEAD~1" etc. If id == nothing then benchmark will be done on the current state of the repo (even if it is dirty).
• juliacmd - Used to execute the benchmarks, defaults to the julia executable that the Pkgbenchmark-functions are called from. Can also include command flags.
• env - Contains custom environment variables that will be active when the benchmarks are run.

Examples

julia> using Pkgbenchmark

julia> BenchmarkConfig(id = "performance_improvements",
                       juliacmd = `julia -O3`,
                       env = Dict("JULIA_NUM_THREADS" => 4))
BenchmarkConfig:
    id: performance_improvements
    juliacmd: `julia -O3`
    env: JULIA_NUM_THREADS => 4

readresults(file::String)

Reads the BenchmarkResults stored in file (given as a path).

writeresults(file::String, results::BenchmarkResults)

Writes the BenchmarkResults to file.