Benchmark

0.4.3

Swift benchmark runner as a SwiftPM command plug-in with CI support.
ordo-one/package-benchmark

What's New

0.4.3

2022-10-28T08:10:46Z

What's Changed

  • chore: Update documentation by @hassila in #38
  • chore: Move to dispatch conditions / NIOLock instead of raw pthread mutex/conditions by @hassila in #39

Full Changelog: 0.4.2...0.4.3

Swift version Swift build and test Swift address sanitizer codecov

Benchmark

Introduction

Benchmark is a harness for easily creating Swift performance benchmarks for both macOS and Linux.

It's intended to be suitable for both ad-hoc smaller benchmarks primarily caring about runtime (in the spirit of Google's swift-benchmark) as well for more extensive benchmarks caring about additional benchmark metrics such as memory allocations, syscalls, thread usage and more.

Benchmark supports both local usage with baseline comparisons for an iterative workflow for the individual developer, but more importantly has good support for integration with GitHub CI with provided sample workflows for automated comparisons between main and a pull request branch to support enforced performance validation for pull requests with customizable thresholds - this is the primary intended use case for the package.

The focus for measurements are percentiles (p0 (min), p25, p50 (median), p75, p90, p99 and p100 (max)) to support analysis of the actual distribution of benchmark measurements. A given benchmark is typically run for a minimum amount of time and/or a given number of iterations, see details in the Benchmark documentation below.

Minimal benchmark + benchmark using async / Swift Concurrency

import BenchmarkSupport
@main extension BenchmarkRunner {}
@_dynamicReplacement(for: registerBenchmarks)

func benchmarks() {

    Benchmark("Minimal benchmark") { benchmark in
      // measure something here
    }

    Benchmark("All metrics, full concurrency, async",
              metrics: BenchmarkMetric.all,
              desiredDuration: .seconds(10)) { benchmark in
        let _ = await withTaskGroup(of: Void.self, returning: Void.self, body: { taskGroup in
            for _ in 0..< 80  {
                taskGroup.addTask {
                    dummyCounter(defaultCounter()*1000)
                }
            }
            for await _ in taskGroup {
            }
        })
    }
}

Running benchmarks

To execute all defined benchmarks, simply run:

swift package benchmark

See the detailed documentation links below for extended usage including delta comparisons and baseline storage etc.

Sample output benchmark run

image

Sample output delta comparison

image

Source and file format stability

The source and file format of baselines are not officially stable yet until release 1.0.0, even though no majors changes are planned currently, there might be source and file format breaking changes in minor releases (not patch releases) until then.

Contents

There's also a sample project using various aspects of this package for those who just want to see how it can be used in practice

Description

  • Swift Tools 5.6.0
View More Packages from this Author

Dependencies

Last updated: Mon Nov 21 2022 10:33:31 GMT-0500 (GMT-05:00)