How nanobench Measures and Reports Benchmark Results

nanobench measures a benchmark by repeatedly running a timed loop (an epoch) until it has a stable sample, reduces each round to a single number, and reports that number with an error estimate rather than a bare average. It is designed to make microbenchmark results trustworthy enough to compare alternatives, and it reports the statistics, histograms, and significance tests needed to judge that trust. The approach fits C++ microbenchmarks where you want defensible numbers without building your own statistics layer; it is not a profiler and does not attribute time to source lines.

Design goals: one number per round, with an estimate

The documentation frames the core problem directly: a benchmark produces many timings, and the reader needs one number plus a sense of how much to trust it. nanobench's stated design goals center on reducing a round to one number and estimating and reporting the uncertainty around it. Everything else in the output — columns, error, comparisons — serves that goal.

This matters because a raw mean hides instability. If a benchmark's timings swing widely, the mean looks precise while being meaningless. nanobench instead reports the central value alongside an error figure so an unstable result is visibly unstable.

How a measurement round works

The ankerl::nanobench::Bench object is the main entry point. You configure it, then call run() with a name and a callable. The relevant knobs for experiment design are:

• warmup() — number of warmup iterations before measurement.
• epochs() — how many measurement epochs to collect.
• minEpochTime() / maxEpochTime() — bounds on how long a single epoch runs.
• minEpochIterations() / epochIterations() — bounds on iterations per epoch.
• clockResolutionMultiple() — ensures each epoch spans enough clock ticks to be measurable.
• batch() — number of operations per iteration, used to compute per-operation cost.
• targetIntervalWidth() / maxEpochs() — control how tightly the collected samples must cluster before stopping.

The practical effect: nanobench keeps sampling until the spread of epochs is narrow enough (or a limit is hit), then reports the result. You are not choosing a fixed iteration count and hoping; the library adapts to how long the code actually takes.

Warmup and serial correlation

The documentation is explicit that warmup is not the answer to serial correlation. Warmup helps with one-time costs (caches, lazy initialization), but if consecutive measurements are correlated — each influenced by the previous — warmup does not remove that dependence. The docs discuss serial correlation as a distinct concern and explain what was rejected in designing around it. The takeaway for a reader: treat warmup as handling startup effects, not as a fix for a benchmark whose successive runs drift.

What the output contains

Results are exposed through Result objects and rendered through templates. The column set is configurable: showColumn(), hideColumn(), isColumnVisible(), and contextColumn() let you choose what appears. Documented columns include:

Performance counters require performanceCounters() to be enabled and depend on platform support.

Statistics and histograms

Show Benchmark Statistics and Show a Histogram are documented output modes. The statistics view gives the summary numbers; the histogram shows the distribution of samples, which is where instability becomes visible. If the histogram is wide or multi-modal, the single reported number deserves skepticism — that is the point of showing it.

Rendering formats

Output is produced by Mustache-like templates via render(). Built-in templates include:

• templates::csv — comma-separated values, including a comparison variant
• templates::json — machine-readable output
• templates::htmlBoxplot — box plots for visual comparison
• templates::pyperf — Python pyperf-compatible output

You can also supply your own template. Time units inside templates are configurable through timeUnit().

Comparing alternatives and judging significance

Bench::compare() runs a comparison between implementations and returns a CompareResult. The API exposes size(), operator[], rounds(), comparisons(), isSignificant(), and fastest(). The isSignificant() flag is the key one: it tells you whether the observed difference is distinguishable from noise given the collected samples. A comparison that is not significant should not be reported as "faster."

The documentation also asks "How precise is a comparison?" — a reminder that even a significant result has a resolution limit. Two implementations within the error bars are, for practical purposes, the same.

Estimating asymptotic complexity

nanobench can estimate complexity growth using complexityN() and complexityBigO(). You run the benchmark across a range of input sizes (set via the size context), and the library fits the growth. The complexityN and relative complexity columns report the fit. This is an estimate from measured points, not a proof — it tells you whether your data looks like O(n) or O(n log n), and a poor fit signals that the model does not describe the behavior.

A concrete example of the workflow

A task where this matters: deciding whether a custom hash map beats std::unordered_map for your workload. You would:

1. Create a Bench, set warmup(), epochs(), and a timeUnit().
2. Call run() for each implementation with the same input sizes, using context() to record the size.
3. Call compare() to get a CompareResult.
4. Check isSignificant() before claiming a winner, and inspect the histogram if the result looks unstable.

If the comparison is not significant, the honest conclusion is that they perform the same at that size — which is itself a useful decision input.

Environment controls

Three environment variables affect runs without recompiling:

• NANOBENCH_ENDLESS — run a specific test endlessly (useful for profiling).
• NANOBENCH_SUPPRESS_WARNINGS — suppress stability warnings.
• NANOBENCH_CONFIG — change settings without recompiling.

The stability warnings are worth heeding: they fire when the collected samples are too spread out to trust, which is exactly the condition under which a reported number should not be used for a decision.