nanobench.ankerl.com

nanobench measures a benchmark by repeatedly running a timed loop (an epoch) until it has a stable sample, reduces each round to a single number, and reports that number with an error estimate rather than a bare average. It is designed to make microbenchmark results trustworthy enough to compare alternatives, and it reports the statistics, histograms, and significance tests needed to judge that trust. The approach fits C++ microbenchmarks where you want defensible numbers without building your own statistics layer; it is not a profiler and does not attribute time to source lines.

Design goals: one number per round, with an estimate

The documentation frames the core problem directly: a benchmark produces many timings, and the reader needs one number plus a sense of how much to trust it. nanobench's stated design goals center on reducing a round to one number and estimating and reporting the uncertainty around it. Everything else in the output — columns, error, comparisons — serves that goal.

This matters because a raw mean hides instability. If a benchmark's timings swing widely, the mean looks precise while being meaningless. nanobench instead reports the central value alongside an error figure so an unstable result is visibly unstable.

How a measurement round works

The ankerl::nanobench::Bench object is the main entry point. You configure it, then call run() with a name and a callable. The relevant knobs for experiment design are:

• warmup() — number of warmup iterations before measurement.
• epochs() — how many measurement epochs to collect.
• minEpochTime() / maxEpochTime() — bounds on how long a single epoch runs.
• minEpochIterations() / epochIterations() — bounds on iterations per epoch.
• clockResolutionMultiple() — ensures each epoch spans enough clock ticks to be measurable.
• batch() — number of operations per iteration, used to compute per-operation cost.
• targetIntervalWidth() / maxEpochs() — control how tightly the collected samples must cluster before stopping.

The practical effect: nanobench keeps sampling until the spread of epochs is narrow enough (or a limit is hit), then reports the result. You are not choosing a fixed iteration count and hoping; the library adapts to how long the code actually takes.

Warmup and serial correlation

The documentation is explicit that warmup is not the answer to serial correlation. Warmup helps with one-time costs (caches, lazy initialization), but if consecutive measurements are correlated — each influenced by the previous — warmup does not remove that dependence. The docs discuss serial correlation as a distinct concern and explain what was rejected in designing around it. The takeaway for a reader: treat warmup as handling startup effects, not as a fix for a benchmark whose successive runs drift.

What the output contains

Results are exposed through Result objects and rendered through templates. The column set is configurable: showColumn(), hideColumn(), isColumnVisible(), and contextColumn() let you choose what appears. Documented columns include:

Performance counters require performanceCounters() to be enabled and depend on platform support.

Statistics and histograms

Show Benchmark Statistics and Show a Histogram are documented output modes. The statistics view gives the summary numbers; the histogram shows the distribution of samples, which is where instability becomes visible. If the histogram is wide or multi-modal, the single reported number deserves skepticism — that is the point of showing it.

Rendering formats

Output is produced by Mustache-like templates via render(). Built-in templates include:

• templates::csv — comma-separated values, including a comparison variant
• templates::json — machine-readable output
• templates::htmlBoxplot — box plots for visual comparison
• templates::pyperf — Python pyperf-compatible output

You can also supply your own template. Time units inside templates are configurable through timeUnit().

Comparing alternatives and judging significance

Bench::compare() runs a comparison between implementations and returns a CompareResult. The API exposes size(), operator[], rounds(), comparisons(), isSignificant(), and fastest(). The isSignificant() flag is the key one: it tells you whether the observed difference is distinguishable from noise given the collected samples. A comparison that is not significant should not be reported as "faster."

The documentation also asks "How precise is a comparison?" — a reminder that even a significant result has a resolution limit. Two implementations within the error bars are, for practical purposes, the same.

Estimating asymptotic complexity

nanobench can estimate complexity growth using complexityN() and complexityBigO(). You run the benchmark across a range of input sizes (set via the size context), and the library fits the growth. The complexityN and relative complexity columns report the fit. This is an estimate from measured points, not a proof — it tells you whether your data looks like O(n) or O(n log n), and a poor fit signals that the model does not describe the behavior.

A concrete example of the workflow

A task where this matters: deciding whether a custom hash map beats std::unordered_map for your workload. You would:

1. Create a Bench, set warmup(), epochs(), and a timeUnit().
2. Call run() for each implementation with the same input sizes, using context() to record the size.
3. Call compare() to get a CompareResult.
4. Check isSignificant() before claiming a winner, and inspect the histogram if the result looks unstable.

If the comparison is not significant, the honest conclusion is that they perform the same at that size — which is itself a useful decision input.

Environment controls

Three environment variables affect runs without recompiling:

• NANOBENCH_ENDLESS — run a specific test endlessly (useful for profiling).
• NANOBENCH_SUPPRESS_WARNINGS — suppress stability warnings.
• NANOBENCH_CONFIG — change settings without recompiling.

The stability warnings are worth heeding: they fire when the collected samples are too spread out to trust, which is exactly the condition under which a reported number should not be used for a decision.

nanobench supports four built-in output formats—CSV, JSON, HTML box plots, and Python's pyperf format—plus custom rendering through Mustache-like templates. You get these by calling render() on a benchmark result or comparison result and passing a template from the ankerl::nanobench::templates namespace. The default console output is a table whose columns you can control; the template formats are for feeding results into other tools or reports.

Default table output and column control

When you run a benchmark without rendering a template, nanobench prints a table. The documentation lists these columns: relative, complexityN, timePerUnit, unitPerSecond, error, instructions, cycles, ipc, branches, branchMisses, total, and _size.

You can shape this table with hideColumn(), showColumn(), and isColumnVisible(). Context values can also be promoted into their own columns via contextColumn() and clearContextColumns(), which is useful when you sweep a parameter and want each value visible per row.

Built-in templates

All built-in templates live in ankerl::nanobench::templates and are passed to render():

The documentation also covers CSV specifically for comparisons ("CSV for a comparison") and a section on how precise a comparison is, so if you plan to compare alternatives, read those before trusting small differences.

Rendering a result

The pattern is: run the benchmark, keep the Result, then render it with a template. Conceptually:

ankerl::nanobench::Bench b;
b.title("example").unit("op");
b.run("something fast", [&] { /* work */ });

// Render the most recent result
ankerl::nanobench::render(ankerl::nanobench::templates::json(), b.results());

Bench::results() returns the collected Result objects, and render() accepts a template plus results. For comparisons, CompareResult has its own render() overload, so you render the comparison outcome rather than individual results.

Custom Mustache-like templates

Beyond the built-ins, render() accepts Mustache-like templates, so you can produce any text format—Markdown tables, custom logs, config fragments—by writing a template with placeholders. The documentation's "Rendering" and "Mustache-like Templates" sections describe the syntax, and "Time Units in Templates" explains how time values are formatted inside templates. If you need a unit other than the default in your output, set it with timeUnit() on the bench or handle it in the template.

Practical notes

• Comparisons need the comparison path. Use compare() and render the resulting CompareResult; isSignificant() and fastest() tell you whether a difference is meaningful before you publish it.
• CSV and JSON are the safe interchange formats. If another tool will read the output, prefer these over scraping the console table.
• Box plots need multiple rounds. The HTML box plot template is only informative when the benchmark ran enough epochs to show spread; check epochs() and related settings if the plot looks degenerate.
• pyperf output targets Python tooling. Use it when your analysis pipeline already uses the pyperf module; otherwise JSON is more general.

For exact template names, placeholder syntax, and the comparison-specific CSV behavior, consult the corresponding sections of the nanobench documentation, since the API reference lists each template function individually.

nanobench is a C++ microbenchmarking library for measuring small code snippets. It is header-only and can be used by direct inclusion or through CMake integration. It is intended for benchmarking code that is fast, slow, or unstable, and it produces results as tables, CSV, JSON, HTML box plots, or pyperf output. Use it when you need a repeatable measurement of a small piece of C++ code and want the result in a form you can read, compare, or feed into another tool.

What problem it solves

Measuring small code snippets with a hand-rolled timer is easy to get wrong: the measured time can be dominated by setup, the compiler can optimize the work away, and a single run says little about variation. nanobench is designed around the experiment rather than just the timer. Its documentation covers design goals, how a round is reduced to one number, estimating and reporting, what was rejected, and serial correlation and why warmup is not the answer. In practice, that means you define the benchmark, let the library run it, and then read a result that includes more than one number.

How you use it

A benchmark is built around ankerl::nanobench::Bench. The main entry point is Bench(), and the workflow is:

1. Create a Bench object.
2. Configure it as needed, for example with title(), name(), context(), batch(), unit(), timeUnit(), warmup(), epochs(), minEpochTime(), maxEpochTime(), minEpochIterations(), or epochIterations().
3. Call run() with the code to measure. doNotOptimizeAway() is available to keep the compiler from removing the work.
4. Read the results through results(), or render them with render().

For comparisons, compare() and CompareResult provide size(), operator[](), rounds(), comparisons(), isSignificant(), and fastest(), so you can check whether a difference is significant rather than just looking at two numbers.

Output formats

The library documents several output paths, each suited to a different use:

Columns can be chosen with showColumn(), hideColumn(), and isColumnVisible(), and context can be shown as columns with contextColumn() and clearContextColumns(). Available columns include relative, complexityN, timePerUnit, unitPerSecond, error, instructions, cycles, ipc, branches, branchMisses, and total.

Reading the output

The documentation has a dedicated section on reading the output and on how precise a comparison is. It also covers asymptotic complexity through complexityN(), complexityBigO(), and the complexityBigO() column, which is useful when you want to see how cost grows with input size rather than only how fast one case runs. For unstable code, the histogram and box plot outputs show the spread that a single average would hide.

Practical notes

• NANOBENCH_ENDLESS runs a specific test endlessly.
• NANOBENCH_SUPPRESS_WARNINGS turns off stability warnings.
• NANOBENCH_CONFIG changes settings without recompiling.
• The documentation includes comparisons against Google Benchmark, nonius, Picobench, Catch2, moodycamel::microbench, sltbench, Celero, and folly Benchmark, with source code and results for each, which is useful if you are choosing between libraries.

If you only need a quick timing of one function, a simple timer may be enough. If you need to compare alternatives, check whether a difference is significant, or present results as a table, CSV, JSON, or box plot, nanobench is built for that job.

To start with nanobench, add its single header to your project (direct inclusion or CMake integration), include nanobench.h, create an ankerl::nanobench::Bench, and call run() with a name and a lambda containing the code you want to measure. The default output is a table whose key columns are timePerUnit (time per iteration), unitPerSecond (throughput), error (relative measurement error), and optional hardware counters such as instructions, cycles, ipc, branches, and branchMisses. You can hide, show, or add columns, and attach context values that appear as extra columns.

Installation

nanobench is distributed as a header-only library, so there is no compiled library to link.

Direct inclusion

Copy the nanobench header into your project and include it:

#include "nanobench.h"

CMake integration

If you use CMake, add nanobench as a dependency and link the nanobench target to your benchmark executable. The documentation lists CMake Integration as the supported path; follow the target name it provides rather than inventing include paths.

Basic usage

The main entry point is ankerl::nanobench::Bench. A minimal benchmark looks like this:

#include "nanobench.h"

int main() {
    ankerl::nanobench::Bench b;
    b.title("my benchmarks")
     .unit("op")
     .warmup(100)
     .minEpochIterations(1000);

    b.run("some work", [] {
        // code under test
    });
}

What each part does:

• title() sets the table heading.
• unit() names what one iteration produces (for example op or byte), which affects how unitPerSecond is labeled.
• warmup() runs the code before measuring so caches and branch predictors are primed.
• minEpochIterations() sets a floor on how many iterations each measurement epoch performs.
• run(name, lambda) executes the lambda repeatedly, measures it, and appends a row to the results.

run() can be called multiple times on the same Bench to build a table of several benchmarks.

Reading the output

The default table reports one row per run() call. The columns you will use most:

Two practical points when interpreting these:

• error is a stability signal, not a correctness guarantee. A large error means the measurement varied a lot between epochs; treat the row as unreliable and investigate (see the documentation's sections on unstable benchmarks and serial correlation).
• Hardware counters are optional. instructions, cycles, ipc, branches, and branchMisses depend on performanceCounters() being enabled and on platform support; do not assume they are populated.

Choosing, hiding, and showing columns

Use showColumn() and hideColumn() to control which columns appear, and isColumnVisible() to query the current state. This is useful when you only care about timing, or when you want counters without the default set.

Adding context as columns

context() attaches a named value to the current benchmark, and contextColumn() promotes a context key into its own table column. clearContext() and clearContextColumns() reset them. This lets you compare variants side by side, for example a context key recording which implementation or input size a row used.

Comparing alternatives

To compare implementations rather than just list them, use compare() on the Bench. It returns a CompareResult whose isSignificant() tells you whether the difference between the fastest and the others is statistically meaningful, and whose fastest() identifies the winner. Use this instead of eyeballing timePerUnit when the numbers are close.

Practical tips

• Start with a small minEpochIterations() and let nanobench's adaptive epoch logic settle; the library adjusts epoch length to hit a target measurement interval.
• Keep the lambda free of setup work you do not intend to measure, or move setup outside run().
• If a benchmark looks unstable, check the documentation's guidance on warmup and serial correlation before increasing iteration counts blindly.
• For machine-readable results, nanobench can render CSV, JSON, HTML box plots, and pyperf output through its render() templates, so you can feed results into other tooling instead of parsing the console table.

nanobench is a single-header C++ benchmarking library that integrates by direct inclusion or CMake, runs each benchmark through a warmup and multiple timed epochs, and reports results as a table, CSV, JSON, HTML box plot, or pyperf data. It is a good fit when you want a low-friction, header-only tool with built-in comparison of alternatives and significance testing. Google Benchmark, Catch2, nonius, Picobench, Celero, and others differ mainly in integration cost, output formats, and how they treat measurement noise. The nanobench documentation publishes side-by-side source code and results for these libraries, so the comparison below follows the dimensions that documentation exposes.

Integration footprint

The header-only design is the clearest differentiator: nanobench adds no link step and no external build target. If your constraint is "benchmark this function without restructuring the build," that matters more than any single performance number.

How results are produced and reported

nanobench's model, as documented, is: run a warmup, then run multiple epochs, reduce each round to one number, and report statistics. It warns when results are unstable and lets you suppress those warnings with NANOBENCH_SUPPRESS_WARNINGS. Output is column-based: you choose which columns appear (showColumn, hideColumn, contextColumn), and the same run can be rendered as a table, CSV, JSON, HTML box plot, or pyperf module output via render() and the templates:: helpers (templates::csv, templates::json, templates::htmlBoxplot, templates::pyperf).

The documentation is explicit that warmup is not a fix for serial correlation, and it discusses what was rejected in designing the experiment. That is a useful signal: the library treats measurement methodology as part of its job, not just timing loops.

Comparison and significance

nanobench has a first-class comparison path: Bench::compare() produces a CompareResult with size(), operator[], rounds(), comparisons(), isSignificant(), and fastest(). In other words, "is A faster than B, and is the difference significant?" is answered by the library rather than by eyeballing two tables. The docs also ask "How precise is a comparison?" — a reminder that a comparison result is only as trustworthy as the run's stability. If you mainly need to rank alternatives, this is the feature to evaluate first.

Choosing by measurement need

• Minimal integration, quick comparisons: nanobench, because of header-only inclusion and built-in compare() with significance.
• You already build against Google Benchmark: stay there if its existing harness and reporting fit your workflow; the nanobench docs show its source and results so you can check whether the numbers you care about differ.
• Benchmarks belong with your test suite: Catch2's benchmarking support keeps them in the same framework, at the cost of coupling benchmarks to tests.
• You need a specific output format: check nanobench's templates (CSV, JSON, HTML box plot, pyperf) against what your tooling consumes before switching; format support is often the deciding factor, not raw timing.

No library is universally better. The documentation's own framing — comparing source code and results across implementations — supports choosing on integration cost, comparison semantics, and output format for your specific task, then verifying with a benchmark of your own workload rather than trusting a cross-library ranking.

Practical check before committing

Write one benchmark of a function you actually care about in nanobench (include the header, define a Bench, call run()), render it as CSV and as an HTML box plot, and run a compare() against a known-slower variant. If the significance result and the output format both fit your workflow, the integration cost is already settled. If you need a feature the docs do not list, that absence is the answer — the reference covers the API surface, and unlisted capabilities should not be assumed.