Hyperfine 2.0

[sharkdp]

See also: Hyperfine 2.0 milestone

This is an (unfinished) planning document for a Hyperfine 2.0 release. This release will probably include some breaking changes (where it can't be avoided), so it makes sense to plan ahead. There are three major features that I would like to include:

• Additional performance metrics
• Interleaved benchmark execution
• Saved benchmarks for later comparison

And there are two behavioral changes that might make sense (motivated by choices made by poop)

• Make --shell=none the default
• Use the first command as a --reference by default

These new features and behavior-changes might lead to breaking changes regarding (1) Hyperfine's presentation of results in the terminal (2) the JSON output format and (3) the command line interface. We will discuss these changes on a high level to gauge their overall impact, and I would love to get feedback on this. Detailed discussions for a particular feature should go in the respective tickets though.

Additional performance metrics

The major new feature I am planning to add will be additional performance metrics. We received many feature requests for this over the years:

• Support for benchmarking memory usage, etc. #86
• Advanced metrics beyond wall time (CPU time, instruction count, ...) #736
• Exporting detailed user/system times (in the JSON output)? #660

Recently, poop implemented this in a separate tool, seemingly because Hyperfine lacks this ability (maybe it will inspire the Hyperfine maintainers to add the extra data points). Here, I'm not so much interested in which exact new metrics we support, but rather in the fact that we need some architectural/output-format changes to be able to support multiple metrics. Ideally, wall clock time and user/system time would not be handled any different from more advanced metrics like CPU counters and peak memory usage.

We probably want to invest into an abstraction that can handle these measurements in a uniform way. This would handle things like taking the actual measurement, providing/handling the different units of measurement, I/O, etc.

We should probably also change the JSON output format in a way that all of these measurements are available in the same structure (which is basically what #660) is asking for.

We also need to design a CLI that allows users to include/exclude additional performance metrics. We need to choose a good default. And we probably want a way to select one of these metrics as the target metric for comparisons.

We also need to become more generic w.r.t. wording/naming in lots of places. It's not necessarily the "fastest" command we're looking for, but maybe the most memory-efficient, or the one with the least cache misses.

Interleaved benchmark execution

Ticket: #21

I think this would mainly affect how we display intermediate benchmark results while they are still running. A disadvantage would be that we would not be able to show the finished results for one command as soon as they are done. But an advantage might be that we could show current estimates for all commands, not just a single one.

An important open part is to think about is how this interacts with warmup/setup/prepare/etc.

Saved benchmarks for later comparison

Ticket: #607

This will also be important for the design of the JSON output format. Ideally, we would save baseline results as JSON and then simply load everything we need from the JSON file again. This might require us to store more information than we currently do. It might be useful to use the same format (not as JSON, but as a struct) internally as well to make sure that everything is available.

Make --shell=none the default

Ticket: #787

Use the first command as a --reference

Ticket: —

This might require a redesign of the --reference command.

[sharkdp]

JSON format v2

Things to keep in mind / ideas:

• We could potentially include the full command line, in order to repeat a benchmark in an easy way
• For "matrix" runs, do we keep the flat structure or should we build something that represents the structure more cleanly?

[krk]

Additional performance metrics

How about supporting only two modes to begin with: default and extended modes. Default mode would collect the current duration metrics. Extended mode would include additional metrics from https://github.com/andrewrk/poop to begin with, which is 6 additional metrics.

This would allow supporting multiple metrics with minimal implementation effort, without blocking implementation of the other features, such as JSON support for all metric data, ability to select individual metrics from CLI, changing wording from fastest etc.

For the sole purpose of supporting multiple metrics with default vs. extended flags in the text output, if you agree this is viable, do you think we need #790 as a prerequisite?