The Computer Language
22.05 Benchmarks Game

Elapsed seconds + source code size

Always look at the source code.

How many times worse, the fastest benchmark programs for selected programming language implementations score — compared to the fastest written in any of the programming languages — when the source code size is included in the score.

Benchmark program geometric mean score (elapsed seconds and source code size) compared to other programming language implementations.

Programs written for multicore may have more source code. Optimized programs may have more source code. Programs with hand-written vector instructions may have more source code. Simple sequential programs may have less source code. Programs written in languages designed for multicore may have less source code. Programs that use libraries may have less source code.

Benchmark program geometric mean score (elapsed seconds and source code size) compared to other programming language implementations.

CPU seconds, median and percentiles

We don't know how much work was done to optimize each program.

To fix this, regex crate must be updated or replaced. I spent two weekends on this.

Benchmark program CPU seconds compared to the fastest for selected programming language implementations.

Notice which box plot IQRs overlap. Notice which spread across too large a range of values for confidence.

Benchmark program CPU seconds compared to the fastest for more programming language implementations.

Look at the source code and see which of the faster programs use hand written vector instructions. Look and you might see simpler programs written in the same programming language.

Look at the spread from the fastest to the slowest C, C++ and Rust programs.

… a pretty solid study on the boredom of performance-oriented software engineers grouped by programming language.

Histogram of program elapsed time / fastest elapsed time.