As-of 2025, BenchExec

Time-to-Solve

0. The host machine is isolated from networks and unloaded. (Of course, that is not how programs are used!)

   To provide each program with a similiar context, file system caches and swap are cleared before measurements are made for each program. As part of that reset, the first measurement is excluded from the mean and confidence intervals. To ameliorate measurement bias we vary UNIX environment size.

   For a different approach, see Robust benchmarking in noisy environments.

    

1. If the program gives the expected output within an arbitrary cutoff time (usually 5 or 10 minutes elapsed time) the program is measured again (11 more times) with output redirected to /dev/null.

2. If the program doesn't give the expected output within an arbitrary timeout (usually one hour) the program is forced to quit.

3. Most measurements shown on the website are either:

   • within the arbitrary cutoff - the lowest elapsed time from 12 BenchExec measurements

   • within the arbitrary cutoff - the 95% t-score Confidence Interval from 11 CPU time BenchExec measurements; or if the CI was miniscule, the Mean

   • outside the arbitrary cutoff - the elapsed time, cpu time and peak memory from one BenchExec measurement

   (For sure, programs taking 4 and 5 hours were only measured once!)

4. Some few programs were changed to make in-process measurements, within the BenchExec container:

   • Some Java programs were changed to make 24 successive calculations and print System.nanoTime() to System.err before and after each calculation

   • Some n-body programs were changed to make a 1st calculation, 4 successive warmup calculations, and a final 6th calculation. Before and after the 1st and 6th calculations; the programs triggered file changes and thus mtime timestamps and thus millisecond elapsed time measurements