The Computer Language
25.02 Benchmarks Game

n-body in-process measurements

Let's perform one calculation and record the "1st in-process" elapsed-seconds (exclude startup and shutdown costs).


Let's perform four more warm-up calculations and then perform one final calculation and record the "6th in-process" elapsed seconds (exclude startup and warmup and shutdown costs).


Also show BenchExec elapsed-seconds as "process / 6" (amortize startup and warmup and in-process measurement overhead and shutdown costs).

  source 6th in-process 1st in-process process / 6
  C gcc #8 5.533 5.536 5.531
  C clang #8 5.667 5.674 5.675
  Intel C #8 6.007 6.009 6.006
  Swift #8 6.660 6.695 6.680
  C# naot #8 7.025 7.009 7.008
  C# #8 7.026 7.177 7.057
  Java naot #8 7.179 7.401 7.218
  Java #8 7.228 7.599 7.645
  Dart #8 7.302 7.294 7.384
  Dart aot #8 7.546 7.393 7.416
  Go #8 9.599 9.596 9.600
  Node.js #8 9.609 9.588 9.683
  Java -Xint #8 160.572 160.927 160.772
  PHP #8 242.454 242.477 240.938
  Ruby #8 317.894 318.026 318.167
  Toit #8 502.172 503.125 503.425
  Python 3 #8 515.733 513.793 515.074

The N=50,000,000 workload used for tiny tiny n-body programs takes seconds and tens-of-seconds. These Java in-process measurements become tenths-of-a-second faster.

24 in-process nbody Java #8 elapsed seconds mtime measurements

otoh Measurement in-process changes the programs that will be measured. The placement of those changes has to be correct. Those changes must have the same effect across different programming languages and implementations. Measurements can exclude one-time startup and warm-up and shutdown costs but sometimes fast startup does matter.


otoh Measurement with BenchExec is independent of the programs that will be measured and isolated from other running processes. Measurements will include one-time startup and warm-up and shutdown costs but sometimes startup is fast.