Wtf kind of benchmark counts the jvm startup time?

How much difference does it make for these tiny programs? otoh For measurements of tenths of a second, tenths of a second is a huge difference. otoh For measurements of seconds and tens-of-seconds, JVM startup, JIT, OSR… are quickly effective.

100 in-process fannkuch-redux Java #8 elapsed secs measurements

100 in-process fannkuch-redux Java #12 elapsed secs measurements

Some experimental studies show 10.9% of process executions don't reach a steady state of peak performance; and 43.5% of process executions were inconsistent; and sometimes they are slower than what came before.

Let's compare the fastest no warmup—include startup-costs—measurements against the fastest JMH after startup & warmup SampleTime p(0.0000) and mean measurements:

java 22 2024-03-19		SampleTime	Mean
i5-3330	No warmup	JMH after startup & warmup
fannkuch-redux #1	10.145	9.815	10.343 N = 35
fannkuch-redux #2	46.275	42.480	43.100 N = 25
fannkuch-redux #3	40.162	39.796	40.378 N = 25
n-body #1	7.892	7.718	7.740 N = 50
n-body #2	7.577	7.332	7.361 N = 50
n-body #3	7.624	7.374	7.390 N = 50
n-body #4	6.905	6.795	6.827 N = 50
n-body #5	6.793	6.677	6.689 N = 50
spectral-norm #1	7.074	6.795	6.800 N = 50
spectral-norm #2	2.375	1.950	2.110 N = 131
spectral-norm #3	1.681	1.569	1.593 N = 175

Let's make an increasing series, each performing one iteration more than the last. Measure elapsed-seconds to include startup-costs and warmup-costs. Average elapsed-seconds per number of in-process iterations to amortize those costs:

fannkuch-redux Java #8 elapsed secs per iteration

nbody Java #8 elapsed secs per iteration

spectral-norm Java #8 elapsed secs per iteration

mandelbrot Java #8 elapsed secs per iteration

fasta Java #8 elapsed secs per iteration