Microbenchmarks¶

Setup¶

The benchmark tests are located in the src/jmh/java folder. The configuration for the benchmarks is located under src/jmh/ressources. If you want to create more benchmarks locate them here. To start the benchmark just use ./gradlew jmh.

Configuration¶

The configuration of the bechmark is managed through the build.gradle file. There you will find the following segment with a currently minimal set of configured properties.

Example configuration:

jmh {
    include = ['io.ox.benchmark.loadbalancer.test.LBLargeSetupTest']
    //include = ['io.ox.benchmark.loadbalancer.test.LBLargeSetupTest', 'io.ox.benchmark.loadbalancer.test.LBMediumSetupTest', 'io.ox.benchmark.loadbalancer.test.LBSmallSetupTest', 'io.ox.benchmark.proxyrule.test.PRLargeSetupTest', 'io.ox.benchmark.proxyrule.test.PRMediumSetupTest', 'io.ox.benchmark.proxyrule.test.PRSmallSetupTest']
    resultFormat = 'JSON'
    jmhVersion = '1.21'
    jvmArgs = ['-Djmh.separateClasspathJAR=true']
}

Global configuration¶

The properties set in build.gradle will overwrite all local properties which were used in your test class. So be careful if you want to run your local tests with a given configuration. If you want different global properties for your tests just add your tests to the include property. Now only your own written tests will run with that setup.

If you want to explicit exclude tests use the exclude property.

Properties¶

To configure the tests you can use the following properties:

include: include pattern (regular expression) for benchmarks to be executed. Pattern: [‘some regular expression’]
exclude: exclude pattern (regular expression) for benchmarks to be executed. Pattern: [‘some regular expression’]
iterations: Number of measurement iterations to do.
benchmarkMode: Benchmark mode. Available modes are: [Throughput/thrpt, AverageTime/avgt, SampleTime/sample, SingleShotTime/ss, All/all]
batchSize: Batch size: number of benchmark method calls per operation. (some benchmark modes can ignore this setting)
fork: How many times to fork a single benchmark. Use 0 to disable forking altogether
failOnError: Should JMH fail immediately if any benchmark had experienced the unrecoverable error? Value: true/false
forceGC: Should JMH force GC between iterations? Value: true/false
jvm: Custom JVM to use when forking. Pattern: ‘myjvm’
jvmArgs: Custom JVM args to use when forking.
jvmArgsAppend: Custom JVM args to use when forking. Pattern: [‘append these’]
jvmArgsPrepend: Custom JVM args to use when forking. Pattern: [‘prepend these’]
humanOutputFile: human-readable output file. Example: project.file(“${project.buildDir}/reports/jmh/human.txt”)
resultsFile: results file. Example: project.file(“${project.buildDir}/reports/jmh/results.txt”)
operationsPerInvocation: Operations per invocation.
benchmarkParameters: Benchmark parameters. Pattern: [:]
profilers: Use profilers to collect additional data. Supported profilers: [cl, comp, gc, stack, perf, perfnorm, perfasm, xperf, xperfasm, hs_cl, hs_comp, hs_gc, hs_rt, hs_thr]
timeOnIteration: Time to spend at each measurement iteration.
resultFormat: Result format type (one of CSV, JSON, NONE, SCSV, TEXT)
synchronizeIterations: Synchronize iterations? Value: true/false
threads: Number of worker threads to run with.
threadGroups: Override thread group distribution for asymmetric benchmarks.
timeout: Timeout for benchmark iteration.
timeUnit: Output time unit. Available time units are: [m, s, ms, us, ns].
verbosity: Verbosity mode. Available modes are: [SILENT, NORMAL, EXTRA]
warmup: Time to spend at each warmup iteration.
warmupBatchSize: Warmup batch size: number of benchmark method calls per operation.
warmupForks: How many warmup forks to make for a single benchmark. 0 to disable warmup forks.
warmupIterations: Number of warmup iterations to do.
warmupMode: Warmup mode for warming up selected benchmarks. Warmup modes are: [INDI, BULK, BULK_INDI].
warmupBenchmarks: Warmup benchmarks to include in the run in addition to already selected. JMH will not measure these benchmarks, but only use them for the warmup.
zip64: Use ZIP64 format for bigger archives. Value: true/false
jmhVersion: Specifies JMH version. Example: ‘1.21’
includeTests: Allows to include test sources into generate JMH jar, i.e. use it when benchmarks depend on the test classes. Value: true/false
duplicateClassesStrategy: Strategy to apply when encountring duplicate classes during creation of the fat jar (i.e. while executing jmhJar task). Example: ‘fail’

Local configuration¶

It is recommended to configure your tests with local config annotations. Just create an abstract class which holds the annotations and extend it in your test classes, e.g.:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark)
@Fork(value = 1, jvmArgs = {"-Xms2G", "-Xmx2G"})
@Warmup(iterations = 3, time = 500, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 5, time = 500, timeUnit = TimeUnit.MILLISECONDS)
public class AbstractBenchmarkConfig {

}

If you want a specific test in one class to run with individual configurations just annotate them as well:

@Benchmark
@Warmup(iterations = 1, time = 500, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 1, time = 500, timeUnit = TimeUnit.MILLISECONDS)
public void benchmarkStickySessionRouting(StickySessionParams data) {
     manager.chooseServer("APP1");
}

Note

Log output can lead to some higher test outcome than expected. If you want to remove the effect of logging to your microbenchmarks configure a higher log level in the logback.xml.