JCR Benchmark

JCR Benchmark

Introduction

One of the first questions before using a new software product or a new version of an existing one is "How fast is it?" and "Does it suit my needs by performance and number of concurrent users?". EXo JCR is trying to answer these questions in regard to Java Content Repository performance and scalability offering JCR benchmarking framework. We choose a Micro-benchmark approach proposed by Japex as a base of our solution. Micro-benchmarks are simple benchmarks that rarely involve complicated deployments and are often used to test specific parts of an application.

This approach allows us to estimate the throughput (or latency or else metrics) of any JCR related use case, from simple JCR API operations like reading Item, adding Node, setting property etc (and even more low level SPI operations if needed) to sophisticated use cases, arisen from JCR based business applications.

Micro-benchmark is similar in spirit to JUnit in that it factors out most of the repetitive programming logic that is necessary to write micro-benchmarks. This logic includes loading and initializing multiple drivers, warming up the VM, forking multiple threads, timing the inner loop, etc.

Micro-benchmark Framework

eXo JCR benchmark Framework is based on Japex tool

Japex is designed to estimate an independent throughput for each of the tests in the suite. Throughput estimation is done based on some input parameters. There are basically two ways to specify that: (i) fix the amount of work and estimate time or (ii) fix the amount of time and estimate work.

Even though Japex will compute a per test and a per driver throughput in transactions per second (TPS)by default, it is possible to customize a test suite in order to calculate throughput using a different unit, such as milliseconds or KBs per second.

Micro-benchmark framework consists of following artifacts:

• TestSuite - a set of tests to run, described in configuration
• Drivers - that encapsulate knowledge about a specific algorithm implementation and extend JapexDriverBase Super interface for all Japex Drivers is defined as follows:

public interface JapexDriver extends Runnable {
      
        public void initializeDriver();
      
        public void prepare(TestCase testCase);    
        public void warmup(TestCase testCase);
        public void run(TestCase testCase);
        public void finish(TestCase testCase);
      
        public void terminateDriver();
    }

• TestCases - define individual operation performance

• Number of threads that will perform the test
• Number of iterations i.e. a number of times each thread performs the test (TestCases)
• Run time i.e. number of time to execute all the tests

• Loading the Drivers configured for this TestSuite. It is possible to define a class path for each Driver individually (using japex.classPath property)
• Instantiating and initializing the Drivers calling Driver.initializeDriver() method. One Driver instance per thread is created and initialized.
• Calling Driver.prepare() and Driver.warmup() methods for preliminary test actions and VM warming up respectively (run time of those opertaions is not taken into account for tests result calculation)
• Performing and timing the inner loop under test. The duration of this phase can be controlled by the parameters japex.runIterations and japex.runTime which refer to the number of iterations or the amount of time per test case respectively
• Calling Driver.finish() where the framework initializes result parameters for further result calculation
• And finally Driver.terminateDriver() is called and benchmarking is finished.

eXo JCR benchmarking

eXo JCR benchmark framework is indended to simplify the creation of tests for Java Content Repository operations.

Configuration

Examples

JCR Standalone

HTTP/WebDAV