Beanshell vs JSR223 vs Java JMeter Scripting: The Performance-Off You've Been Waiting For!

from https://www.blazemeter.com/blog/beanshell-vs-jsr223-vs-java-jmeter-scripting-its-performance

There are several options to perform custom JMeter scripting and extend baseline JMeter functionality. Check out the most popular extension mechanisms, compare performance and analyze which one is the best. 

 

It's Battle of the Bands, JMeter style.

 

Beanshell V. JSR223 V. Java Request Sampler

 

In our previous post,  JMeter Performance and Tuning Tips (written by the fantastik Philipe Mouwad) we recommended using  JSR 223 + Groovy for Scripting. It’s the best option as Groovy scripts can be compiled into native Java code (assuming some requirements are met) so Groovy script execution performance can be almost as fast as Java code. 
 

So if you’re about to use scripts for something once, quick 'n dirty (e.g. reading configuration files at the beginning of the test) you’re welcome to use Beanshell/Javascript/whatever you’re comfortable with.

 

But, if you’re about to do some extensive load testing through scripting (i.e. constructing massive HTTP requests from calculated data) you need to consider either Groovy or custom Java requests or a JMeter Sampler.

 

For comparison purposes, we’re going to use the same simple code which generates a 1Mb string of random alphanumeric characters. 

 

 

Comparison metrics will be collected using  10 users x 100 iterations with a  session duration of 1 hour (standard BlazeMeter session length), load generation will be done from single JMeter Console by execution of the above code and demonstrating key performance indicators of following engines:

 

• Beanshell (as is)

• JSR223 (Groovy as language, compilation caching enabled)

• Java (as JMeter Java Request Sampler)

 

Including associated CPU/RAM cost on load generator side (BlazeMeter Console)

 

Tests will be using following environment:

 

• Test Type - JMeter Test (Sandbox)

• Threads - 10 threads per Thread Group

• Server Type - Large

• Session Time - 1 hour

• Engine - Console Only

• Server - 1

• Threads per engine - 10

• Rampup - no ramp-up

• Iterations - 100

• Server OS - Linux

• Server CPU - 2x

• Java 7 x64 1.7.0_03

• JVM arguments - -server  -Xms3072m -Xmx6144m -XX:NewSize=64m -XX:MaxNewSize=128m  -XX:MaxTenuringThreshold=2  -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -XX:PermSize=64m -XX:MaxPermSize=64m 

 

Beanshell Sampler

 

Configuration

 

For Beanshell, no pre-requisites are required, everything works out-of-the-box. All we need is to do is add a Beanshell Sampler and paste 1Mb random string generation code. After uploading test scripts to BlazeMeter's testing dashboard and setting appropriate threads, iterations and duration we got the following results:

 

Load Test Results

 

 

JSR233 Sampler

 

Configuration

 

As mentioned, for scripting it’s recommended to use JSR233 + Groovy. Groovy isn’t shipped with JMeter, it needs to be downloaded separately.  To get started:

 

1. Download latest groovy binary bundle from Groovy website download area

2. Locate groovy-all-${VERSION}.jar under “embeddable” folder of distribution and drop it to JMeter/lib folder. Or upload it to BlazeMeter in the “Files” area, BlazeMeter will place it into the corresponding location

3. Add JSR233 Sampler to Thread Group specifying “groovy” as Language

4. Set Compilation cache key to something unique 

5. Paste 1Mb random string generation code to Script area

 

Important to note: 

 

• Use .groovy files instead of keeping the Groovy code inside the sampler. However if you need to have code directly in sampler, make sure that you have Compilation Cache Key set. If you have > 1 JSR233 Sampler – make sure that they use different keys

• Don’t refer any variables as ${VAR} inside the Groovy script. Use either vars.get(“VAR”) or the Parameters stanza of JSR233 Sampler

 

Load Test Results

 

 

Java Request

 

Configuration

 

Java Request is your own implementation of JavaSamplerClient, all described methods should have appropriate code. The absolute minimum is override of runTest() method but if you intent to parameterize your Java Request you need to provide appropriate logic  to read inputs and conditional interpretation of requests flow to determine whether Sampler passed or not. 

 

The example code for generation of 1Mb random string via Java Request will look as follows:

 

 

Compiled class needs to be placed in /lib/ext folder of your JMeter installation preferably in .jar form so JMeter could automatically pick it up, elsewise you’ll need to amend JMeter classpath. 

 

When using BlazeMeter just upload the .jar file altogether with your script and other extensions (if any) and the BlazeMeter engine will pick it up. 

 

 

2013/12/05 11:09:38 INFO - BlazeMeter: BlazeMeter startup script completed

2013/12/05 11:09:39 INFO - jmeter.protocol.java.sampler.JavaSampler: Created class: com.blazemeter.JavaRequest

 

 

Load Test Results

 

 

Conclusion

 

 

Beanshell is recommended to be used for once-only activities like reading configuration file somewhere in single-threaded setUp Thread Group or in situations, where no possibility to use alternatives exists. 

 

JSR233/Groovy is quite reasonable option for scripting but only with “compilation” feature.

 

And the winner is......... Java Request which provides blazing performance and cutting-edge productivity!

 

Learn More

 

Check out BlazeMeter's Knowledge Base for "loads" of  JMeter, BlazeMeter and load testing tutorials!