Distributed Testing
When a single machine can't generate enough load (CPU or memory maxes out before reaching the target thread count), you distribute the load across multiple machines. JMeter has built-in support for this using a controller-worker architecture.
When You Need Distributed Testing
Signs that a single machine is not enough:
-
CPU usage on the load generator is consistently above 80% during the test
-
JMeter runs out of memory (OutOfMemoryError)
-
You can't reach the target thread count without the machine slowing down
-
Response times are inflated because the load generator itself is the bottleneck, not the server
Rule of thumb: A single machine can typically handle 300-1000 threads depending on the script complexity, hardware specs, and whether you're making lightweight API calls or heavy page loads. Monitor your load generator machine during tests.
Architecture
┌──────────────────┐
│ Controller │
│ (your machine) │
│ │
│ Sends .jmx │
│ Collects results│
└──────┬───────────┘
│
┌────────────┼────────────┐
│ │ │
┌──────▼───┐ ┌──────▼───┐ ┌──────▼───┐
│ Worker 1 │ │ Worker 2 │ │ Worker 3 │
│ (remote) │ │ (remote) │ │ (remote) │
│ │ │ │ │ │
│ Runs the │ │ Runs the │ │ Runs the │
│ test │ │ test │ │ test │
└──────────┘ └──────────┘ └──────────┘
-
Controller - the machine that sends the test plan and collects results. Does not generate load itself (by default)
-
Workers - remote machines that actually run the test and generate load
-
Each worker runs the same test plan with the same thread count
-
If you configure 100 threads and have 3 workers, you get 300 total threads
Prerequisites
Distributed testing fails in ways that look like bugs in your script - an empty .jtl, a test that hangs, an RMI error with no obvious cause. Almost all of it traces back to a handful of things being wrong before the test ever runs. Get these right first and most of the mystery disappears.
Matching Versions on Every Machine
| Must match | Why it matters |
|---|---|
| JMeter version | The controller serializes the test plan and the workers deserialize it. A mismatch produces silent serialization failures, not a helpful error message |
| Java major version | Same reason - the objects crossing the wire must agree on both ends |
| Plugins | Any plugin your .jmx depends on must be installed on every worker, or the plan fails to load there while the controller looks fine |
Important: Version mismatch is the most common cause of "it connects, but nothing happens". Check it first, every time.
Ports to Open
JMeter needs RMI in both directions: controller → worker to send the test plan, and worker → controller to send results back. The return path is the one people forget, and it is exactly why a test can start cleanly and still leave you with an empty .jtl.
| Port | Lives on | Used for | Property | Default |
|---|---|---|---|---|
1099 | Worker | Controller connects here first | server_port | 1099 |
50000 (example) | Worker | The worker's RMI engine | server.rmi.localport | 4000 |
50000-50002 (example) | Controller | Workers send results back here | client.rmi.localport | 0 - random |
Two things to notice:
client.rmi.localportdefaults to0, meaning a random high port. You cannot write a firewall rule for a port that changes every run, which is why the return path silently fails. Pin it.- The controller opens up to three consecutive ports starting at
client.rmi.localport, so open a small range rather than a single port.
On each worker:
server_port=1099
server.rmi.localport=50000
On the controller:
client.rmi.localport=50000
Then open them. On a Linux worker using firewalld:
sudo firewall-cmd --permanent --add-port=1099/tcp
sudo firewall-cmd --permanent --add-port=50000-50100/tcp
sudo firewall-cmd --reload
sudo firewall-cmd --list-ports
# Expected: 1099/tcp 50000-50100/tcp
If the workers are cloud VMs, the provider's own firewall (security list, security group, NSG) has to allow the same ports - that is a separate layer from the OS firewall, and both must agree.
RMI SSL
Since JMeter 4.0, RMI uses SSL by default - server.rmi.ssl.disable ships as false - and expects a keystore at bin/rmi_keystore.jks. You have two choices:
Internal test network (simplest): turn it off on both sides.
# jmeter.properties, on the controller AND every worker
server.rmi.ssl.disable=true
Otherwise: generate a keystore with bin/create-rmi-keystore and copy bin/rmi_keystore.jks to every worker and the controller.
Important: Whichever you choose, it must match on both ends. Disabling SSL on only one side produces connection errors that say nothing about SSL.
Network Reachability
Four separate paths have to work, and it is worth confirming each before blaming JMeter:
- Controller → worker on the RMI port
- Worker → controller on the return ports (see above)
- Worker → target application - the workers generate the load, so they need access to the system under test, not just the controller
- No NAT between controller and workers - if the controller sits behind a home or office router, workers have no address to call back to. See NAT breaks distributed testing below
Verify before starting, from the controller:
# Is the worker's RMI port reachable?
nc -zv worker1-ip 1099
Important: If a worker has more than one IP - a cloud VM with a private and a public address, or any multi-NIC host - it announces itself to the controller using whichever one Java picks, and that may not be the one the controller can reach. Pin it explicitly when starting the worker:
jmeter-server -Djava.rmi.server.hostname=<the IP the controller uses>
Pre-Flight Checklist
- Same JMeter version on controller and all workers
- Same Java major version everywhere
- Required plugins installed on every worker
-
server_portandserver.rmi.localportset on workers, ports open in the OS firewall -
client.rmi.localportset on the controller, small port range open - Cloud firewall rules match the OS firewall rules
-
server.rmi.ssl.disable=trueon both sides, or the keystore copied everywhere - Workers can reach the target application
- Controller is not behind NAT relative to the workers
- Test data files present on every worker at the expected path
Tip: Prove the setup with a throwaway plan against a public endpoint before involving your real script - see Use a dummy test plan for setup validation below. It separates "my distributed setup is broken" from "my script is broken", which are very different afternoons.
Setting Up Remote Machines
With the prerequisites satisfied, each worker needs the test plan's data on disk and the JMeter server process running.
Test data files (CSV) must be copied to the same path on every machine - or use the same filename with different content per machine, so each worker drives a distinct slice of the data. See Section 14.
Start the JMeter Server on Each Worker
On each remote machine, run:
jmeter-server
Or on Windows:
jmeter-server.bat
This starts the JMeter server process, listening for connections from the controller. Default RMI port is 1099.
Configuring JMeter for Distributed Mode
On the controller machine, edit jmeter.properties (in the JMeter bin/ folder):
remote_hosts=worker1-ip:1099,worker2-ip:1099,worker3-ip:1099
Replace worker1-ip, worker2-ip, etc. with the actual IP addresses of your worker machines.
RMI Configuration
The RMI ports and SSL settings both machines need are covered in Prerequisites - see Ports to Open and RMI SSL. If you skipped ahead, that is the section to go back to when a worker refuses to connect.
Running Distributed Tests
From CLI (Recommended)
Run on all configured remote hosts:
jmeter -n -t test-plan.jmx -l results.jtl -r
The -r flag tells JMeter to run on all remote hosts listed in remote_hosts.
To run on specific workers only:
jmeter -n -t test-plan.jmx -l results.jtl -R worker1-ip:1099,worker2-ip:1099
What Happens During Execution
- Controller sends the
.jmxfile to all workers - Each worker starts the test with the configured thread count
- Workers send results back to the controller in real-time
- Controller aggregates all results into the single
.jtlfile - When the test ends, the controller generates the report (if
-e -oflags were used)
Important Considerations
Thread Count is Per Worker
If the test plan has 100 threads and you have 3 workers, the total is 300 threads. Adjust your Thread Group accordingly:
| Target Total Users | Workers | Threads Per Worker (in .jmx) |
|---|---|---|
| 300 | 3 | 100 |
| 500 | 5 | 100 |
| 1000 | 4 | 250 |
CSV Data Distribution
If each user needs unique data (e.g., unique login credentials), you need to split the CSV data so each worker gets different rows. Two approaches:
-
Same filename, different content - each worker has a file named
users.csvbut with different rows. This is the cleanest approach (see Section 14) -
Thread Group offset - use the same full CSV on all machines and configure CSV Data Set Config sharing mode
Files Are Not Automatically Distributed
JMeter sends the .jmx file to workers, but not supporting files like:
- CSV data files
- JAR files for plugins
- External scripts
You must copy these to each worker machine manually or via a script (see Section 15).
Timers and Think Time
Timers work the same in distributed mode. Each worker applies think time independently.
Backend Listener in Distributed Mode
If using a Backend Listener for Grafana monitoring, each worker sends data directly to InfluxDB. The results are automatically merged in Grafana since they share the same application and testTitle fields.
Practical Lessons Learned
These lessons came from hands-on distributed testing with Linux VMs and Windows machines. To work through the setup yourself, see Section 13 — Try Distributed Testing on Your Own Machine, which walks the whole thing end to end on free local VMs. For the cloud-specific version, see the archived OCI Linux Slave Setup.
RMI SSL must be disabled on both sides
The configuration is in RMI SSL above; what that section cannot convey is how unhelpful the failure is. JMeter 5.x enables RMI SSL by default and expects rmi_keystore.jks. Since we don't generate certificates for internal testing, disable it on both controller and slave:
# Controller: jmeter.properties (or -Jserver.rmi.ssl.disable=true)
server.rmi.ssl.disable=true
# Slave: in start script
jmeter-server -Dserver.rmi.ssl.disable=true
If you only disable one side, you'll get cryptic connection errors.
NAT breaks distributed testing
JMeter requires bidirectional RMI: controller → slave (send test plan) and slave → controller (send results). If your controller is behind NAT (home router, office firewall), slaves can't send results back.
Symptom: Test starts on slaves, but JTL is empty and console shows summary = 0.
Solutions: Use same-LAN machines, port forwarding, VPN, or run the controller in the cloud alongside the slaves.
Console summary = 0 is often a display quirk
In distributed mode, the console summariser sometimes shows summary = 0 even when results are being collected. Always check the actual JTL file — it usually has the data.
-J vs -G for properties
-
-Jsets properties on the controller only -
-Gsets properties on all slaves (sent via RMI) -
Use
-Gfor things like thread count and duration that slaves need:-Gthreads=10 -Gduration=60
Enterprise antivirus is a silent blocker
Corporate security tools (Symantec, CrowdStrike, etc.) can silently drop JMeter RMI traffic. If distributed testing works on a personal PC but not on a work PC, the antivirus is likely the cause. Ask IT for exceptions on java.exe and JMeter ports.
Use a dummy test plan for setup validation
Before testing with your real application scripts, create a lightweight test plan hitting a public endpoint like httpbin.org. This isolates network/configuration issues from application-specific problems. See test_plan/Dummy-HTTP-Test.jmx in the project.
Tips
-
Test with 1 worker first - validate the distributed setup works before adding more workers
-
Monitor worker machines - check CPU and memory on workers during the test to ensure they're not overloaded
-
Keep JMeter versions in sync - mismatched versions between controller and workers cause silent failures
-
Use the same network - controller and workers should ideally be on the same network or low-latency connection. High latency between controller and workers can affect result collection
-
Automate worker setup - when you have many workers, manually copying files and starting jmeter-server is tedious. Use batch scripts (see Section 15)
-
Check firewall rules - the most common distributed testing issue is connectivity, in both directions. Work through the Pre-Flight Checklist before blaming the script
-
RMI hostname matters — on cloud VMs,
hostname -Ireturns the private IP but external controllers connect via the public IP. Use the public IP for-Djava.rmi.server.hostnamewhen controller is external, private IP when in the same VCN