Designing and Operating Gatling’s Distributed Mode: A Practical Architecture Note

Problem & Takeaway

When a single machine can’t generate the required virtual users (VUs) or traffic volume, Gatling’s Distributed Mode lets you spread the load across several nodes. The key challenge is to do this without introducing hidden biases or security holes. The takeaway: a minimal, secure master‑worker topology, clear data trust boundaries, and robust monitoring are essential to keep distributed tests reliable and actionable.

1. Requirements Overview

• All nodes must run the same Gatling version (2026‑10 or later).
• Simulation classes, dependencies, and any custom feeders must be available on every worker’s classpath.
• Master and workers share a common network domain; either a private LAN or a VPN tunnel.
• Optional: TLS encryption on the control channel (default plain TCP on port 9999).
• JVM memory on the master must accommodate the aggregated metrics – typically 1–2 GB per 10,000 VUs.
• Workers should be isolated from other workloads to avoid CPU or network contention.
• Each worker’s JVM should be configured with -Xmx equal to the expected per‑worker VU count times 8 MB (a rough rule of thumb).

2. Minimal Architecture

The simplest distributed test uses a single master and one or more workers. The master coordinates scenario execution and aggregates results after each worker finishes.

# Master (runs the simulation and collects results)
java -jar gatling-charts-highcharts-bundle-3.7.1.jar -m -s com.example.MySimulation

# Worker (listens on TCP 9999 and exposes HTTP /results)
java -jar gatling-charts-highcharts-bundle-3.7.1.jar -w -s com.example.MySimulation

Key points:

• -m starts the master; -w starts a worker.
• Both nodes must point to the same simulation class (com.example.MySimulation).
• The master automatically discovers workers on the same network segment via UDP multicast, or you can explicitly specify a worker host with --worker.
• Workers expose an HTTP endpoint (default http://worker:9999/results) that the master polls after the test completes.

Worker‑to‑Master Control Channel

Gatling uses a lightweight TCP protocol on port 9999 (configurable via --port). The master sends a handshake, the worker acknowledges, and the master streams a list of VU counts to assign. No shared filesystem is required, which keeps the design simple but introduces a single point of failure on the master.

3. Trust & Data Boundaries

Because the master aggregates metrics, it can see all VU traffic data. This means:

• Session data or authentication tokens transmitted by the simulation are exposed to the master. If the simulation contains sensitive information, run the distributed test inside a secure network or enable TLS on the control channel.
• Workers should not log request bodies or headers that contain secrets to disk.
• Use Gatling’s --tls flag to enforce encryption on the control channel:

  java -jar gatling-charts-highcharts-bundle-3.7.1.jar -m --tls --tls-key /path/key.pem --tls-cert /path/cert.pem
  

• Verify the TLS handshake by inspecting the master and worker logs for a “TLS handshake completed” message.

4. Operational Monitoring & Checks

Monitoring in distributed mode is split between the master and each worker. The master’s dashboard shows aggregated metrics; workers expose per‑node metrics via the HTTP endpoint.

• Master JVM memory: jcmd GC.heap_info should show free space > 10 % of -Xmx during the test. If OOM occurs, increase the master’s heap or reduce the number of VUs.
• Worker CPU & network: Use top or htop to ensure each worker is pinned to dedicated cores (e.g., taskset -c 0-3 java ...). Verify network saturation with iftop or nload.
• Latency between master and workers: Run ping -c 5 worker-host before the test. A round‑trip > 5 ms can distort metrics if workers are geographically distant.
• Result file integrity: After the test, the master produces target/gatling/results/xxxx/xxxx.html. Open the HTML and confirm the “Total VUs” equals the sum of workers’ VU counts.
• Worker logs: Search for “Worker finished” and “Master received results” entries to ensure all workers reported back.

5. Failure Modes & Mitigations

• Master crash: All workers stop reporting. Mitigation: run a secondary master with --secondary flag, or implement a simple watchdog script that restarts the master on failure.
• Network partition: A worker loses connectivity to the master. Mitigation: use a VPN with redundancy and monitor tcpdump for packet loss.
• Metric aggregation bottleneck: Master’s JVM runs out of memory when merging large result sets. Mitigation: increase -Xmx or reduce per‑worker VU count.
• Missing worker results: If a worker fails silently, the master will report fewer VUs. Mitigation: enable --verbose logs and set a timeout for result collection.
• Latency spikes not visible in real‑time: Distributed Mode aggregates metrics only after workers finish. Mitigation: run parallel local tests to spot spikes or instrument the simulation with pause checkpoints to isolate problematic sections.

6. When to Reconsider the Design

Use the following guidelines to decide if a different architecture is needed:

• Need real‑time cross‑node correlation: If you must see per‑node latency spikes live, consider using a distributed tracing system (e.g., Zipkin, Jaeger) integrated with Gatling, or run a lightweight proxy that aggregates metrics in real time.
• High security requirements: For tests involving sensitive user data, avoid exposing all traffic to the master. Instead, keep each worker isolated and aggregate results manually via secure logs.
• Very large VU counts (millions): The master’s memory overhead grows linearly with VUs. In this case, partition the test into multiple master‑worker clusters and merge results offline.
• Geographically dispersed load generation: If workers are in different regions, the control channel latency may degrade coordination. Use a multi‑master approach or move the master to a central location with high‑bandwidth links.
• Need per‑worker analytics: If you require separate dashboards per worker, deploy an additional metrics exporter (e.g., Prometheus) on each worker and collect data centrally.

7. Practical Verification Checklist

1. Spin up at least two worker VMs with the same Gatling bundle and simulation class.
2. Run the master with --tls and verify TLS handshake in logs.
3. Execute a short test (e.g., 10 VUs per worker, 5 s duration).
4. After completion, open the master’s results.html and confirm the total VU count equals the sum of workers.
5. Check master JVM memory usage; ensure no OOM errors.
6. Simulate a worker crash mid‑run and observe master’s timeout behavior.

Following this architecture note will give you a robust, secure, and scalable Gatling distributed test environment. Adjust the parameters to fit your specific load and security needs, and always validate the setup with a small pilot before scaling to production workloads.