Using Gatling Assertions to Enforce Performance SLAs in CI Pipelines

Problem & Takeaway

When you run load tests in a CI pipeline, you want the pipeline to fail automatically if the application no longer meets its SLA. Gatling’s assertion DSL lets you declare global or per‑scenario thresholds that abort the simulation on breach, giving instant feedback. The key is to design the test so the assertions are realistic, the data boundaries are clear, and operational checks confirm the thresholds are being evaluated correctly.

Requirements

• Gatling 3.x (the assertion API is stable from 3.0 onward).
• Java 17+ or Scala 2.13 (Gatling is distributed as a Maven/Gradle dependency).
• Basic knowledge of HTTP DSL and CSV feeders.
• CI environment (Jenkins, GitHub Actions, GitLab CI, etc.) that can run mvn gatling:test or ./gradlew gatlingRun.
• An SLA definition: e.g., max response time < 200 ms, p95 < 300 ms, error rate < 0.5%.

Minimal Architecture

The smallest viable design is a single scenario that reads data from a CSV feeder, hits an HTTP endpoint, and applies two assertions:

1. Global max response time.
2. Global error rate.

Example folder layout:

src/test/scala
├─ GatlingAssertionsTest.scala
└─ data
   └─ users.csv

Sample Test

Run the following code in GatlingAssertionsTest.scala. Replace {BASE_URL} and other placeholders with your actual values.

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class GatlingAssertionsTest extends Simulation {
  val httpProtocol = http
    .baseUrl("{BASE_URL}")
    .acceptHeader("application/json")

  val feeder = csv("users.csv").circular

  val scn = scenario("API Load")
    .feed(feeder)
    .exec(
      http("Get User")
        .get("/api/users/${id}")
        .check(status.is(200))
    )

  setUp(
    scn.inject(atOnceUsers(10)) // minimal load for illustration
  ).protocols(httpProtocol)
    .assertions(
      global.responseTime.max.lt(200),
      global.failedRequests.percent.lt(0.5)
    )
}

Explanation of key parts:

• global.responseTime.max.lt(200) – abort if any response exceeds 200 ms.
• global.failedRequests.percent.lt(0.5) – abort if error rate > 0.5 %.
• Both assertions run after the simulation completes; if either fails, Gatling exits with a non‑zero status, causing the CI job to fail.

Trust & Data Boundaries

Assertions rely on:

• The feeder delivering realistic traffic patterns. If the CSV contains only a handful of IDs, the test might not stress the system enough.
• The HTTP protocol configuration (timeouts, retries). A misconfigured timeout can inflate response times and trigger false positives.
• The environment where the test runs. A bottleneck on the load generator (CPU, I/O) can cause slow responses that are unrelated to the application.

Mitigation:

• Keep the feeder size proportional to the target load.
• Use realistic timeout values (e.g., httpProtocol.connectTimeout(5.seconds)).
• Run the test on a dedicated machine or CI agent with sufficient resources.

Operational Checks

1. Validate DSL syntax. Run ./gradlew gatlingRun locally to ensure no compile errors.
2. Confirm threshold values. Check that the numbers match your SLA. A common mistake is to use milliseconds instead of seconds.
3. Parse the report. Gatling generates target/gatling-/simulation-/report.json. Verify that the assertions section contains the expected entries and that failures are flagged.
4. In CI, add a post‑step that greps the report for \"failedAssertions\" and fails the job if any are present.

Failure Modes & Investigation Flow

Mode	Possible Cause	Immediate Action
Assertion aborts on max response time	Application latency spike	Check application logs; verify load generator health.
Assertion aborts on error rate	Transient network glitch	Run the test again; if repeatable, investigate API error handling.
False positive due to load generator bottleneck	CPU saturation on CI agent	Scale the agent or reduce user count.
Assertion never triggers despite SLA breach	Threshold too low/too high	Re‑evaluate SLA numbers; adjust with tolerance margin.

When to Redesign

• When the application scales to hundreds of concurrent users, the single‑scenario design may no longer reflect real traffic patterns. Add multiple scenarios with different user mixes.
• When you need to test API composition (multiple endpoints per request). Use chained requests or exec(session => ...) to model the flow.
• When you want early failure detection (abort on first breach). Gatling’s default behavior is to run the entire simulation before evaluating assertions. To abort immediately, use abortOnFail in the setUp configuration.
• When external monitoring is required. Combine Gatling assertions with Prometheus alerts for a holistic view.

Practical Check – Verify Assertion Logic

To confirm that assertions work, purposely slow the endpoint (e.g., add Thread.sleep(250) on the server). Run the test locally:

./gradlew gatlingRun

Expected outcome:

• The console shows Simulation aborted: Assertion failed: global.responseTime.max.
• The report JSON contains an assertions array with an entry for max response time marked as failed.

If the simulation completes without aborting, double‑check that the assertion threshold is indeed .lt(200) and that the test is using the correct scenario.

Limitations & Caveats

• Assertions evaluate only after the simulation ends unless abortOnFail is used.
• They rely on the accuracy of the simulated traffic; they cannot detect issues that only appear under production load patterns.
• Too tight thresholds may cause flaky CI failures due to transient network jitter.
• Gatling’s reporting is static; for real‑time monitoring, integrate with external dashboards.

Conclusion

By tying performance thresholds to Gatling assertions, you embed SLA enforcement directly into your test code. The minimal design—single scenario, CSV feeder, two assertions—provides a clear, maintainable foundation that can grow as your system evolves. Always validate thresholds against real SLAs, monitor the test environment, and be ready to adjust your design when scaling or when new failure modes emerge.