Choosing Between k6 Thresholds and Custom Checks for Load Test Pass/Fail Criteria

Decision: thresholds vs custom checks

When authoring a k6 script you need a way to tell the test whether it passed or failed. k6 offers two complementary mechanisms: thresholds (declarative limits on built‑in or custom metrics) and custom checks (imperative JavaScript assertions that run per request). The choice affects early abort behavior, reporting granularity, and complexity of the pass/fail logic.

Constraints and decision factors

• Do you need the test to stop immediately when a metric exceeds a limit?
• Is the success condition a simple numeric bound or does it require inspecting response payloads?
• How much noise can you tolerate from high‑cardinality custom metrics?
• Do you want per‑item pass/fail details in the test summary?

Comparison of options

Aspect	Thresholds	Custom Checks
Evaluation timing	After each iteration (or per VU) on the aggregated metric value	Per request, immediately after the sample is collected
Early abort	Can abort the whole test when breached (if abortOnFail is true)	Does not abort; only increments pass/fail counters
Supported data	Built‑in metrics (e.g., http_req_duration) and custom metrics via the metrics API	Any JavaScript expression; can access response body, headers, cookies, etc.
Reporting	Single pass/fail line in the end‑of‑test summary	Detailed counts per check name; appears in the “checks” section of the output
Complexity	Simple threshold expression; no scripting needed	Requires writing a check function; can include try/catch to avoid counting exceptions as failures

Trade‑offs

Thresholds are ideal when you have a clear numeric SLA (e.g., 95th percentile response time < 500 ms) and you want the test to stop as soon as the SLA is violated, saving time and resources. Their downside is that they work on aggregated values; if you attach a threshold to a high‑cardinality custom metric (e.g., a per‑endpoint latency histogram) you may see noisy aborts caused by occasional outliers.

Custom checks excel when the pass/fail condition depends on response content (e.g., validating a JSON field, checking for an error code) or when you need per‑request visibility. Because they run per sample, they do not trigger early aborts, but you can drive a threshold from a check‑based metric (e.g., threshold on check_fails) to obtain abort behavior when needed.

Using both together gives the best of both worlds: define checks for detailed validation, then expose a custom metric that counts check failures and apply a threshold on that metric to abort when the failure rate exceeds an acceptable level.

Concrete implementation

The following example demonstrates a typical decision flow:

1. Define a check that ensures the JSON response contains a status field equal to "OK".
2. Expose a custom metric my_check_failures that increments when the check fails.
3. Set a threshold on my_check_failures to abort the test if more than 1 % of iterations fail the check.
4. Add a second threshold on the built‑in metric http_req_duration to enforce a 500 ms max latency.

import { check, sleep } from 'k6';
import http from 'k6/http';
import { Counter } from 'k6/metrics';

// Custom metric that counts check failures
const myCheckFailures = new Counter('my_check_failures');

export const options = {
  thresholds: {
    // Abort if latency exceeds 500 ms for any request
    'http_req_duration': ['p(95)<500'],
    // Abort if more than 1 % of iterations fail the check
    'my_check_failures': ['rate<0.01'],
  },
};

export default function () {
  const res = http.get('https://test-api.k6.io/public/crocodiles/1/');
  
  // Custom check – returns true/false
  const ok = check(res, {
    'status is OK': (r) => r.json('status') === 'OK',
  });
  
  // If the check fails, increment the custom metric
  if (!ok) {
    myCheckFailures.add(1);
  }
  
  sleep(1);
}

Where to run: execute the script on any machine with k6 ≥ 0.45.0 installed (the version that supports the Counter API). No special privileges are needed beyond permission to execute the binary and outbound network access to the target host. Save the script as script.js and run:

k6 run script.js

Expected checks: the terminal output will show a section like:

 checks.........................: 100.00% ✓ 1000 ✗ 0
 my_check_failures..............: 0      ✓ 0   ✗ 0
 http_req_duration..............: avg=120ms min=30ms p(95)=420ms p(99)=480ms

If a request exceeds 500 ms, k6 will print a threshold violation and terminate early, showing something like:

 running (00:00:12), 000/500 VUs, 1200 complete iterations
 threshold http_req_duration on p(95)<500: FAILED, value=560ms
 test aborted due to threshold violation

If the JSON check fails on more than 1 % of iterations, the test will abort after the threshold on my_check_failures is breached.

Limitations and verification

• Thresholds on high‑cardinality custom metrics can cause premature aborts; mitigate by aggregating (e.g., using a Rate or Counter as shown).
• Custom checks that throw JavaScript exceptions are counted as failures unless wrapped in try/catch. Always guard against unexpected parsing errors.
• The early‑abort behavior means that later iterations are not executed, which may hide issues that only appear under sustained load. Run a separate test without abort thresholds to collect full‑duration data if needed.

Practical way to verify the behavior:

1. Run the script against a local mock service that you can control (e.g., using httpbin or a simple Node server).
2. First, ensure all requests succeed and latency stays under 500 ms; the test should complete the full iteration count.
3. Then, inject a delay that pushes one request over 500 ms and confirm the test stops with a threshold violation on http_req_duration.
4. Finally, modify the mock to return an invalid JSON payload for 2 % of requests and verify the abort occurs due to the my_check_failures rate threshold.