Time Limit Exceeded (TLE) error when switching judges for a small coding‑practice app

Direct Answers

1. Verify TLE parity without false positives: Build a benchmark suite of 50–200 historically accepted submissions per problem/language that run at 70–95% of the observed limit. Run them on both judges in shadow mode and compare verdicts, wall-clock time, and CPU time. Any divergence in TLE verdicts signals a threshold mismatch.

2. Monitoring strategy during cutover: Mirror 5–10% of live traffic to the new judge while the old judge remains authoritative. Track TLE rate, verdict distribution, and 95th/99th percentile runtimes per problem/language. Alert on a statistically significant shift (e.g., >2σ change in TLE rate using a two-proportion z-test at α=0.01).

3. Safety margin acceptability and calibration: A 10–20% margin on the configured limit is standard to absorb container startup jitter, CPU scheduling variance, and language-runtime differences. Calibrate by measuring the coefficient of variation (CV) of runtimes for the same submission across 100 repeated runs on the new infrastructure; set margin = 2–3 × CV × mean runtime. Avoid margins >30% to prevent inefficient solutions from passing.

Likely Explanation vs. Confirmed Facts

• Likely: LeetCode’s per-language, per-problem limits are unpublished, empirically derived, and can change without notice. Container overhead (cold starts, cgroup setup) adds 10–50 ms per run. Kernel or CPU architecture differences (AMD vs. Intel) can shift tight-loop performance.
• Confirmed: The only reliable parity check is empirical verdict comparison on a representative corpus. A canary with statistical alerting per problem/language detects shifts that aggregate metrics mask. Safety margins derived from observed runtime CV on the target infrastructure are reproducible and auditable.

Steps for This Migration

1. Assemble benchmark corpus. Collect accepted submissions that historically run near the limit (70–95% of observed max). Include cold/warm runs for each language runtime.
2. Run shadow comparison. Execute the corpus on both judges in parallel for ≥24 hours. Record verdict, wall time, CPU time, exit code. Target >99.5% agreement on non-TLE verdicts and <0.5% false TLEs.
3. Document new judge limits. Record exact configuration (--cpus, --memory, ulimit -t) per language. Verify time accounting (wall vs. CPU) and thread aggregation behavior with a multi-threaded test submission.
4. Calibrate safety margin. Run a fixed submission 100 times on the new judge. Compute CV; if CV <5% for CPU-bound workloads, set margin = 2–3 × CV × mean. Validate that configured limit + margin matches the 99.9th percentile of accepted runtimes.
5. Deploy canary. Route 5% of live traffic to new judge for 48 hours. Compare TLE rates per problem/language against baseline using two-proportion z-test (α=0.01). No significant increase → proceed.
6. Cut over. Gradually increase traffic share while monitoring per-problem/language dashboards. Keep rollback capability until 99.9th percentile runtimes stay within margin for 7 days.

Verification Checklist

Check	Target	Method
Verdict agreement (non-TLE)	>99.5%	Shadow run 24h
False TLE rate	<0.5%	Shadow run 24h
Runtime CV (CPU-bound)	<5%	100 repeated executions
Canary TLE shift	No significant increase (z-test α=0.01)	48h at 5% traffic
Limit + margin vs. P99.9	Margin covers P99.9	Post-canary analysis

One Diagnostic Detail Needed

Does the new judge report wall-clock time or CPU time, and is the accounting method documented per language runtime? This determines whether a conversion factor (e.g., 1 CPU second ≈ 1 wall second on dedicated cores) must be validated before the benchmark suite can be trusted.