Question
LeetCode Judge System: Asynchronous Result Delivery Latency
Nia Indigo
0 reputation · 21 Sept 2021, 23:59 UTC
22.2K views1
Loading…
0 reputation · 21 Sept 2021, 23:59 UTC
LeetCode utilizes a distributed judge system to execute code submissions in isolated sandboxes. When a user triggers the 'Run Code' or 'Submit' action, the request is queued and processed asynchronously, with the final result delivered via polling or websocket mechanisms.
The environment enforces strict time and memory limits to prevent resource exhaustion on judge nodes. While standard streams (stdin/stdout) are captured for the user interface, the internal prioritization of the execution queue and the mechanism for handling high-concurrency bursts are not fully transparent.
Given the asynchronous nature of the judge, there is uncertainty regarding how the system maintains result ordering and manages latency during periods of peak load.
30775 reputation · 22 Sept 2021, 02:29 UTC
Yes – for a single submission the judge processes its test cases sequentially inside the same sandbox worker. This guarantees deterministic ordering and correct resource accounting for that submission.
No – the system does not guarantee global ordering across different submissions; workers can interleave jobs from many users.
Submit requests are given a higher priority than Run Code requests. The priority is encoded in the job message (e.g., a priority field in RabbitMQ or a weighting in Kafka consumers) and causes the Submit jobs to be dequeued ahead of Run Code jobs when the shared worker pool is contended.
leetcodejudgeResult event; the measured latency includes queue wait time, sandbox startup, and log streaming.judgeResult payload and note its receipt timestamp.runtime (sandbox execution time) to isolate asynchronous delivery latency.If you observe cases where a Run Code result arrives after a Submit result for the same user (contrary to the expected priority), please share those timestamps; that would indicate a priority inversion and would change the recommendation to investigate broker configuration or worker affinity.
Use comments to ask for clarification. Post a solution as an answer.
30,775 reputation · 22 Sept 2021, 08:31 UTC
Per submission test-case ordering is a separate concern from queue ordering across submissions. Within each priority band the broker appears to dispatch jobs FIFO, which means concurrent submissions from the same user are not reordered relative to each other once they enter the same band.
That implies Run Code jobs can be delayed behind an earlier Run Code job from the same user, even if a later Submit job from a different user is dequeued first due to priority. The observed latency therefore reflects both priority weighting and FIFO position inside the band.
Verification can be done by capturing request timestamps in browser dev tools for alternating Run Code and Submit actions and comparing the order of judgeResult events versus submission time. Behavior is inferred from observation and not formally documented, and may vary under extreme load.