Architecting a Secure Remote Code Execution Engine for Algorithmic Judging

The Challenge: Executing Untrusted Code at Scale

The primary technical hurdle in building a platform like LeetCode is the "untrusted code problem." You must allow users to upload and execute arbitrary logic in multiple languages while ensuring they cannot access the underlying host system, steal data from other users, or crash the execution node via resource exhaustion.

The core takeaway is that isolation must be layered. Relying on a single mechanism (like a Docker container) is insufficient because container escapes exist. A robust judge system combines virtualization, syscall filtering, and strict resource quotas to create a multi-layered sandbox.

Minimum Viable Architecture

To minimize the attack surface, the system should be split into three distinct zones: the Web Frontend, the Task Queue, and the Judge Workers.

1. The Asynchronous Boundary

The web server must never execute code directly. Instead, it pushes a submission payload (code, language ID, and problem ID) into a distributed task queue (e.g., RabbitMQ or Redis). This decouples the user-facing API from the high-risk execution environment, preventing a malicious script from locking up the web server's threads.

2. The Execution Worker

The worker node pulls tasks from the queue and initializes a transient sandbox. The smallest suitable design involves:

• Containerization: A lightweight container (e.g., Docker or containerd) providing a clean filesystem and isolated network namespace.
• Runtime Wrapper: A small binary that manages the execution of the user's code, captures stdout/stderr, and monitors the process.
• Resource Constraints: Cgroups (control groups) to enforce hard limits on CPU and RAM.

Trust Boundaries and Security Hardening

The trust boundary exists at the edge of the sandbox. Everything inside the container is considered compromised. To prevent the user from breaking out of the sandbox, the following restrictions are required:

Syscall Filtering

Many language runtimes (like Python or Node.js) have built-in modules that can interact with the OS. To prevent this, use seccomp (Secure Computing Mode) to filter system calls. For example, you should block execve (to prevent spawning shells) and socket (to prevent network egress).

Network Isolation

The sandbox must be launched with network access disabled (e.g., --network none in Docker). This prevents the code from performing SSRF (Server-Side Request Forgery) attacks against your internal infrastructure or using the judge as a botnet node.

Operational Checks and Resource Limits

In a judging system, a crash is not always a system failure; often, it is a valid result of the user's code. The system must distinguish between Infrastructure Failures and Submission Failures.

Outcome	Detection Method	System Action
Time Limit Exceeded (TLE)	Wall-clock timer in the wrapper	SIGKILL the process; return TLE status
Memory Limit Exceeded (MLE)	Cgroup memory limit hit	OOM Killer triggers; return MLE status
Runtime Error (RE)	Non-zero exit code	Capture stderr; return RE status
System Crash	Worker heartbeat timeout	Restart worker; requeue task

Example Configuration: Resource Constraints

When launching the execution container on a Linux worker, the following flags (or equivalent API calls) are necessary to enforce the boundary:

# Example Docker run command for a judge worker
# Run as a non-privileged user, no network, limited memory and CPU
docker run --rm \
  --network none \
  --memory="256m" \
  --cpus="1.0" \
  --pids-limit 64 \
  --user 1000:1000 \
  --security-opt seccomp=/path/to/judge-seccomp.json \
  leetcode-python-runtime:latest python3 -c "$(cat user_code.py)"

Risk Note: Running as root inside the container is a critical vulnerability. If a user finds a kernel exploit, they can gain root access to the host. Always specify a non-privileged --user.

Failure Modes and Design Pivots

As the platform grows, the "one container per submission" model may hit a bottleneck due to the overhead of container startup times (cold starts).

When to Pivot

• Latency Issues: If container initialization takes longer than the actual code execution, pivot to a Warm Pool of pre-started containers that are reset between runs.
• Vertical Scaling Limits: If a single worker node cannot handle the concurrent load, transition to a Kubernetes-based cluster where pods are dynamically scaled based on queue depth.
• Side-Channel Attacks: If high-security isolation is required (to prevent Spectre/Meltdown attacks), move from containers to MicroVMs (e.g., AWS Firecracker), which provide a stronger hardware-virtualized boundary.

Verification and Testing

To verify the sandbox is working as intended, attempt the following "malicious" submissions:

1. Filesystem Probe: Try to read /etc/passwd. The result should be a "Permission Denied" error or a restricted view of the container's filesystem.
2. Network Probe: Try to perform a DNS lookup or HTTP request to an external site. The request should timeout or be blocked immediately.
3. Resource Exhaustion: Submit an infinite loop (e.g., while True: pass) and a memory hog (e.g., [0] * 10**9). Verify that the system returns TLE and MLE respectively without affecting other concurrent submissions.