Latency spikes during concurrent notebook launches in default server configuration

Direct answer

The diagnostic boundary is the relative ordering of the kernel-spawn timestamp and the HTTP request lifecycle in Tornado. In the default single-worker deployment, if the kernel-spawn initiation is logged after the request has entered the I/O loop but before the response completes, the delay is I/O-loop contention: the request callback is waiting behind other callbacks on the same loop. If the spawn initiation is logged promptly but the gap to kernel-ready is dominated by process creation, that is kernel-launch serialization: the kernel manager is spending time forking and initializing a separate Python process.

What is confirmed versus likely

Confirmed in the default configuration: the server runs a single Tornado worker, each kernel launch spawns a separate Python process, and the core does not parallelize those spawns. Current stable releases do not provide a supported built-in setting for parallel kernel starts without manually adding worker processes or an external multi-process deployment. What is likely under concurrent load is that the single I/O loop serializes HTTP and WebSocket callbacks, while kernel spawns add process-creation latency that is not overlapped by the core server.

Steps to separate the two causes

1. Record three timestamps for each concurrent launch: HTTP request start, kernel-spawn initiation, and kernel-ready.
2. Compare Tornado I/O-loop callback durations in server logs or metrics against kernel-manager process-creation times. Long callback waits before spawn indicate I/O contention; long spawn-to-ready gaps indicate serialization.
3. Repeat the load test with extension-registered WebSocket handlers disabled to isolate their contribution to I/O-loop load.
4. Verify the deployed notebook version and configuration to confirm no manual multi-worker or multi-process settings are active before applying single-worker conclusions.

Factoring in extension WebSocket handlers

Extensions that register WebSocket handlers run on the same Tornado I/O loop as core handlers. They therefore participate in the same contention and can add callback latency that masks or amplifies kernel-spawn delays. Measure baseline latency with extensions disabled, then re-enable them under identical concurrent load to attribute the delta.

One missing detail that changes the recommendation

Are kernel startup directories on local disk or a shared network filesystem? If they are on a network filesystem, kernel-launch serialization is likely I/O-bound and the priority shifts to storage placement or pre-warming; if they are local, CPU-bound process creation or I/O-loop contention is the more probable bottleneck.

Version note: exact kernel-spawn timing and handler registration behavior can differ across notebook versions and host infrastructure. Treat the boundary above as a diagnostic method rather than a fixed constant.