TeX macro expansion engine and pdfTeX driver: \write18 shell-escape blocking under restricted mode
0 reputation · 29 Sept 2024, 19:50 UTC
0 reputation · 29 Sept 2024, 19:50 UTC
The goal is to clarify how TeX’s macro expansion engine coordinates with the pdfTeX output driver when a \write18 shell-escape command spawns a long-running external process while the engine operates in the restricted shell-escape mode that became the default in 2020.
Macro expansion proceeds independently of the backend; \write18 is expanded during token processing and triggers a shell call before any page is shipped out. pdfTeX buffers output until a page completes, so a blocking external command stalls the main TeX process, mimicking resource-exhaustion symptoms. Restricted mode causes \write18 to expand to empty text without error, silently dropping expected side-effects. Behavior differs across pdfTeX, XeTeX, and LuaTeX, and security policies can override the shell-escape setting at runtime.
31275 reputation · 30 Sept 2024, 02:26 UTC
The short answer: in restricted shell-escape mode, pdfTeX does not block on your external command at all — it never runs it. The \write18 token is expanded to empty text during token processing, the shell call is silently dropped, and compilation proceeds. If you are seeing a stall that looks like resource exhaustion, that stall is almost certainly not caused by restricted mode; it is either an unrestricted-mode \write18 whose child process is genuinely long-running, or an unrelated I/O or memory problem. The two failure modes are distinguishable, and the rest of this answer explains how.
TeX's macro expansion engine and the pdfTeX output driver are only loosely coupled: \write18 is expanded during token processing, before any page is shipped out, and pdfTeX buffers page content until a page completes. Two consequences follow:
\write18 expands to nothing. No subprocess is spawned, nothing appears in ps or system monitors, and the log contains no shell-invocation line. Expected side effects (generated files, fetched data) simply never happen.The reliable discriminator is the absence of evidence, checked deliberately:
runsystem(...), "written to shell", or engine-specific variants). No such line plus missing side effects means restricted mode dropped the call — nothing was ever blocked.ps aux | grep pdftex and look for the child process in the same process tree.! TeX capacity exceeded or the OS OOM-killer logs a kill. Those are errors, not hangs.Do not trust the "Shell escape enabled" banner alone — wrapper scripts and redirected output can suppress it, and site security policies can override the setting at runtime. Verify empirically with a probe document:
\immediate\write18{echo ok > /tmp/tex_escape_probe}
\byeCompile it, then check whether /tmp/tex_escape_probe exists. Presence proves unrestricted execution; absence proves the call was dropped. For a build system, run this probe once in a setup step and gate the real build on the result, rather than parsing logs. You can also force intent explicitly with -shell-escape or -no-shell-escape so the mode is a decision, not an environment accident.
There is no documented pdfTeX-level timeout for \write18: the engine waits until the child exits, and process-group behavior is inherited from the invoking shell. If you need a bound, impose it yourself — wrap the command in timeout 60s ... (GNU coreutils) or have your build system kill the whole process group. Note also that XeTeX and LuaTeX route shell escape through different handlers, so the restricted-mode silencing described here is verified for pdfTeX and should be re-probed per engine rather than assumed identical.
-shell-escape on a trusted build host only.timeout and log its start/end times so a future stall is attributable to the child, not the engine.One caveat for review: exact log wording (runsystem vs. other formats) varies by TeX Live release, so confirm the banner strings against your installed version before hard-coding log parsing.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.