LuaTeX static font-metric arrays vs. Lua GC heap when loading many OpenType fonts

Short answer

Yes, a document loading 50+ large OpenType fonts can exhaust memory before lua_mem_limit is reached, but in practice the failure you hit first is usually the Lua heap cap, not a static C‑side limit, because LuaTeX's font structures are dynamically reallocated rather than fixed‑size pools. There is no documented texmf.cnf parameter that raises a “static ceiling”; for font data the ceiling is the OS process limit.

What actually allocates where

• C‑side structures — font directory entries and per‑font character/glyph info tables — are allocated by the TeX‑side allocator with malloc/realloc. They grow dynamically, are not garbage‑collected, and are not counted against lua_mem_limit. Once a font is loaded, this is the retained cost.
• Lua heap — during loading, luaotfload parses each font into large Lua tables (glyphs, features, GSUB/GPOS data) before handing a compacted form to the engine. These tables are counted against lua_mem_limit and are the transient spike.

Hyphenation patterns and the format file add further C‑side overhead, but they are per‑language, not per‑font, so they do not scale with your 50 fonts.

Which limit fails first

Because there is no small hard‑coded MAX_FONTS/MAX_GLYPHS ceiling, C‑side exhaustion appears as ordinary process memory pressure or a failed realloc, not a tidy “font memory exceeded” error. Whether you reach it before the Lua cap depends on:

• Glyph count and OpenType table complexity per font (CJK fonts with tens of thousands of glyphs dominate; a 50‑font Latin family may be comfortable).
• Cache state. A cold luaotfload cache allocates heavily in Lua during loading; a warm cache shifts cost toward retained C‑side structures. First and subsequent runs can fail differently.
• The configured lua_mem_limit relative to available RAM.

With default settings and typical CJK‑heavy fonts, the Lua heap cap is the more likely first failure during the loading phase; retained C‑side growth becomes the constraint only with a warm cache and a generous Lua limit.

The texmf.cnf question

No. texmf.cnf memory keys (main_memory, font_mem_size, etc.) govern classic TeX arrays; LuaTeX's font structures grow dynamically within the OS process limit, and lua_mem_limit caps only the Lua allocator. Raising the C‑side headroom means raising OS limits (ulimit, cgroup), not a configuration key — which matches your constraint of staying within the process limit.

Stepwise RSS and GC timing

Yes, this is observable and expected. luaotfload builds a large temporary table per font; RSS climbs in plateaus as fonts load, then drops (or stops growing) when a GC cycle reclaims the temporaries. The retained step that survives GC is the C‑side font data. So a ps trace showing saw‑tooth growth with a rising floor is the signature of Lua temporaries plus C‑side retention — not a leak.

How to verify on your system

1. Set a low lua_mem_limit in texmf.cnf (e.g., 50000) and compile a minimal document that loads the 50 fonts via fontspec. If it aborts with a Lua memory error, the heap is your binding constraint; if it survives, raise the limit and watch for process‑level exhaustion.
2. Inside the document, print collectgarbage('count') (Lua heap, KB) after each font load and compare against ps -o rss= -p <pid> sampled externally. The gap between RSS and Lua heap approximates C‑side retention.
3. Run once cold (luaotfload-tool --cache=erase) and once warm to separate loading‑time Lua allocation from persistent font metrics.

Version predictability, 1.12–1.17

The architecture (dynamic C font tables plus a capped Lua heap) is stable across this range, so the kind of failure is predictable. A single numeric exhaustion threshold is not: per‑glyph costs and luaotfload table sizes vary by version, font, and distribution patches. Treat any fixed “N fonts is safe” figure as unverified until measured on your exact fonts and build.

Missing diagnostic detail

To give a definitive recommendation, please share the approximate total glyph count or average OpenType table size of the 50 fonts you intend to load. This determines whether C‑side retention is likely to be the bottleneck.