Predicting inlining behavior for nested functor applications

Short answer

There is no documented, stable predicate that tells you "this functor application will be inlined." The OCaml module system is largely static, but native compilation can still represent structures as runtime records and functors as functions. Whether a nested application gets specialized depends on optimizer internals that change between releases, so the only reliable strategy is to make the instantiated path visible to Flambda and verify the result empirically. There is also no attribute that forces a specific functor application to inline — the inlining attributes that exist apply to ordinary functions, not to module-level specialization.

What the overhead actually is

First, a correction to the framing: nested functor overhead is usually not object-style dynamic dispatch. It is indirect calls through module records and closures, plus lost cross-function optimization because the compiler cannot see through the module boundary. First-class modules (packing a module as a value) are the genuinely dynamic case and are much harder to specialize — if your hot path goes through a first-class module, that is the first thing to restructure.

Likely criteria (heuristic, version-sensitive)

Based on how the optimizer is designed, the decision appears to depend on:

• Whether Flambda is enabled — non-Flambda builds expose far less cross-unit inlining information, so cross-compilation-unit specialization is much less likely.
• Visibility: whether the functor body and argument body are in the same compilation unit, or whether cross-unit information (.cmx / Flambda export info) is available at build time.
• Estimated code size versus benefit, recursion in the functor body, and the optimization level (-O2/-O3 with Flambda).
• Whether the instantiated result feeds calls the optimizer considers hot.

The exact weights are internal and differ between OCaml 4.14 and 5.x. Do not assume behavior measured on one switch carries over to another.

What you can actually do

Since you cannot force a functor application directly, the practical levers are:

1. Instantiate close to use. Apply the functor in the same compilation unit as the hot consumer. Same-unit visibility is the strongest signal you can give the optimizer.
2. Keep the hot boundary small and non-recursive. Small leaf functions are far more likely to survive the inliner's cost model.
3. Annotate functions, not functors. Use [@inline] / [@inline always] on the hot function definitions or call sites inside the instantiated module. This biases the function inliner after specialization; it does not trigger specialization itself.
4. Avoid first-class modules on the critical path if specialization matters.
5. Be cautious with global inlining limits. Raising them can help, but in deeply nested chains it often causes code bloat and worse instruction-cache behavior — measure before keeping the change.

How to verify instead of guess

Because the heuristics are opaque, treat verification as part of the workflow:

• Build with your project's real opam switch and compare non-Flambda versus Flambda artifacts.
• Enable Flambda's inlining reports (-inlining-report on supported versions) and check whether the target application's hot functions were specialized.
• Inspect the generated assembly (-S or a disassembler) and look for remaining indirect calls through module records on the critical path.
• Run a focused before/after microbenchmark of the instantiated path with fixed allocation and GC settings, then confirm the win survives in a full-application profile. A single call through a module boundary does not represent hot-loop behavior.

Bottom line

Confirmed: the module system is static in principle, but native code only eliminates functor overhead when the optimizer can see enough to specialize, and Flambda is the main enabler. Likely but not guaranteed: same-unit instantiation plus small annotated hot functions gets you specialization in practice. Not available: a per-application "force inline this functor" switch. If you can share your compiler version and whether the chain crosses compilation units, the recommendation can be narrowed further.