Non-deterministic file(GLOB_RECURSE) ordering undermines build reproducibility - CMake
0 reputation · 06 Oct 2021, 19:32 UTC
0 reputation · 06 Oct 2021, 19:32 UTC
The goal is to make file(GLOB_RECURSE) return source files in the same order on every machine and CMake version so generated build files and test discovery remain reproducible. CMake documentation states that GLOB_RECURSE ordering is unspecified and may change between releases; list(SORT) can impose order but is not applied automatically.
This assumes CMake 3.20 or later. Downstream custom scripts and test discovery can depend on list order, and no built-in cross-toolchain ordering guarantee is documented.
Does CMake offer a built-in option or policy to make file(GLOB_RECURSE) return sorted results by default? If not, is calling list(SORT) immediately after file(GLOB_RECURSE) the supported pattern for stable ordering across file systems? Are there version or platform cases where list(SORT) still cannot produce identical ordering?
35775 reputation · 07 Oct 2021, 07:21 UTC
As of CMake 3.20 and newer, there is no built‑in option, policy, or variable that forces file(GLOB_RECURSE) to return a sorted list. The documentation explicitly states that the order of the returned file list is unspecified and may change between releases. The only supported way to obtain a deterministic order is to sort the list yourself immediately after the GLOB_RECURSE call, using list(SORT).
file(GLOB_RECURSE VAR RECURSE "*.cpp") (or whatever pattern you need).list(SORT VAR).
list(SORT) performs a byte‑wise lexicographic comparison (essentially strcmp), the resulting order is identical on any platform that presents the same byte sequences for each path.add_library, add_executable, or test discovery scripts.file(GLOB) returns paths with backslashes (\), while Unix‑like systems use forward slashes (/). Sorting the raw strings will produce different orders unless you normalise the separator first (e.g., string(REPLACE "\\" "/" …)).list(SORT) compares raw bytes, the relative order of two files that differ only in Unicode normalization can change across systems. Normalising the strings (e.g., to NFC) before sorting is required for reproducibility when such filenames exist.list(SORT) guarantees stable ordering for duplicate strings, but duplicates are identical and do not affect the final set of unique files.list(SORT) still produce different orders?Only in the two scenarios above—different separator styles or differing Unicode representations—will the sorted order diverge. If your source tree contains no such variations, the sorted list will be identical across machines and CMake versions.
If your build system still shows non‑deterministic ordering after applying list(SORT), consider:
Answering these questions will help determine whether additional normalisation steps are necessary.
In short: CMake has no built‑in option to sort GLOB_RECURSE results. The documented, supported pattern is to call list(SORT) immediately after the glob. On typical file systems this yields a stable order, but you must account for path separator and Unicode normalization differences if your source tree contains such variations.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.