Macro System: Unresolved Hygiene Constraint

Question

Macro System: Unresolved Hygiene Constraint
Crystal’s macro system lacks a well‑defined hygiene rule. Macros cannot reliably reference local variables defined outside the macro, leading to compile‑time errors when the same code is built in different contexts. The order of macro expansion is unspecified; different compiler versions may generate code in a different order, producing non‑deterministic output for identical source files. Runtime macro evaluation is not supported, preventing macros from adapting to environment variables or runtime state, which hampers reproducibility in CI/CD workflows. Shards’ optional dependency handling does not enforce compatibility with macro definitions, and this can cause subtle build failures in repeatable environments.
Given these uncertainties, what is the intended hygiene rule for macros in Crystal? Will future compiler releases enforce a deterministic expansion order? How can CI pipelines be made deterministic under the current macro system?

Muntasir · Answer

Answer to the Question
Crystal’s macros are hygienic in the sense that identifiers created inside a macro are scoped to the macro body and cannot clash with outer symbols unless they are explicitly passed in. The language spec does not enforce a deterministic order of macro expansion, so different compiler revisions may expand macros in a different sequence. To make CI pipelines deterministic under the current system, pin the exact Crystal compiler version, avoid relying on implicit environment data inside macros, and pass all required values explicitly to the macro as arguments.
Likely Explanation

Hygiene: A macro can only see symbols that are explicitly supplied to it; local variables from the surrounding code are invisible unless passed as macro arguments.
Determinism: The compiler expands macros during parsing in a left‑to‑right pass, but this order is not formally guaranteed; future releases may change the internal algorithm.
Runtime evaluation: Macros are expanded at compile time; they cannot adapt to environment variables or runtime state, which is why reproducibility in CI can be fragile if such data is used.

Confirmed Facts

Current releases (up to 0.34.x) show stable expansion order for a given source file when the same compiler is used.
Pinning the compiler version and build flags (e.g., --release, --no-debug, --no-color) yields identical binaries across machines.
Macros that read environment variables or perform file I/O during expansion will produce non‑deterministic output even if the compiler is pinned.

Steps to Achieve Deterministic CI Builds

Pin the Compiler
Record the exact output of crystal --version and use that version in your CI definition (e.g., crystal: 0.34.1 in Docker or crystal --version 0.34.1 in a script).
Avoid Implicit Dependencies
Never refer to local variables or functions outside the macro without passing them as macro arguments. If a macro needs a value from the surrounding scope, call it like macro my_macro(val) and invoke it with my_macro(my_var).
Use Explicit Build Flags
Run the compiler with a fixed set of flags: crystal build src/main.cr --release --no-debug --no-color. Store the flag set in a CI environment variable to ensure consistency.
Verify Determinism
After each CI run, compare the produced binary or intermediate files byte‑by‑byte with a cached reference. A simple checksum (e.g., sha256sum) can be used for quick comparison.
Document Macro Contracts
In your codebase, annotate macros with comments that list required arguments and any external data they depend on. This makes it easier for future contributors to maintain determinism.

Question for the Reader
Which Crystal compiler version are you currently using? Knowing the exact version will help confirm whether the deterministic behavior you observe aligns with the compiler’s documented behavior.

Macro System: Unresolved Hygiene Constraint