TextVectorization: adapt() with max_tokens vs. a fixed vocabulary for reproducible token indices

Question

I'm designing a text preprocessing pipeline with tf.keras.layers.TextVectorization where token indices must stay identical across training sessions, because downstream checkpoints and a transfer-learning step depend on a stable mapping. Assume a recent TensorFlow 2.x release.

My understanding is that adapt() builds the vocabulary from token frequencies, and when max_tokens truncates the list, ties between equally frequent tokens may be resolved in a way that isn't documented as deterministic—especially on large or distributed datasets. The alternative is to skip adapt() entirely and pass an explicit, sorted vocabulary list at layer construction, which guarantees order but means I maintain the token list myself and lose the convenience of frequency-based pruning.

I've also read that manually sorting the vocabulary after adapt() can break the layer's internal mapping, so that doesn't seem like a safe middle ground.

Given the constraint of bit-identical token indices across runs and machines:

Is the non-determinism in adapt() vocabulary ordering documented behavior, and does tf.config.experimental.enable_op_determinism() affect it?
Is supplying a precomputed fixed vocabulary the intended approach for reproducible pipelines, or is there a supported way to make adapt() deterministic?
Are there trade-offs of the fixed-vocabulary route (e.g., OOV handling, max_tokens interaction) I should account for?

Tasadduq Burney · Answer

AnswerWhen token indices must be identical across runs, the ordering produced by adapt() is not guaranteed to be deterministic. The layer sorts tokens by descending frequency, but ties are broken by the order in which the dataset iterator yields elements. That order can change with parallel loading, shuffling, or different hardware, so adapt() alone does not give bit‑identical vocabularies.tf.config.experimental.enable_op_determinism() only affects floating‑point operation reproducibility; it does not influence the vocabulary construction or tie‑breaking in adapt().The supported way to obtain a stable mapping is to pass an explicit vocabulary list when constructing the layer. This bypasses adapt(), guarantees the same token‑to‑index mapping on every run and machine, and ignores max_tokens (the full list is used). Out‑of‑vocabulary tokens are sent to the OOV index you configure.Trade‑offs of the fixed‑vocabulary approach: you lose automatic frequency‑based pruning, must maintain the list yourself, and max_tokens has no effect. Ensure the list is a plain Python list of strings; mismatched OOV settings can cause unexpected handling.To decide whether deterministic adapt() is sufficient for your pipeline, run adapt() on a static, non‑shuffled tf.data.Dataset and capture the vocabulary. Repeat the process in separate invocations. If the ordering varies, use a fixed vocabulary; if it stays constant under your exact seeding and loading configuration, deterministic adapt() may be adequate.

TextVectorization: adapt() with max_tokens vs. a fixed vocabulary for reproducible token indices

1 answer

Answer

Your perspective belongs here.

0 question comments