TextVectorization: adapt() with max_tokens vs. a fixed vocabulary for reproducible token indices
0 reputation · 02 Jan 2022, 00:48 UTC
0 reputation · 02 Jan 2022, 00:48 UTC
I'm designing a text preprocessing pipeline with tf.keras.layers.TextVectorization where token indices must stay identical across training sessions, because downstream checkpoints and a transfer-learning step depend on a stable mapping. Assume a recent TensorFlow 2.x release.
My understanding is that adapt() builds the vocabulary from token frequencies, and when max_tokens truncates the list, ties between equally frequent tokens may be resolved in a way that isn't documented as deterministic—especially on large or distributed datasets. The alternative is to skip adapt() entirely and pass an explicit, sorted vocabulary list at layer construction, which guarantees order but means I maintain the token list myself and lose the convenience of frequency-based pruning.
I've also read that manually sorting the vocabulary after adapt() can break the layer's internal mapping, so that doesn't seem like a safe middle ground.
Given the constraint of bit-identical token indices across runs and machines:
adapt() vocabulary ordering documented behavior, and does tf.config.experimental.enable_op_determinism() affect it?adapt() deterministic?max_tokens interaction) I should account for?A thoughtful contribution can make all the difference. Be the first to share one.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.