Limits of custom split functions in tf.keras TextVectorization

Custom split handling

The TextVectorization layer tokenizes input based on the split argument, which may be a string or a callable. When a callable is supplied, the entire input string is first standardize‑ed before the split function is invoked. This ordering can alter token boundaries if the split logic depends on whitespace that has been removed.

Vocabulary build uncertainty

During adapt(), the vocabulary size is influenced by both the split function and max_tokens. Custom splits that produce many short tokens can inflate the vocabulary beyond the intended limit. Moreover, users report that the same split function yields different vocabularies when executed under TensorFlow 2.13 versus 2.14, implying a non‑deterministic ordering or internal change.

Unresolved questions

• Does the order in which tokens are processed by a custom split remain deterministic across TensorFlow releases?
• Is there a stable, documented method to inspect the token‑to‑index mapping after adaptation?
• Can we guarantee a fixed vocabulary size when using a custom split that may generate numerous short tokens?