Keras

Limits of custom split functions in tf.keras TextVectorization

Custom split handling The TextVectorization layer tokenizes input based on the split argument, which may be a string or a callable. When a callable is supplied, the entire input string is first standardize ‑ed before the split function is invoked. This ordering can alter token boundaries if the split logic depends on whitespace that has been removed. Vocabul

Community & Careers · Editorial question

tokenizationkerastensorflow