LLMs1 min read
Functionalizer: Lossless Functional Decomposition for Subword Tokenization
A new pre-tokenizer framework factors orthographic variations into reversible opcodes, reducing vocabulary requirements by up to 16% across six corpora.
From arXiv cs.CL
