A semantic tokenization scheme where token geometry reflects semantic relationships [R]

I have been thinking about an alternative tokenization and representation scheme for language models and would be interested in hearing whether similar ideas have been explored before, as well as potential advantages or flaws. The core observation is that modern tokenizers (BPE, SentencePiece, etc.) primarily capture statistical structure in text. While this is highly effective, the resulting token assignments are not explicitly organized according to semantic relationships. Concepts that are semantically related may end up with completely unrelated token identifiers, and semantic structure is learned later through embeddings and training. The idea is to construct a tokenization scheme in which the symbolic representation itself carries semantic information. For example, instead of assigning arbitrary identifiers to concepts, we could learn a mapping from concepts to short character strings such that semantically similar concepts receive similar codes. A concept like “dog” might receive a code close to those assigned to “wolf” and “fox”, while more distant concepts such as “car” would receive codes that are farther away in the code space. One possible implementation would be: 1) Build a semantic graph using resources such as WordNet, embedding similarity, or a combination of both. 2) Learn a compact symbolic encoding for concepts. 3) Optimize the encoding so that distances between codes correlate with semantic distances in the graph. 4) Train language models directly on these codes. An extension of the idea is to treat a standard keyboard layout as a fixed geometric space. The keyboard itself is not semantically meaningful, but it provides a globally agreed-upon metric structure. The learned encoding could exploit distances between characters and positions when constructing semantic codes. For example, if two concepts are semantically close, their symbolic representations would differ only slightly. Ambiguous concepts could potentially occupy positions that reflect