Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining
arXiv:2610.33016v1 Announce Type: new Abstract: Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it spend 2.6 to 5.8 times as many tokens on Nepali as on the same English text. Merge-based vocabulary extension cannot join the split pieces: for Llama-3 and Qwen3 no number of new merges brings this pr
阅读原始来源
来源记录
研究分析
研究主张
论文标识：2610.33016。 来源描述：arXiv:2610.33016v1 Announce Type: new Abstract: Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it spend 2.6 to 5.8 times as many tokens on Nepali as on the same English text. Merge-based vocabulary extension cannot join the split pieces: for Llama-3 and Qwen3 no number of new merges brings this pr
评测与比较记录
可供比较的近期同类记录：“BlockLock: Reproducible LLM Inference with Fixed Batch Geometry” (arXiv, 2026年11月6日)；“CCDWIKI: Decoupling Case Formulation from Response Generation in Longitudinal CBT Dialogue” (arXiv, 2026年11月6日)。主题相近仅用于导航，未核验其是否属于同一需求或独立证据。
结论适用范围
结论只能在所述模型版本、任务、数据与推理预算内解释。社区票数、下载量或榜单位置不能证明另一工作流中的可靠性。
有效性与复现缺口
需核对样本选择、基线配置、不确定性、数据污染风险、负面结果以及代码或权重可获取性；此处没有本站复现实验记录。
复核路线
阅读论文或模型卡及评测附录，针对明确基线复现一个有限结论，再测试代表性用户任务并记录结论失效的场景。