arXiv cs.CLPaper
Improving Language Identification for Code-Switched Utterances with Integer Linear Programming
Code-switching breaks LLM tokenizers and training pipelines, and underrepresented in training data means poor performance in multilingual systems. The ILP formulation is cleaner and more interpretable than neural approaches. For anyone building multilingual systems, especially in regions where code-switching is common, this solves a real bottleneck. The code is released for reproducibility.