Azerbaijani LLM Surges on SageMaker with Custom Tokenizer

Azercell Telecom, Azerbaijan’s leading provider, teamed with the AWS Generative AI Innovation Center to adapt a foundation model to a morphologically rich language with limited training data and no blueprint for efficient LLM training in Azerbaijani. The result is a production ready framework on Amazon SageMaker AI designed for telecom use cases and a customer facing chatbot. Kernel level optimizations run on an ml.p5.48xlarge instance, delivering 23 percent higher training throughput and 58 percent lower peak GPU memory usage. The framework also achieved a 2x improvement in tokens per word through a custom tokenizer, effectively doubling the amount of Azerbaijani text that can fit inside the model’s context window.

The project rests on three sequential stages, each producing artifacts that feed the next. Stage 1 focuses on tokenizer development, building an efficient tokenizer tailored to Azerbaijani. The team evaluated three approaches: baseline English optimized tokenizers, vocabulary extension, and custom monolingual tokenizers, and measured encoding efficiency with standardized metrics. The explicit emphasis on tokenization reflects a practical truth in training models for less-resourced languages: the quality and efficiency of the tokenizer can dominate both throughput and the usable context length for a given compute budget.

Beyond tokenizer work, the collaboration aimed to establish a pipeline that could scale in production. The six week effort yielded a framework that leverages PyTorch, Hugging Face Transformers, and Liger Kernels on SageMaker AI to deliver a production friendly workflow. The improvements were not just theoretical. On a single high end GPU instance class, the team reported meaningful reductions in memory pressure and faster iteration cycles, which are critical for telecom applications that demand responsive, reliable AI assistants.

For practitioners, the Azercell case offers concrete takeaways. First, for morphologically rich or low resource languages, investing in a language specific tokenizer can unlock substantial gains in context efficiency and training throughput. Second, kernel level optimizations on appropriate hardware can yield material memory and speed benefits, enabling larger models or longer context windows without ballooning costs. Third, a staged, artifact driven workflow that feeds subsequent stages with each milestone helps keep a complex project on track and production ready at the six week mark. Finally, when the end use is a customer facing chatbot, a production pipeline that demonstrates tangible gains in speed and memory can directly translate into better user experiences in telecom contexts.

The Azercell milestone demonstrates how a disciplined, engineering first approach can translate research into deployable systems for a language with limited data. By tightening the tokenizer, tuning kernels, and leveraging SageMaker AI, teams can push practical gains that matter in real world customer interactions, not just in published benchmarks.