Azerbaijani LLM training on SageMaker boosts throughput 23%

A six week SageMaker sprint cut GPU memory by 58 percent and set a new bar for language model efficiency in a low resource, morphologically rich language. Azercell Telecom LLC, Azerbaijan’s leading telecom provider, teamed with AWS Generative AI Innovation Center to build an Azerbaijani large language model on Amazon SageMaker AI for telecom use cases and a customer facing chatbot.

The effort produced a production ready framework on SageMaker AI that delivered a 23 percent higher training throughput and 58 percent lower peak GPU memory usage through kernel level optimizations on an ml.p5.48xlarge instance. The team reports that these gains came alongside a practical reshaping of the training pipeline to squeeze more Azerbaijani data into the model’s footprint, a crucial factor for languages with limited training data. Benchmarks indicate that these improvements stem from low level improvements in how the training kernels handle the data and the compute.

Central to the project is a three stage pipeline that feeds into a production ready framework. Stage 1 is tokenizer development, built to create an efficient tokenizer for Azerbaijani. The team evaluated three approaches to tokenization, including a baseline English optimized tokenizer, a vocabulary extension strategy, and a custom monolingual tokenizer. The paper shows that the custom monolingual tokenizer delivered superior encoding efficiency, setting the foundation for better data utilization in the next stages.

The tokenizer work paid off in a big way. The team reports a 2x improvement in tokens per word using the custom tokenizer, effectively doubling the amount of Azerbaijani text that can fit inside the model’s context window. In practical terms this is a big lever for a language with rich morphology and relatively sparse training data, because more content can be considered during each generation step without expanding the model size. The result is not only faster training but potentially stronger in domain alignment for telecom use cases, where the model must understand customer service language, device terminology, and local slang.

From an engineering perspective the project illustrates a clear constraint driven approach. Start with the data bottleneck: the morphology of Azerbaijani means tokenization quality can dominate model performance, so investing in a dedicated monolingual tokenizer yielded outsized returns. The kernel level optimizations on the SageMaker run demonstrated that even with a fairly standard LLM setup, hardware aware tuning can unlock meaningful efficiency. The team notes that the gains were achieved on a production style workflow, which bodes well for teams seeking to ramp up real world deployments quickly.

Looking ahead, experts should watch for how tokenizer quality scales across broader telecom tasks, and how these gains translate to model quality on customer facing tasks such as chatbots and support routing. There is also a case to be made for applying similar low resource language strategies to other morphologically complex languages, though careful attention to data availability and evaluation is required. Finally, balancing cost and performance remains an ongoing engineering tradeoff; the same kernel level wins may not transfer identically to different hardware or cloud configurations without careful adaptation.

The paper shows that a focused, language specific tokenizer combined with hardware aware optimizations can materially improve both throughput and memory footprints in a real world LLM project. Benchmarks indicate that the resulting setup not only trains faster but also allows more language content to be processed within each context window, a practical win when data budgets are tight and latency matters for customer experiences.