NVIDIA’s runnable recipe groups expert calculations, uses 8-bit values, and fuses several operations; its benchmark reports up to 2.21× higher throughput on eight B200 GPUs.

NVIDIA’s BioNeMo tutorial describes a runnable implementation recipe for training biological foundation models built with a mixture-of-experts design. It combines NVIDIA Transformer Engine software with grouped computation, MXFP8 precision and fused operations.

The goal is straightforward: spend less time launching small GPU jobs, store less temporary data, and move fewer values between training steps. The recipe is intended for developers familiar with Python, PyTorch and distributed training.

Why mixture-of-experts models create overhead

A dense transformer sends every token through every part of each layer. A mixture-of-experts model takes a different route: it contains many smaller subnetworks, called experts, but activates only a small subset for each token.

That approach can expand a model’s capacity without running every expert for every token. However, the savings depend on how efficiently the software handles those experts.

A simple implementation may visit the experts one at a time in a Python loop. Each visit can launch a separate GPU operation. The GPU then spends more effort starting and scheduling work instead of performing useful calculations.

NVIDIA’s tutorial describes this as a fragmented expert-computation problem. It compares the approach with a Hugging Face implementation that processes experts separately.

Grouping the expert work

The BioNeMo recipe uses a Transformer Engine operation called GroupedLinear. Instead of launching one separate linear operation for each expert, it gathers the relevant expert weights and input tokens, then submits their transformations together.

The experts still retain separate weight matrices. Grouping does not turn them into one shared expert. It changes how their work reaches the GPU.

Different experts may receive different numbers of tokens, so GroupedLinear accepts a count for each expert. It can then send the local expert calculations through a grouped matrix-multiplication path rather than scheduling every calculation independently.

In practical terms, this reduces the number of separate errands the GPU must organize. That can lower launch and scheduling overhead, especially when many experts each handle relatively small groups of tokens.

How MXFP8 reduces memory use

The recipe’s second technique changes the size of many numbers used during training. BF16 represents each weight and activation with 16 bits. FP8 and MXFP8 use 8 bits for those values instead.

Using fewer bits can reduce memory use for weights and activations. That matters when a model has many parameters or when biological workloads use long sequences, both of which increase pressure on GPU memory.

MXFP8 adds a scaling factor to each block of 32 consecutive values. Scaling helps preserve a useful numerical range even when each value uses fewer bits.

The method does not make every training value 8-bit. NVIDIA’s tutorial says the model keeps master weights in 16-bit form. During low-precision calculations, the system converts weights and activations into MXFP8, then converts results back to a higher-precision format when needed.

Those conversions create their own work. A basic implementation could run quantization and dequantization as separate operations, adding memory movement and framework overhead.

NVIDIA says MXFP8 is hardware-accelerated on its Blackwell GPUs. That hardware support is important: the benefits described here depend partly on using compatible NVIDIA equipment, rather than applying equally to every GPU.

Why fused operations matter

Transformer Engine also combines several connected operations into a fused path. In this context, “fused” means the system handles multiple steps together instead of repeatedly storing and retrieving intermediate results.

The recipe combines two grouped linear operations with ScaledSwiGLU between them. SwiGLU is an activation function used in some transformer feed-forward layers. Here, the operation also combines routing probabilities with the experts’ calculations.

Transformer Engine’s Sequential interface recognizes this pattern. When the sequence matches, it replaces the separate operations with a fused operation for the forward pass and a matching fused operation for the backward pass.

The fused path combines grouped expert computation, MXFP8 handling, SwiGLU work and routing-weight scaling. It also avoids materializing some intermediate results, so the system does not need to create and store every temporary output separately.

That can reduce framework overhead and unnecessary movement of temporary data. The underlying idea is not simply “use smaller numbers.” It is to prevent the efficiency gained from lower precision from being lost during the steps surrounding the main calculation.

What NVIDIA measured

NVIDIA reports that the BioNeMo recipe delivered up to 2.21 times the throughput of a Hugging Face baseline in a training benchmark using eight NVIDIA B200 Tensor Core GPUs.

The tutorial provides an eight-GPU Mixtral-8x7B configuration using expert parallelism and MXFP8 precision. Expert parallelism distributes experts across GPUs so the devices can share the model’s expert workload.

The 2.21× figure is NVIDIA’s reported benchmark result. The available description identifies the GPU count and baseline, but not the complete workload settings, batch size, sequence lengths, power use, cost or statistical variation.

Those details affect whether another training run sees a similar gain. A result on eight B200 GPUs should not be treated as a guaranteed improvement for different biological models, workloads or hardware.

What a team needs to check

The recipe requires at least two GPUs for expert parallelism. NVIDIA says the fused MXFP8 GroupedMLP kernel requires Blackwell GPUs.

For a team with compatible hardware, the practical path is to start with the tutorial’s two-GPU sanity configuration, then compare performance on its own workload using the larger setup. The team also needs to measure whether MXFP8 preserves acceptable model quality and training stability for its biological task; the tutorial’s reported throughput result does not answer those questions.

The broader lesson is that mixture-of-experts efficiency comes from the complete training path. Grouped operations reduce separate expert launches, MXFP8 reduces the size of many values, and fused kernels limit conversion steps and temporary data. The reported speedup is promising for the specified setup, but teams should validate those gains and the model’s results on their own hardware before planning around them.