NVIDIA’s open-source Kubernetes tool runs distributed tests, then narrows a failed cluster test to likely nodes.

A GPU cluster can look healthy and still struggle with a large AI training job. NVIDIA says a 512-GPU job may run slowly or fail because of one weak GPU, a network link that degrades under load, or traffic taking a slower path.

NVIDIA Cluster Readiness Engine, or NVCRE, is an open-source Kubernetes controller announced in NVIDIA’s developer documentation. It tests the cluster with distributed workloads before production jobs use it, then reports which nodes failed and why.

That answers a practical question for platform teams: not merely “Are the machines alive?” but “Can these machines work together at the scale our AI job needs?”

Why ordinary health checks miss the problem

A health check may confirm that each GPU, network link, and software component responds. It does not necessarily show what happens when dozens of machines exchange data continuously.

Distributed training depends on that exchange. Each machine, called a node, handles part of the job and shares results with the others. If one node is slower, the rest may wait for it.

NVIDIA’s documentation describes NVCRE as a way to test the conditions that production workloads actually create. On Kubernetes, such tests normally require teams to configure GPU and network resources, set communication options, start all pods together, and clean up after failures.

NVCRE packages much of that setup into Kubernetes resources that administrators can inspect with kubectl and manage through Git-based workflows.

The test has three layers

An operator creates a Certification resource. It specifies which nodes to test and which categories to run, such as a communication test or a training test.

Each category creates a Workflow. The workflow manages settings, iterations, platform details, and the test’s target. It then creates a child Job, which runs the workload on the selected nodes and records measurements.

Results move back up the chain. The job records failed nodes and reasons, the workflow reports its category result, and the certification summarizes the cluster.

This structure matters because it connects a failure to both a test and a machine. A report can distinguish a hardware failure on one node from a bandwidth target missed by another.

NVCRE’s built-in catalog includes five NVIDIA Collective Communications Library variants. These test operations such as all-reduce, all-gather, all-to-all, and loopback communication, including loopback across NVIDIA NVSwitch.

The catalog also includes NVIDIA Data Center GPU Manager’s level-four diagnostic suite and NVIDIA NeMo pretraining tests using Nemotron 5 models with 8-billion and 56-billion parameters.

NVCRE detects the GPU architecture and cloud platform from the target nodes. NVIDIA says it then derives settings such as the number of GPUs per node, communication variables, and platform-specific networking.

How it narrows a failure to specific nodes

The most useful feature appears when a whole group fails.

Suppose a 64-node all-reduce test reports low bandwidth. The result may implicate every node because the machines communicate as one group. Checking them manually can take considerable engineering time.

NVCRE’s diagnose mode uses topology-aware hierarchical group testing. It splits the failing group into smaller groups, reruns the test on each half, and continues splitting failed groups until it reaches the configured minimum group size.

Groups that still fail at that point become suspect groups. The output identifies a smaller set of likely problem nodes and gives the reason for each failure.

Topology awareness means the test respects how machines are arranged and connected. A test inside one high-speed connection domain measures something different from a test spread across several racks. NVCRE lets operators choose strategies for individual nodes, rack-level groups, the full cluster, or adaptive diagnosis.

The tool also limits how many diagnostic jobs run at once. That matters because too many simultaneous tests could overload the same network fabric being measured.

It tests workloads, not just hardware

NVCRE’s WorkloadRun resource lets an operator provide a container image, select a framework, and choose the number of nodes. The framework options are torch, mpi, or exec.

The controller can create the matching training runtime, add shared memory, set communication variables, and configure platform networking. A gang-aware scheduler is optional but useful on busy shared clusters. It places all workload pods together rather than allowing some pods to occupy GPUs while waiting for others.

Operators define pass and fail rules using measured values. For example, a team could set a minimum communication bandwidth or a minimum training performance ratio.

NVIDIA says NVCRE ships with no default thresholds. The example thresholds in its documentation are illustrative values for a particular NVIDIA system class, not universal standards. Teams must decide what “ready” means for their own hardware, models, and network.

That is an important operational detail. A test can finish successfully while still failing its performance target, and NVCRE records that as a validation failure.

What NVCRE does not do

NVCRE records failed nodes and reasons, but it does not automatically cordon or taint nodes, change node conditions, or remove machines from service.

NVIDIA describes a separate NVSentinel integration that can turn failed certification results into health events. Configured policies may then quarantine or drain nodes or start external remediation. This keeps testing separate from the actions that affect scheduling.

The tool requires Kubernetes 1.29 or later, kubectl, Helm 3.x, and NVIDIA GPU Operator on the target cluster. Some GB200 and GB300 NVL72 catalog entries also require NVIDIA’s DRA Driver for GPUs.

For platform teams, NVCRE is best understood as an active readiness test: it spends cluster resources to expose failures that passive monitoring may not see. Its documentation does not establish how its diagnosis compares with existing NCCL, DCGM, or custom validation systems, so teams should measure that fit in their own environment.

The next practical step is to run a small certification against representative nodes, define local performance thresholds, and confirm that the test can operate without disrupting shared workloads.