Burn-in testing with NCCL, HPL, and NeMo: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)

Burn-in Testing with NCCL, HPL, and NeMo: A Worked Example

Burn-in testing is a critical phase in the NVIDIA-Certified Professional: AI Infrastructure exam's Cluster Test and Verification section. It ensures the reliability and stability of AI clusters under sustained workloads by stressing key components such as networking, compute, and storage. This worked example focuses on applying burn-in testing using NCCL, HPL, and NeMo within a realistic multi-node NVIDIA AI infrastructure environment.

Scenario Overview

Consider a 4-node NVIDIA DGX cluster interconnected via NVLink Switches and high-speed InfiniBand fabric. The goal is to perform burn-in testing to validate the cluster's stability and performance before production deployment. The testing will cover:

• NCCL for collective communication stress
• HPL (High-Performance Linpack) for compute and memory stress
• NeMo for AI model training workload stress

Step 1: Preparing the Environment

1. Ensure all nodes have the latest NVIDIA drivers, CUDA toolkit, and NCCL libraries installed.
2. Verify cluster interconnects (NVLink, InfiniBand) are operational and firmware is up to date.
3. Configure SSH key-based access between nodes for seamless orchestration.
4. Deploy the clusterKit node assessment tool to confirm node health and readiness.

Step 2: Burn-in Testing with NCCL

Objective: Stress the cluster's collective communication pathways to detect potential issues in NVLink Switches and cable signal quality.

1. Launch the NCCL all-reduce benchmark across all 4 nodes using the command:

mpirun -np 16 -H node1:4,node2:4,node3:4,node4:4 \ -bind-to none -map-by slot \ -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \ ./build/all_reduce_perf -b 8 -e 512M -f 2

1. Monitor the output for bandwidth consistency and error messages indicating communication faults.
2. Validate NVLink Switch operation by checking for absence of link errors in system logs.

Step 3: Burn-in Testing with HPL

Objective: Evaluate compute and memory subsystem stability under sustained floating-point intensive workload.

1. Configure the HPL input file (HPL.dat) to match cluster specifications (e.g., problem size, block size).
2. Run HPL on each node in parallel, coordinating via MPI:

mpirun -np 16 -H node1:4,node2:4,node3:4,node4:4 \ ./xhpl

1. Record the Gflops achieved and monitor for any computation errors or node failures.
2. Repeat the test for several hours to simulate prolonged load.

Step 4: Burn-in Testing with NeMo

Objective: Stress the AI training pipeline and storage I/O by running a representative NeMo model training workload.

1. Deploy a NeMo model training script (e.g., a BERT or speech recognition model) configured for distributed training across all nodes.
2. Use NVIDIA's torch.distributed backend with NCCL for efficient communication.
3. Initiate training with a moderate batch size and monitor GPU utilization, memory usage, and network throughput.
4. Check for training stability, convergence behavior, and any runtime errors.

Step 5: Analysis and Verification

• Compile logs from NCCL, HPL, and NeMo runs to identify any anomalies.
• Confirm no hardware errors, communication faults, or performance degradation occurred during burn-in.
• Validate that firmware versions on switches and BlueField devices remained stable and unchanged.
• Document results as part of cluster certification evidence.

Summary of Key Commands

• mpirun -np 16 -H node1:4,node2:4,node3:4,node4:4 ./build/all_reduce_perf -b 8 -e 512M -f 2 — NCCL burn-in test
• mpirun -np 16 -H node1:4,node2:4,node3:4,node4:4 ./xhpl — HPL burn-in test
• Distributed NeMo training script launch with NCCL backend

By following this structured, step-by-step burn-in testing approach, candidates demonstrate mastery of verifying cluster robustness using NVIDIA's critical AI infrastructure tools, a key competency for the NVIDIA-Certified Professional: AI Infrastructure certification.

More in this topic

NCCL verification including NVLink Switch validation — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Quick Reference — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Cluster Test and Verification — NVIDIA-Certified Professional: AI InfrastructureCable signal quality and correctness verification — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Practice Questions — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Storage testing — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo: Common Mistakes — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Common Mistakes — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)East-west fabric bandwidth verification — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo: Practice Questions — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo: Quick Reference — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Switch and BlueField firmware confirmation — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Single-node stress testing and HPL execution — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)