Single-node stress testing and HPL execution: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)

Single-Node Stress Testing and HPL Execution: A Worked Example In the NVIDIA-Certified Professional: AI Infrastructure exam, single-node stress...

Single-Node Stress Testing and HPL Execution: A Worked Example

In the NVIDIA-Certified Professional: AI Infrastructure exam, single-node stress testing and High-Performance Linpack (HPL) execution are critical components of cluster test and verification. This worked example demonstrates how to perform these tasks step-by-step on a realistic AI infrastructure node to validate its readiness and performance.

Scenario

You are tasked with verifying a newly deployed AI compute node equipped with NVIDIA GPUs, NVLink interconnects, and high-speed networking. The goal is to stress test the node to ensure hardware stability and run the HPL benchmark to measure floating-point performance, confirming the node meets expected specifications.

Step 1: Prepare the Node Environment

Verify system health: Check hardware status using nvidia-smi to confirm all GPUs are recognized and operating normally.
Update drivers and CUDA: Ensure the latest NVIDIA drivers and CUDA toolkit are installed to support benchmarking tools.
Set environment variables: Configure CUDA_VISIBLE_DEVICES and other relevant variables to target all GPUs on the node.

Step 2: Execute Single-Node Stress Test

Run GPU stress test: Use nvidia-smi --stress-test or a third-party tool like stress-ng configured for GPU workloads to push GPUs to maximum utilization.
Monitor temperatures and power: Continuously monitor GPU temperature, power draw, and clock speeds to detect thermal throttling or instability.
Duration: Run the stress test for a minimum of 30 minutes to simulate sustained load.

Example

Run the following command to stress test all GPUs:

nvidia-smi --stress-test -d 1800

Monitor output for any errors or warnings.

Step 3: Configure and Run HPL Benchmark

Obtain HPL binary: Download or compile the HPL benchmark optimized for the node’s architecture.
Configure HPL.dat: Set matrix size (N), block size (NB), and process grid (P x Q) parameters based on node GPU count and memory capacity.
Launch HPL: Execute the benchmark using MPI across all GPUs on the node.
Record results: Capture the achieved GFLOPS and compare against expected performance baselines.

Example

Sample HPL.dat configuration snippet:

N 100000 NB 192 P 2 Q 2

Run the benchmark:

mpirun -np 4 ./xhpl

Analyze output for sustained performance and stability.

Step 4: Analyze and Document Results

Check for errors: Review logs for hardware errors, memory faults, or kernel panics during tests.
Validate performance: Confirm HPL GFLOPS meet or exceed vendor specifications for the node configuration.
Report findings: Document test duration, environmental conditions, and any anomalies for audit and troubleshooting.

Conclusion

This step-by-step single-node stress testing and HPL execution process ensures that the AI infrastructure node is stable, performant, and ready for integration into the larger cluster. Mastery of these procedures is essential for candidates preparing for the NVIDIA-Certified Professional: AI Infrastructure certification exam, particularly for the cluster test and verification domain.

More in this topic

Ready to test your knowledge?

Put what you've learned into practice with a quick quiz and track your progress.

Test your knowledge →