Single-node stress testing and HPL execution: Quick Reference — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)

Single-Node Stress Testing and HPL Execution: Quick Reference This quick reference covers the essential facts and procedures for single-node stress...

Single-Node Stress Testing and HPL Execution: Quick Reference

This quick reference covers the essential facts and procedures for single-node stress testing and High-Performance Linpack (HPL) execution as part of cluster test and verification in NVIDIA AI Infrastructure environments.

Key Concepts

Single-node stress testing: Validates the stability and performance of an individual compute node under heavy load.
HPL (High-Performance Linpack): Benchmark used to measure floating-point computing power of a node, critical for verifying HPC and AI workloads.
Purpose: Detect hardware faults, thermal issues, power delivery problems, and configuration errors before cluster-wide deployment.

Stress Testing Essentials

Test Duration: Typically run for several hours to ensure sustained stability.
Load Types: CPU, GPU, memory, and interconnect stress tests to simulate real-world AI workloads.
Monitoring: Track temperature, power consumption, error rates, and system logs during testing.
Common Tools: NVIDIA-supplied stress utilities, vendor-specific diagnostics, and system monitoring tools.

HPL Execution Overview

Benchmark Objective: Solve a dense system of linear equations to measure node floating-point performance.
Configuration: Set matrix size (N) and block size (NB) appropriate to node memory and architecture.
Execution: Run HPL with optimized BLAS libraries (e.g., cuBLAS) and tuned parameters for peak performance.
Output Metrics: Gflops achieved, residual error, and runtime stability.

Best Practices

Ensure firmware and drivers are up to date before testing.
Use consistent environment variables to replicate production settings.
Run tests during maintenance windows to avoid impacting other workloads.
Document all test configurations and results for audit and troubleshooting.

Common Issues and Troubleshooting

Thermal throttling: Check cooling systems and airflow if performance drops during tests.
Memory errors: Use memory diagnostics if HPL reports residual errors or crashes.
Power instability: Verify power supply and cabling integrity if node fails stress tests.

Worked Example

Problem: Run HPL on a single NVIDIA DGX node to verify peak floating-point performance.

Solution:

Set matrix size N = 100000 and block size NB = 192 based on node memory capacity.
Configure environment to use cuBLAS and optimized MPI settings.
Execute HPL benchmark and monitor GPU temperatures and power draw.
Confirm Gflops output meets expected performance baseline and no errors are reported.

More in this topic

Ready to test your knowledge?

Put what you've learned into practice with a quick quiz and track your progress.

Test your knowledge →