ClusterKit node assessment: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)
Cluster Test and Verification: ClusterKit Node Assessment The NVIDIA-Certified Professional: AI Infrastructure certification emphasizes the...
Cluster Test and Verification: ClusterKit Node Assessment
The NVIDIA-Certified Professional: AI Infrastructure certification emphasizes the importance of thorough testing and verification of AI infrastructure. One critical component of this is the ClusterKit node assessment, which ensures that each node in the cluster is functioning optimally. This article provides a detailed, step-by-step worked example of how to conduct a ClusterKit node assessment.
Worked Example: ClusterKit Node Assessment
Scenario: You are tasked with assessing a new cluster node that will be integrated into an existing NVIDIA AI infrastructure. The goal is to ensure that the node meets performance and reliability standards before deployment.
Step 1: Prepare the Environment
Before starting the assessment, ensure that the following are in place:
- The node is physically connected to the cluster.
- All necessary software and drivers are installed, including the latest BlueField firmware.
- Access to monitoring tools and performance benchmarks is available.
Step 2: Conduct Single-Node Stress Testing
Begin by performing a single-node stress test to evaluate the node's performance under load:
- Run a stress test using a tool like HPL (High-Performance Linpack) to measure the node's floating-point performance.
- Monitor CPU and GPU utilization, memory bandwidth, and temperature during the test.
- Ensure that the node maintains stable performance without overheating or throttling.
Step 3: Verify nCCL and NVLink Switch
Next, verify the NVIDIA Collective Communications Library (nCCL) and NVLink switch:
- Execute nCCL tests to ensure that data transfers between GPUs are efficient and error-free.
- Validate the NVLink Switch by checking the connectivity and performance metrics.
Step 4: Assess Cable Signal Quality
Check the quality of the cables connecting the node:
- Use a cable tester to verify signal integrity and correctness.
- Replace any cables that do not meet the required specifications.
Step 5: Confirm Firmware Versions
Ensure that the firmware on the switch and BlueField is up to date:
- Access the firmware management interface.
- Compare the installed versions against the latest available from NVIDIA.
- Update any outdated firmware accordingly.
Step 6: Perform East-West Fabric Bandwidth Verification
Test the bandwidth between nodes in the cluster:
- Use benchmarking tools to measure east-west traffic.
- Ensure that the bandwidth meets the expected thresholds for your application.
Step 7: Conduct Burn-in Testing
Finally, perform burn-in testing:
- Run a combination of NCCL, HPL, and NeMo workloads for an extended period (e.g., 24-48 hours).
- Monitor for any errors, crashes, or performance degradation during this period.
By following these steps, you can ensure that the ClusterKit node is fully assessed and ready for deployment in your NVIDIA AI infrastructure. This thorough testing process is essential for maintaining the reliability and performance of AI applications.