East-west fabric bandwidth verification: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)

East-West Fabric Bandwidth Verification: Worked Example

In the context of the NVIDIA-Certified Professional: AI Infrastructure certification, east-west fabric bandwidth verification is a critical step in ensuring the high-performance communication required within AI clusters. This process validates the data transfer rates between nodes across the cluster fabric, which is essential for distributed AI workloads.

Scenario Overview

Consider a 4-node NVIDIA AI cluster interconnected via NVSwitch fabric. The goal is to verify that the east-west bandwidth between nodes meets the expected performance thresholds, ensuring efficient data exchange during AI training and inference tasks.

Step 1: Preparation and Environment Setup

• Ensure all nodes are powered on and connected through the NVSwitch fabric.
• Verify that the latest clusterKit and NVIDIA drivers are installed on each node.
• Confirm firmware versions on switches and BlueField DPUs are up to date to avoid compatibility issues.
• Disable any background processes that could interfere with bandwidth testing.

Step 2: Selecting the Bandwidth Testing Tool

Use nCCL (NVIDIA Collective Communications Library) bandwidth tests, which are designed to measure inter-node communication performance in GPU clusters.

Step 3: Running the nCCL Bandwidth Test

1. On each node, launch the nCCL bandwidth test with parameters targeting all other nodes in the cluster.
2. Example command:

Command:

./build/all_reduce_perf -b 8 -e 512M -f 2 -g 8

Explanation: This runs an all-reduce operation with message sizes from 8 bytes to 512 megabytes across 8 GPUs per node.

1. Monitor the output for bandwidth metrics reported for each pair of nodes (east-west links).
2. Record the bandwidth values for analysis.

Step 4: Analyzing Bandwidth Results

• Compare the measured bandwidth against the expected theoretical maximum of the NVSwitch fabric (e.g., 300 GB/s per link).
• Identify any links with significantly lower bandwidth, which may indicate cable issues, firmware mismatches, or hardware faults.

Step 5: Troubleshooting and Verification

• If bandwidth is below expectations, verify cable signal quality and correctness using diagnostic tools.
• Check switch and BlueField firmware versions and update if necessary.
• Re-run the nCCL test after corrective actions to confirm improvements.

Step 6: Documenting and Reporting

Compile a report detailing the test configuration, commands used, bandwidth results, and any corrective actions taken. This documentation supports cluster validation and certification requirements.

Summary

East-west fabric bandwidth verification using nCCL bandwidth tests is a systematic process involving preparation, execution, analysis, and troubleshooting. This ensures the AI cluster fabric delivers the necessary performance for demanding AI workloads, a key competency for the NVIDIA-Certified Professional: AI Infrastructure certification.

More in this topic

NCCL verification including NVLink Switch validation — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)East-west fabric bandwidth verification: Quick Reference — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Switch and BlueField firmware confirmation: Quick Reference — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Quick Reference — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Cluster Test and Verification — NVIDIA-Certified Professional: AI InfrastructureCable signal quality and correctness verification — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Practice Questions — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Storage testing — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo: Common Mistakes — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Switch and BlueField firmware confirmation: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Switch and BlueField firmware confirmation: Common Mistakes — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)East-west fabric bandwidth verification: Common Mistakes — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment: Common Mistakes — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)East-west fabric bandwidth verification — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo: Worked Example — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo: Practice Questions — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Burn-in testing with NCCL, HPL, and NeMo: Quick Reference — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)ClusterKit node assessment — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)East-west fabric bandwidth verification: Practice Questions — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Switch and BlueField firmware confirmation: Practice Questions — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Switch and BlueField firmware confirmation — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)Single-node stress testing and HPL execution — Cluster Test and Verification (NVIDIA-Certified Professional: AI Infrastructure)