Command-line diagnostics (ibping, ibstat, ibdiagnet): Worked Example — Troubleshooting Tools (NVIDIA-Certified Professional: AI Networking)
{ "title": "NVIDIA-Certified Professional: AI Networking - Command-Line Diagnostics Worked Example", "category": "NVIDIA-Certified Professional: AI...
{ "title": "NVIDIA-Certified Professional: AI Networking - Command-Line Diagnostics Worked Example", "category": "NVIDIA-Certified Professional: AI Networking", "hashtags": "NVIDIA, AI Networking, troubleshooting, ibping, ibstat, ibdiagnet", "content": "
Command-Line Diagnostics in NVIDIA AI Networking: A Worked Example
Effective troubleshooting in AI networking environments requires mastery of key command-line tools such as ibping, ibstat, and ibdiagnet. These utilities provide critical insights into InfiniBand fabric health, connectivity, and performance, essential for maintaining low-latency, high-throughput AI workloads.
Scenario Overview
Imagine a scenario where an AI cluster experiences intermittent latency spikes and occasional packet loss during GPU-accelerated training jobs. The goal is to diagnose the root cause using command-line diagnostics to verify connectivity, port status, and fabric integrity.
Step 1: Verify Basic Connectivity with ibping
ibping tests the reachability between nodes over the InfiniBand network, analogous to ICMP ping in Ethernet.
- Command: ibping -S on the server node to start a ping server.
- On the client node: ibping -c 10 SERVER_GUID sends 10 ping packets to the server's GUID.
Reasoning: Successful replies confirm that the basic InfiniBand connection is active and responsive.
Example Output
Sent 10, received 10, 0% packet loss, average latency 5 us
This indicates stable connectivity with low latency.
Step 2: Check Port and Link Status with ibstat
ibstat provides detailed information about the InfiniBand ports on a node, including link state, physical state, and rate.
- Command: ibstat
Reasoning: Identifying ports in a down or error state helps isolate hardware or link issues.
Example Output
CA 'mlx5_0' CA type: MT26428 Number of ports: 2 Port 1: State: Active Physical state: LinkUp Rate: 100 Gb/sec (4X FDR) Port 2: State: Down Physical state: Polling
Port 2 is down, which could indicate a disconnected cable or faulty hardware.
Step 3: Diagnose Fabric Issues with ibdiagnet
ibdiagnet performs comprehensive diagnostics on the InfiniBand fabric, checking for errors, congestion, and configuration mismatches.
- Command: ibdiagnet -r runs a full report.
Reasoning: This tool helps detect subtle issues like excessive retries, congestion, or topology problems affecting performance.
Example Output Summary
PortCounters: Port 1: 0 errors, 0 congestion events Port 2: 100 link down events Congestion: Detected on switch SW1 port 3
The report highlights frequent link down events on Port 2 and congestion on a switch port, guiding targeted remediation.
Summary and Next Steps
Using ibping, ibstat, and ibdiagnet in sequence provides a structured approach to diagnosing AI networking issues:
- Confirm basic connectivity and latency with ibping.
- Check physical port status and link health with ibstat.
- Analyze fabric-wide errors and congestion with ibdiagnet.
This methodical troubleshooting enables rapid identification of hardware faults, connectivity problems, and fabric congestion, essential for maintaining optimal AI networking performance.
For more detailed guidance on these tools and their options, consult the official NVIDIA networking documentation and the Mellanox diagnostic tools guide.
" }
More in this topic
Ready to test your knowledge?
Put what you've learned into practice with a quick quiz and track your progress.
Test your knowledge →