Initial provisioning and high availability setup: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)
Initial Provisioning and High Availability Setup in NVIDIA InfiniBand Networking NVIDIA InfiniBand Networking is a critical component for...
Initial Provisioning and High Availability Setup in NVIDIA InfiniBand Networking
NVIDIA InfiniBand Networking is a critical component for high-performance AI environments, providing low latency and high throughput. This worked example focuses on the initial provisioning and high availability (HA) setup—a foundational step for deploying robust, scalable AI networking infrastructures.
Scenario Overview
Consider a data center environment preparing to deploy an AI cluster with 8 InfiniBand switches. The goal is to provision the network for initial operation and configure high availability to ensure continuous uptime in case of switch or link failures.
Step 1: Physical and Logical Network Preparation
- Verify all InfiniBand switches are racked and powered on.
- Connect switches in a redundant topology, typically a fat-tree or mesh, to support HA.
- Assign management IP addresses to each switch for out-of-band management.
Step 2: Initial Provisioning of Switches
- Access each switch: Use SSH to connect to the switch management interface.
- Firmware verification: Confirm all switches run compatible firmware versions to avoid interoperability issues.
- Configure basic switch settings: Set hostname, management IP, and enable SNMP for monitoring.
- Enable subnet manager: Designate one switch as the Subnet Manager (SM) to control routing and address assignment across the fabric.
Step 3: Configure High Availability
- Redundant SM setup: Configure a backup Subnet Manager on a secondary switch to take over if the primary fails.
- Link aggregation: Enable link aggregation (LAG) on switch ports connecting to critical nodes to increase bandwidth and provide failover.
- Spanning Tree Protocol (STP): Although InfiniBand uses routing protocols, ensure no loops exist in the physical topology to prevent broadcast storms.
Step 4: Validate Network Fabric
- Run ibnetdiscover or equivalent tools to verify the fabric topology and connectivity.
- Check that all nodes and switches appear correctly and that the SM is active.
- Test failover by simulating switch or link failure and confirming the backup SM and redundant links maintain fabric operation.
Worked Example: Provisioning and HA Setup on a 4-Switch Fabric
Problem: Provision a 4-switch InfiniBand fabric with high availability for an AI cluster.
Solution:
- Connect switches: Arrange switches in a ring topology with redundant links.
- Assign management IPs: Switch1: 192.168.1.1, Switch2: 192.168.1.2, Switch3: 192.168.1.3, Switch4: 192.168.1.4.
- Access Switch1 via SSH: ssh admin@192.168.1.1
- Set hostname: hostname Switch1
- Enable SM on Switch1: sm start
- Configure backup SM on Switch2: ssh admin@192.168.1.2, then sm backup start
- Enable LAG on links between Switch1-Switch2 and Switch3-Switch4: Use switch CLI commands to aggregate ports.
- Verify fabric: Run ibnetdiscover from a connected host to confirm all switches and links are active.
- Test failover: Disconnect Switch1’s SM port and confirm Switch2 takes over SM duties without fabric disruption.
This stepwise approach ensures a resilient InfiniBand fabric ready for demanding AI workloads.
For more detailed guidance on NVIDIA InfiniBand provisioning and high availability, refer to the official NVIDIA Networking documentation at https://www.nvidia.com/en-us/networking/.
More in this topic
Ready to test your knowledge?
Put what you've learned into practice with a quick quiz and track your progress.
Test your knowledge →