Initial provisioning and high availability setup: Quick Reference — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)

Initial Provisioning and High Availability Setup for NVIDIA InfiniBand Networking

This quick reference covers the essential facts and procedures for initial provisioning and setting up high availability (HA) in NVIDIA InfiniBand environments, a critical component of the NVIDIA-Certified Professional: AI Networking exam.

1. Initial Provisioning Overview

• Purpose: Prepare InfiniBand fabric for operation by configuring switches, hosts, and subnet manager.
• Subnet Manager (SM): Central controller responsible for fabric management, routing, and device discovery.
• Key Steps:
• Power on all switches and hosts in the fabric.
• Ensure SM is enabled and running on a designated switch or external device.
• Verify fabric topology and connectivity using ibnetdiscover or equivalent tools.
• Configure basic switch parameters such as GUIDs and port states.

2. High Availability (HA) Setup

• Goal: Ensure continuous fabric operation despite failures of SM or switches.
• SM Redundancy: Deploy multiple SM instances in standby mode for failover.
• Switch Redundancy: Use redundant power supplies and network paths.
• HA Configuration Steps:
• Configure primary and secondary SMs with identical configurations.
• Enable SM failover protocols to allow seamless transition.
• Validate HA by simulating SM failure and confirming fabric stability.

3. Important Definitions

• Subnet Manager (SM): Software component managing InfiniBand fabric routing and device configuration.
• GUID: Globally Unique Identifier assigned to each InfiniBand device and port.
• Port States: Physical and logical states of switch ports (e.g., Down, Active).
• Fabric: The interconnected network of InfiniBand switches and hosts.

4. Key Commands and Tools

• ibstat: Displays port status and configuration.
• ibnetdiscover: Discovers and displays fabric topology.
• opensm: Starts the Subnet Manager service.
• ufmcli: Used for monitoring and managing fabric health (though more relevant to monitoring than provisioning).

5. Best Practices

• Always document initial configurations and backup SM settings.
• Test HA failover regularly to ensure reliability.
• Keep firmware and software versions consistent across switches.
• Use VLANs and partition keys (PKeys) to isolate traffic as needed (covered in other topics).

Note: This quick reference focuses solely on initial provisioning and HA setup. Other aspects like PKey configuration, QoS, adaptive routing, and uFM monitoring are covered separately.

More in this topic

Partition key (PKey) configuration for multi-tenancy — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)NVIDIA InfiniBand Networking — NVIDIA-Certified Professional: AI NetworkingInitial provisioning and high availability setup: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup: Common Mistakes — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)UFM-based monitoring of link status and bandwidth — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Practice Questions — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup: Practice Questions — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Common Mistakes — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Quick Reference — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)