QoS and adaptive routing implementation: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)

QoS and Adaptive Routing Implementation in NVIDIA InfiniBand Networking

Quality of Service (QoS) and adaptive routing are critical components in NVIDIA InfiniBand networking environments, especially for AI workloads requiring high throughput and low latency. This worked example demonstrates a step-by-step approach to implementing QoS and adaptive routing in a realistic multi-tenant AI data center scenario.

Scenario Overview

An AI data center hosts multiple tenant clusters sharing an InfiniBand fabric. The goal is to configure QoS to prioritize latency-sensitive AI training traffic while ensuring bandwidth fairness. Additionally, adaptive routing must be enabled to optimize traffic flow dynamically and avoid congestion.

Step 1: Define QoS Policies

• Identify traffic classes: Classify traffic into High Priority (AI training jobs) and Standard Priority (background data transfers).
• Set bandwidth allocation: Reserve 70% of link bandwidth for High Priority and 30% for Standard Priority.
• Configure service levels (SLs): Assign SL 4 for High Priority and SL 2 for Standard Priority to differentiate traffic.

Step 2: Configure QoS on Switches

• Access the InfiniBand switch management interface using ibswitches CLI or NVIDIA Unified Fabric Manager (UFM).
• Create QoS groups with defined SLs and bandwidth shares:
• Example CLI command: qos create group high_priority sl=4 bandwidth=70%
• Example CLI command: qos create group standard_priority sl=2 bandwidth=30%
• Apply QoS groups to relevant ports connecting tenant clusters.

Step 3: Enable Adaptive Routing

• Verify that the InfiniBand switches support adaptive routing and that firmware is up to date.
• Enable adaptive routing globally on the fabric:
• Example CLI command: routing adaptive enable
• Configure routing tables to allow multiple paths between source and destination nodes.
• Set thresholds for congestion detection to trigger path switching dynamically.

Step 4: Validate QoS and Adaptive Routing

• Use UFM or perfquery tools to monitor traffic distribution and bandwidth utilization per SL.
• Run synthetic traffic tests simulating AI training workloads and background transfers.
• Observe adaptive routing behavior by introducing artificial congestion and verifying traffic rerouting.

Step 5: Monitor Using uFM

• Leverage NVIDIA uFM to continuously monitor link status, bandwidth usage, and QoS compliance.
• Set alerts for QoS violations or link congestion to proactively manage the fabric.

Worked Example: Configuring QoS and Adaptive Routing

Problem: A tenant's AI training cluster requires guaranteed 70% bandwidth with low latency (SL 4), while other tenants share the remaining 30% bandwidth (SL 2). Adaptive routing should optimize traffic paths to avoid congestion.

Solution:

1. Login to the InfiniBand switch CLI.
2. Create QoS groups:qos create group high_priority sl=4 bandwidth=70%qos create group standard_priority sl=2 bandwidth=30%
3. Assign QoS groups to ports connected to tenant clusters:portcfg qos high_priority port 1-10portcfg qos standard_priority port 11-20
4. Enable adaptive routing:routing adaptive enable
5. Verify routing tables allow multiple paths:routing show
6. Run traffic tests and monitor with uFM:ufm monitor links
7. Confirm that high priority traffic maintains bandwidth and latency targets, and that adaptive routing reroutes traffic during congestion.

This step-by-step approach ensures robust QoS enforcement and dynamic traffic optimization in NVIDIA InfiniBand networks, critical for high-performance AI workloads.

More in this topic

Partition key (PKey) configuration for multi-tenancy — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)NVIDIA InfiniBand Networking — NVIDIA-Certified Professional: AI NetworkingQoS and adaptive routing implementation — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)UFM-based monitoring of link status and bandwidth — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Practice Questions — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Common Mistakes — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Quick Reference — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)