Configure RoCE for high-speed, low-latency communication: Worked Example — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)
Configuring RoCE for High-Speed, Low-Latency Communication: A Worked Example Remote Direct Memory Access over Converged Ethernet ( RoCE ) is a...
Configuring RoCE for High-Speed, Low-Latency Communication: A Worked Example
Remote Direct Memory Access over Converged Ethernet (RoCE) is a critical technology in NVIDIA Spectrum Networking, enabling high-speed, low-latency communication essential for AI workloads. This worked example demonstrates the step-by-step process to configure RoCE in a realistic data center environment to optimize AI networking performance.
Scenario Overview
An AI data center requires configuring RoCE on NVIDIA Spectrum switches and connected servers to achieve efficient, low-latency communication between GPU clusters. The goal is to enable lossless Ethernet with priority flow control (PFC), configure Quality of Service (QoS), and verify the setup.
Step 1: Prepare the Network Environment
Identify the RoCE-capable interfaces on NVIDIA Spectrum switches and ensure the connected servers have RDMA-capable NICs (e.g., NVIDIA ConnectX).
Ensure firmware and drivers on switches and NICs support RoCE v2.
Step 2: Enable Priority Flow Control (PFC)
PFC is essential to create a lossless Ethernet fabric for RoCE traffic.
On the Spectrum switch, enter configuration mode for the relevant interfaces:
Example Commands
Switch CLI:
configure terminal
interface ethernet 1/1
priority-flow-control mode on
priority-flow-control priority 3 on
exit
Here, priority 3 is designated for RoCE traffic. Repeat for all interfaces connecting to RDMA NICs.
Step 3: Configure Quality of Service (QoS)
Assign RoCE traffic to a high-priority QoS queue to minimize latency.
Create a QoS policy that maps DSCP or VLAN priority to priority 3.
ECN helps manage congestion without dropping packets, improving RoCE performance.
Enable ECN on the switch interfaces:
Example Commands
Switch CLI:
interface ethernet 1/1
ecn enable
exit
Step 5: Configure RoCE on the Servers
On each server, configure the RDMA NIC to enable RoCE v2.
Set the PFC priority to match the switch configuration (priority 3).
Example Linux commands:
Example Commands
Linux Shell:
ethtool -A eth0 rx on tx on
rdma link add roce0 type roce
mlxconfig -d /dev/mst/mt4115_pciconf0 set RoCE=1
Step 6: Verify the Configuration
Use show priority-flow-control on the switch to confirm PFC is active on priority 3.
Run netstat -s and ibv_devinfo on servers to verify RDMA interfaces are up and RoCE is enabled.
Test latency and throughput using ib_send_bw or ib_write_bw tools.
Verification Commands
Switch CLI:
show priority-flow-control interface ethernet 1/1
Server Shell:
ibv_devinfo
ib_send_bw -d roce0
Step 7: Monitor and Troubleshoot
Utilize NVIDIA NetQ to monitor RoCE traffic in real time and detect any congestion or packet loss. In-band telemetry can help diagnose issues by providing detailed path and congestion data.
Regularly check PFC counters and ECN statistics to ensure the network remains lossless and low latency.
Summary
This example demonstrated configuring RoCE on NVIDIA Spectrum switches and servers to enable high-speed, low-latency communication. Key steps included enabling PFC, configuring QoS and ECN, setting up RoCE on NICs, and verifying the setup with diagnostic tools. Mastery of these steps is essential for the NVIDIA-Certified Professional: AI Networking exam and real-world AI network deployments.