Configure networking for cluster nodes, DPUs, and switches: Worked Example — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)

Configuring Networking for Cluster Nodes, DPUs, and Switches: A Worked Example

In the NVIDIA-Certified Professional: AI Operations certification, configuring networking for cluster nodes, Data Processing Units (DPUs), and switches is a critical skill. This example walks through a realistic scenario to demonstrate step-by-step how to configure networking in an NVIDIA AI cluster environment using Base Command Manager (BCM) and DOCA Services.

Scenario Overview

You are tasked with configuring the networking for a new AI cluster consisting of:

• 5 compute nodes equipped with NVIDIA DPUs
• A set of NVIDIA Spectrum switches
• BCM installed for cluster management
• Kubernetes installed on the cluster nodes

The goal is to ensure seamless communication between nodes, DPUs, and switches, enabling optimal AI workload performance and management.

Step 1: Verify Physical Connectivity and Switch Configuration

Before configuring software networking, confirm physical connections:

• Ensure each compute node's DPU is connected to the Spectrum switch via the appropriate high-speed link (e.g., 100GbE).
• Check switch port configurations for correct VLAN assignments and link aggregation settings.

Use the switch management interface or CLI to verify port status and set VLANs if necessary:

• Example CLI command on Spectrum switch: show interfaces status
• Configure VLANs for AI traffic isolation: vlan 100 and assign ports accordingly.

Step 2: Configure Network Interfaces on Cluster Nodes

On each compute node, configure the network interfaces associated with the DPU:

1. Identify the DPU interface name using ip link.
2. Assign static IP addresses or configure DHCP as per cluster policy.
3. Configure bonding or teaming if multiple interfaces are used for redundancy.

Example commands on a Linux node:

• sudo ip link set eth1 up
• sudo ip addr add 192.168.1.10/24 dev eth1

Step 3: Initialize and Configure DOCA Services on the DPUs

Deploy DOCA Services on the DPU ARM cores to enable advanced networking features:

1. Use BCM to deploy DOCA Services via the UI or CLI.
2. Verify DOCA Services status with doca status on each DPU.
3. Configure DOCA networking policies for traffic steering and security.

Step 4: Integrate Networking with Kubernetes

Ensure Kubernetes networking is configured to leverage the DPU capabilities:

• Install NVIDIA CNI plugins compatible with DOCA Services.
• Configure Kubernetes network policies to allow pod-to-pod communication across nodes.
• Use BCM Base View to monitor network performance and troubleshoot connectivity issues.

Step 5: Validate Network Configuration and Performance

Perform connectivity tests and monitor network health:

• Ping between cluster nodes and verify latency and packet loss.
• Use bcmmgr commands or BCM UI to check link status and throughput.
• Check switch logs for errors or misconfigurations.

Step 6: Troubleshoot Common Issues

If issues arise, follow these diagnostic steps:

• Verify physical cable connections and switch port status.
• Check IP addressing and subnet masks for consistency.
• Review DOCA Services logs on DPUs for errors.
• Use BCM’s diagnostic tools to identify bottlenecks or misconfigurations.

Worked Example: Configuring Networking on Node 3

Problem: Node 3’s DPU interface is not communicating with the switch.

Solution:

1. Run ip link on Node 3 to confirm the DPU interface is eth2.
2. Check interface status: sudo ip link show eth2 shows interface is down.
3. Bring interface up: sudo ip link set eth2 up.
4. Assign IP address: sudo ip addr add 192.168.1.13/24 dev eth2.
5. Ping switch IP (192.168.1.1): ping 192.168.1.1 succeeds.
6. Check switch port status for Node 3’s port; it shows as disabled.
7. Enable port on switch: interface ethernet 1/3 then no shutdown.
8. Re-test ping from Node 3; communication is now successful.

This stepwise approach resolved the connectivity issue by addressing both node interface and switch port configuration.

More in this topic

Apply patches, firmware updates, and image synchronization: Worked Example — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Manage job scheduling with Slurm or Kubernetes — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Administer user accounts, roles, and permissions in BCM — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Configure networking for cluster nodes, DPUs, and switches: Common Mistakes — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Diagnose and resolve cluster issues — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Apply patches, firmware updates, and image synchronization: Common Mistakes — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Installation and Deployment — NVIDIA-Certified Professional: AI OperationsDescribe the Mission Control toolkit: Worked Example — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Describe the Mission Control toolkit — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Describe the Mission Control toolkit: Common Mistakes — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Install Run:ai and Slurm — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Diagnose and resolve cluster issues: Worked Example — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Administer user accounts, roles, and permissions in BCM: Quick Reference — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Administer user accounts, roles, and permissions in BCM: Worked Example — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Diagnose and resolve cluster issues: Practice Questions — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Monitor performance with Base Command Manager Base View — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Administer user accounts, roles, and permissions in BCM: Common Mistakes — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Install and initialize Kubernetes on NVIDIA hosts using BCM — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Apply patches, firmware updates, and image synchronization: Practice Questions — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Administer user accounts, roles, and permissions in BCM: Practice Questions — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Configure networking for cluster nodes, DPUs, and switches: Practice Questions — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Diagnose and resolve cluster issues: Common Mistakes — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Diagnose and resolve cluster issues: Quick Reference — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Apply patches, firmware updates, and image synchronization — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Apply patches, firmware updates, and image synchronization: Quick Reference — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Describe the Mission Control toolkit: Practice Questions — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Deploy DOCA Services on DPU Arm — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Configure networking for cluster nodes, DPUs, and switches: Quick Reference — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Configure networking for cluster nodes, DPUs, and switches — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)Describe the Mission Control toolkit: Quick Reference — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)