Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Worked Example — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)

{ "title": "NVIDIA-Certified Professional: AI Networking - Deploying the NVIDIA Network Operator for RDMA and InfiniBand: Worked Example", "category": "NVIDIA-Certified Professional: AI Networking", "hashtags": "NVIDIA, AI Networking, Kubernetes, RDMA, InfiniBand, Network Operator", "content": "

Deploying the NVIDIA Network Operator for RDMA and InfiniBand: Worked Example

In this worked example, we will walk through the step-by-step process of deploying the NVIDIA Network Operator on a Kubernetes cluster to enable RDMA (Remote Direct Memory Access) and InfiniBand networking capabilities. This deployment is essential for high-performance AI workloads that require low latency and high throughput networking.

Scenario

You are tasked with deploying the NVIDIA Network Operator on a Kubernetes cluster consisting of three nodes equipped with NVIDIA Mellanox InfiniBand adapters. The goal is to enable RDMA and InfiniBand support for AI applications that demand accelerated networking.

Step 1: Prepare the Kubernetes Cluster

• Ensure Kubernetes cluster version is compatible (typically 1.18+).
• Verify that all nodes have NVIDIA Mellanox InfiniBand hardware installed and drivers configured.
• Install kubectl and configure access to the cluster.

Step 2: Install Prerequisites

• Install Operator Lifecycle Manager (OLM) if not already present, to manage the lifecycle of the NVIDIA Network Operator.
• Verify that helm is installed for optional deployment methods.

Step 3: Deploy the NVIDIA Network Operator

1. Clone or access the NVIDIA Network Operator repository or use the official OperatorHub manifest.
2. Apply the operator manifests to the cluster:

kubectl apply -f https://raw.githubusercontent.com/Mellanox/network-operator/master/deploy/crds/mellanox_v1_mlnxnetworkpolicy_crd.yaml kubectl apply -f https://raw.githubusercontent.com/Mellanox/network-operator/master/deploy/operator.yaml

1. Confirm the operator pods are running:

kubectl get pods -n mlnx-operator

You should see the operator pod in Running status.

Step 4: Create a Network Policy Custom Resource

The operator uses a custom resource called MlnxNetworkPolicy to configure RDMA and InfiniBand settings.

Create a YAML file rdma-policy.yaml with the following content:

apiVersion: mellanox.com/v1 kind: MlnxNetworkPolicy metadata: name: rdma-policy spec: rdma: enabled: true infiniband: enabled: true nodeSelector: matchLabels: ib: enabled

Apply the policy:

kubectl apply -f rdma-policy.yaml

Step 5: Label Nodes for InfiniBand

Label the nodes that have InfiniBand hardware:

kubectl label nodes NODE_NAME ib=enabled

Replace NODE_NAME with the actual node names.

Step 6: Verify Operator Functionality

• Check that the MlnxNetworkPolicy resource is in Ready state:

kubectl get mlnxnetworkpolicy rdma-policy -o yaml

• Look for status conditions indicating successful configuration.
• Verify that the operator has deployed the necessary DaemonSets and configured the RDMA and InfiniBand interfaces.
• Check the logs of the operator pod for any errors:

kubectl logs -n mlnx-operator POD_NAME

Step 7: Test RDMA and InfiniBand Connectivity

Deploy a test pod with RDMA tools installed and run connectivity tests between nodes to confirm that RDMA and InfiniBand are operational.

Example: Testing RDMA Connectivity

Deploy a pod with RDMA tools:

kubectl run rdma-test --image=mellanox/rdma-tools --restart=Never -- sleep 3600

Exec into the pod and run ibv_devinfo to verify InfiniBand devices are visible:

kubectl exec -it rdma-test -- ibv_devinfo

Successful output indicates the operator has correctly configured the network stack.

Summary

This example demonstrated the deployment of the NVIDIA Network Operator on Kubernetes to enable RDMA and InfiniBand networking. By applying the MlnxNetworkPolicy custom resource and labeling nodes, the operator configures the cluster to support high-performance AI workloads requiring accelerated networking.

For more detailed information, refer to the official NVIDIA Network Operator documentation at https://github.com/Mellanox/network-operator.

" }

More in this topic

Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Quick Reference — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Kubernetes Integration — NVIDIA-Certified Professional: AI NetworkingVerify operator functionality — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Common Mistakes — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Practice Questions — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)