Verify operator functionality: Quick Reference — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)

Verify Operator Functionality — Quick Reference

This quick reference provides essential steps and key points for verifying the functionality of the NVIDIA Network Operator deployed in Kubernetes environments, focusing on RDMA and InfiniBand support.

Key Concepts

• NVIDIA Network Operator: A Kubernetes operator that automates deployment and management of NVIDIA networking components, including RDMA and InfiniBand drivers and configurations.
• RDMA (Remote Direct Memory Access): Enables high-throughput, low-latency networking by allowing direct memory access between computers without CPU involvement.
• InfiniBand: A high-performance network architecture commonly used in HPC and AI workloads, supported by the NVIDIA Network Operator.

Verification Checklist

1. Confirm Operator Deployment: Use kubectl get pods -n operator-namespace to verify the NVIDIA Network Operator pod is running without errors.
2. Check Custom Resource Status: Inspect the status of the NVIDIA network custom resources (CRs) with kubectl get nvidia-network-cr -n operator-namespace -o yaml to ensure desired state reconciliation.
3. Validate RDMA and InfiniBand Driver Installation: On worker nodes, confirm that RDMA and InfiniBand kernel modules and drivers are loaded using commands like lsmod | grep ib and ibv_devinfo.
4. Verify Network Interface Configuration: Check that the InfiniBand interfaces (e.g., ib0) are up and configured properly with ip link show ib0.
5. Confirm Pod Annotation and Injection: Ensure pods requiring RDMA/InfiniBand have the correct annotations and that the operator has injected necessary sidecars or configurations.
6. Review Operator Logs: Use kubectl logs operator-pod -n operator-namespace to identify any errors or warnings during reconciliation.
7. Test RDMA Connectivity: Deploy test pods with RDMA tools (e.g., ibv_rc_pingpong) to validate end-to-end RDMA communication.

Common Commands

• kubectl get pods -n operator-namespace — Check operator pod status
• kubectl describe nvidia-network-cr -n operator-namespace — Detailed CR status
• lsmod | grep ib — Verify InfiniBand kernel modules
• ibv_devinfo — RDMA device info
• ip link show ib0 — InfiniBand interface status
• kubectl logs operator-pod -n operator-namespace — Operator logs

Best Practices

• Always verify the operator version matches the Kubernetes cluster and NVIDIA driver versions.
• Monitor operator logs continuously during deployment and upgrades.
• Use dedicated namespaces to isolate NVIDIA Network Operator resources.
• Test RDMA functionality with representative workloads to ensure performance and stability.

Following this quick-reference guide ensures that the NVIDIA Network Operator is functioning correctly within Kubernetes, enabling robust RDMA and InfiniBand networking for AI workloads.

More in this topic

Verify operator functionality: Common Mistakes — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Quick Reference — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Verify operator functionality: Practice Questions — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Worked Example — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Verify operator functionality: Worked Example — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Kubernetes Integration — NVIDIA-Certified Professional: AI NetworkingVerify operator functionality — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Common Mistakes — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)Deploy the NVIDIA Network Operator for RDMA and InfiniBand: Practice Questions — Kubernetes Integration (NVIDIA-Certified Professional: AI Networking)