UFM-based monitoring of link status and bandwidth: Quick Reference — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)

UFM-Based Monitoring of Link Status and Bandwidth: Quick Reference

This quick reference provides essential facts and guidelines for using NVIDIA's Unified Fabric Manager (UFM) to monitor InfiniBand link status and bandwidth in AI networking environments.

Key Concepts

• UFM (Unified Fabric Manager): Centralized management and monitoring tool for NVIDIA InfiniBand fabrics.
• Link Status: Real-time health and connectivity state of InfiniBand links between nodes and switches.
• Bandwidth Monitoring: Measurement and reporting of data throughput on InfiniBand links to ensure performance and detect bottlenecks.

UFM Monitoring Capabilities

• Health Checks: Continuous assessment of link operational status (up/down, errors, congestion).
• Performance Metrics: Bandwidth utilization, packet rates, and error counters per link.
• Alerts and Notifications: Configurable thresholds trigger alerts for link degradation or bandwidth anomalies.
• Historical Data: Time-series data collection for trend analysis and capacity planning.

Monitoring Setup Essentials

• Initial Provisioning: Ensure all InfiniBand switches and Host Channel Adapters (HCAs) are registered with UFM.
• Access Control: Configure user roles and permissions within UFM for secure monitoring operations.
• Network Topology Discovery: Enable automatic discovery to map fabric links and nodes accurately.

Link Status Monitoring Rules

• Links are considered healthy when operational state is Active with no errors.
• UFM flags links as Down if physical connectivity is lost or administrative shutdown occurs.
• Warning states occur when error counters exceed predefined thresholds (e.g., CRC errors, link retries).
• Regular polling intervals (default 30 seconds) maintain up-to-date status.

Bandwidth Monitoring Guidelines

• Measure bandwidth as Gbps or Gb/s per link.
• Monitor both current utilization and peak usage to identify congestion.
• Set QoS policies in UFM to prioritize critical traffic and manage bandwidth allocation.
• Use adaptive routing data from UFM to correlate bandwidth usage with path selection.

Troubleshooting Tips

• Check UFM logs for link flaps or frequent state changes indicating physical issues.
• Verify firmware and driver versions on switches and HCAs for compatibility with UFM.
• Use UFM’s fabric health dashboard to quickly locate problematic links.
• Correlate bandwidth drops with application performance metrics to isolate network causes.

Example: Monitoring Link Status and Bandwidth with UFM CLI

Step 1: List all fabric links and their status:

• ufm fabric links show

Step 2: Check bandwidth utilization for a specific link:

• ufm fabric links stats --link LINK_ID

Step 3: Set alert threshold for bandwidth exceeding 80%:

• ufm alerts create --type bandwidth --threshold 80 --link LINK_ID

For detailed commands and configuration options, refer to the official NVIDIA UFM documentation at https://docs.nvidia.com/networking/ufm/.

More in this topic

UFM-based monitoring of link status and bandwidth: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Partition key (PKey) configuration for multi-tenancy — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)UFM-based monitoring of link status and bandwidth: Common Mistakes — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)NVIDIA InfiniBand Networking — NVIDIA-Certified Professional: AI NetworkingInitial provisioning and high availability setup: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Partition key (PKey) configuration for multi-tenancy: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup: Common Mistakes — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)UFM-based monitoring of link status and bandwidth — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Partition key (PKey) configuration for multi-tenancy: Common Mistakes — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)UFM-based monitoring of link status and bandwidth: Practice Questions — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup: Quick Reference — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Partition key (PKey) configuration for multi-tenancy: Practice Questions — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Practice Questions — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Initial provisioning and high availability setup: Practice Questions — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Common Mistakes — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)Partition key (PKey) configuration for multi-tenancy: Quick Reference — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)QoS and adaptive routing implementation: Quick Reference — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)