Monitor in real time with NetQ: Worked Example — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)

Monitoring NVIDIA Spectrum Networks in Real Time with NetQ: A Worked Example NVIDIA NetQ is a powerful network operations tool designed to provide...

Monitoring NVIDIA Spectrum Networks in Real Time with NetQ: A Worked Example

NVIDIA NetQ is a powerful network operations tool designed to provide real-time visibility, telemetry, and troubleshooting capabilities for NVIDIA Spectrum-based AI networking environments. This worked example demonstrates how to monitor a high-performance AI cluster network using NetQ, focusing on key operational tasks relevant to the NVIDIA-Certified Professional: AI Networking certification.

Scenario Overview

You are the network engineer responsible for a multi-node AI training cluster connected via NVIDIA Spectrum switches. The cluster uses RoCE (RDMA over Converged Ethernet) for low-latency communication. Your goal is to monitor network health, verify QoS policies, and quickly identify any congestion or packet drops affecting AI workloads.

Step 1: Accessing the NetQ Dashboard

Log into the NetQ controller interface via the web portal or CLI.
Navigate to the Topology View to visualize the Spectrum switch fabric and connected hosts.
Confirm that all Spectrum switches and AI compute nodes are reporting telemetry data.

Step 2: Verifying RoCE Configuration and Health

Use NetQ’s netq show roce command to list RoCE-enabled interfaces.
Check for any packet drops or latency spikes on these interfaces.
Example command: netq show roce --dscp 46 to filter for priority traffic.

Step 3: Monitoring QoS and ECN Metrics

Run netq show qos to verify that QoS policies are applied correctly on Spectrum switches.
Check ECN (Explicit Congestion Notification) counters to detect early congestion signals.
Example: netq show ecn --interface swp1 to view ECN marks on a specific port.

Step 4: Diagnosing with In-Band Telemetry (INT)

Enable INT on selected flows to collect detailed path and latency data.
Use netq int trace to capture and analyze telemetry packets.
Identify any hops causing increased latency or packet loss.

Step 5: Real-Time Alerts and Event Monitoring

Configure NetQ alert policies to notify on threshold breaches such as high packet drop rates or QoS violations.
Example: Set an alert for >1% packet drops on RoCE interfaces.
Monitor the Events tab for real-time notifications and historical event logs.

Step 6: Reporting and Continuous Monitoring

Generate periodic reports summarizing network performance metrics.
Use NetQ’s API to integrate monitoring data with external dashboards or automation tools.

Worked Example: Identifying a RoCE Packet Drop Issue

Problem: AI training jobs report degraded performance. Suspected network packet drops on RoCE traffic.

Solution Steps:

Run netq show roce to list RoCE-enabled interfaces and check drop counters.
Identify interface swp3 on switch sp1 showing 2% packet drops.
Use netq show ecn --interface swp3 to check for congestion signals; ECN marks are elevated.
Enable INT tracing on flows through swp3 with netq int trace --interface swp3.
Analyze telemetry data to find a downstream switch port with buffer overflow.
Adjust QoS priority flow control settings on the affected port to alleviate congestion.
Monitor with NetQ to confirm packet drops reduce to near zero and AI job performance improves.

This step-by-step approach illustrates how NetQ enables rapid diagnosis and resolution of network issues impacting AI workloads in NVIDIA Spectrum environments.

For more detailed guidance on NVIDIA Spectrum Networking and NetQ, refer to the official NVIDIA documentation and training resources.

More in this topic

Ready to test your knowledge?

Put what you've learned into practice with a quick quiz and track your progress.

Test your knowledge →