Diagnose with in-band telemetry and WJH: Worked Example — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)

Diagnosing NVIDIA Spectrum Networks with In-Band Telemetry and WJH: A Worked Example

NVIDIA Spectrum Networking leverages advanced telemetry and diagnostic tools to maintain high-performance, low-latency AI networking environments. This worked example focuses on using in-band telemetry and WJH (Wireline Jitter and Health) to diagnose network issues in a realistic AI data center scenario.

Scenario Overview

An AI research facility experiences intermittent latency spikes and packet loss in its high-speed RoCE (RDMA over Converged Ethernet) fabric. The network team suspects congestion and wants to verify Quality of Service (QoS) policies and identify jitter sources using in-band telemetry and WJH diagnostics.

Step 1: Enable In-Band Telemetry on Spectrum Switches

• Access the Spectrum switch CLI or management interface.
• Configure the telemetry pipeline to embed in-band network telemetry (INT) data into packets traversing the network.
• Example CLI command: int enable telemetry pipeline
• Verify telemetry is active by checking telemetry counters and flow statistics.

Step 2: Deploy WJH for Jitter and Health Monitoring

• Install and configure the WJH diagnostic module on the Spectrum switch.
• Set WJH to monitor latency variation (jitter) on critical RoCE flows.
• Configure thresholds for jitter alerts based on AI workload sensitivity.
• Example: wjh monitor add flow roce1 threshold 5us

Step 3: Capture Telemetry Data and Analyze

• Use NVIDIA NetQ or equivalent telemetry collector to gather INT data from the network.
• Examine telemetry reports for packet delay variation, queue depths, and congestion points.
• Identify flows with abnormal latency spikes or packet drops.

Step 4: Correlate WJH Alerts with Telemetry Insights

• Review WJH-generated jitter alerts and timestamps.
• Cross-reference with telemetry data to pinpoint exact switch ports and queues causing jitter.
• Example finding: Port 5 shows queue buildup during peak AI training hours causing jitter above threshold.

Step 5: Implement Remediation

• Adjust QoS policies to prioritize RoCE traffic more aggressively on affected ports.
• Enable Priority Flow Control (PFC) to reduce packet loss during congestion.
• Re-run telemetry and WJH diagnostics to confirm improvements.

Worked Example Summary

Problem: Latency spikes and packet loss in RoCE fabric impacting AI training.

Solution Steps:

1. Enabled in-band telemetry on Spectrum switches to collect real-time network data.
2. Configured WJH to monitor jitter on critical flows.
3. Collected and analyzed telemetry data to identify congestion points.
4. Correlated WJH jitter alerts with telemetry to locate problematic ports.
5. Modified QoS and enabled PFC to mitigate congestion and jitter.
6. Validated network health improvements with follow-up telemetry and WJH monitoring.

This diagnostic approach ensures AI workloads maintain the required low-latency, high-throughput networking performance essential for NVIDIA Spectrum environments.

More in this topic

Simulate networks with NVIDIA Air: Worked Example — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)NVIDIA Spectrum Networking — NVIDIA-Certified Professional: AI NetworkingEnable and verify QoS, ECN, and priority flow control: Practice Questions — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Simulate networks with NVIDIA Air: Common Mistakes — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Simulate networks with NVIDIA Air: Practice Questions — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Enable and verify QoS, ECN, and priority flow control — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Simulate networks with NVIDIA Air — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Configure RoCE for high-speed, low-latency communication — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Simulate networks with NVIDIA Air: Quick Reference — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Monitor in real time with NetQ — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Diagnose with in-band telemetry and WJH: Common Mistakes — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Enable and verify QoS, ECN, and priority flow control: Common Mistakes — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Diagnose with in-band telemetry and WJH — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Diagnose with in-band telemetry and WJH: Practice Questions — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Set up multi-tenancy with BGP-EVPN — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Enable and verify QoS, ECN, and priority flow control: Worked Example — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Install NVIDIA DOCA and configure SuperNIC functionality — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Enable and verify QoS, ECN, and priority flow control: Quick Reference — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)Diagnose with in-band telemetry and WJH: Quick Reference — NVIDIA Spectrum Networking (NVIDIA-Certified Professional: AI Networking)