Partition key (PKey) configuration for multi-tenancy: Worked Example — NVIDIA InfiniBand Networking (NVIDIA-Certified Professional: AI Networking)

Partition Key (PKey) Configuration for Multi-Tenancy in NVIDIA InfiniBand Networking In NVIDIA InfiniBand environments, Partition Keys (PKeys) are...

Partition Key (PKey) Configuration for Multi-Tenancy in NVIDIA InfiniBand Networking

In NVIDIA InfiniBand environments, Partition Keys (PKeys) are essential for enabling multi-tenancy by isolating traffic between different user groups or tenants on the same physical network. Proper PKey configuration ensures secure and efficient communication within each tenant's partition without interference.

Scenario Overview

Consider a data center hosting two AI research teams, Team Alpha and Team Beta, sharing the same InfiniBand fabric. Each team requires isolated network partitions to prevent cross-tenant traffic and maintain security and performance. We will configure PKeys to create two separate partitions and assign appropriate PKeys to each team's nodes.

Step-by-Step Worked Example

Identify the Number of Partitions NeededWe need two partitions: one for Team Alpha and one for Team Beta.
Assign PKey ValuesPKeys are 16-bit values where the lower 15 bits represent the partition ID and the highest bit indicates membership (full or limited).For example:
- Team Alpha: PKey = 0x8001 (full membership, partition ID 0x0001)
- Team Beta: PKey = 0x8002 (full membership, partition ID 0x0002)
Configure PKeys on SwitchesUsing the NVIDIA Unified Fabric Manager (uFM) or switch CLI, add the PKeys to the switch's PKey table:
- Command example: ibswitchcfg --add-pkey 0x8001
- Command example: ibswitchcfg --add-pkey 0x8002
Assign PKeys to Host Channel Adapters (HCAs)On each host node, configure the HCA to use the appropriate PKey:
- Team Alpha nodes: assign PKey 0x8001
- Team Beta nodes: assign PKey 0x8002
This is typically done via the ibv_devinfo and ibv_rc_pingpong tools or through configuration files specific to the InfiniBand stack.
Verify Partition MembershipCheck that each node's HCA reports the correct PKey membership:
- Use ibv_devinfo -v to list PKeys and verify the assigned PKey is present and marked as full membership.
Test Connectivity Within and Across PartitionsPerform connectivity tests:
- Within the same partition (e.g., Team Alpha node to Team Alpha node): communication should succeed.
- Across partitions (e.g., Team Alpha node to Team Beta node): communication should fail, confirming isolation.

Worked Example Summary

Problem: Configure PKeys for two tenant teams on a shared NVIDIA InfiniBand fabric to ensure network isolation.

Solution Steps:

Define PKeys 0x8001 and 0x8002 for Team Alpha and Team Beta respectively.
Add these PKeys to the switch PKey table using ibswitchcfg.
Assign corresponding PKeys to each team's nodes' HCAs.
Verify PKey membership and test connectivity to ensure isolation.

This configuration enables secure multi-tenancy on the InfiniBand network, supporting concurrent AI workloads without interference.

For more detailed commands and best practices, refer to the NVIDIA InfiniBand User Guide.

More in this topic

Ready to test your knowledge?

Put what you've learned into practice with a quick quiz and track your progress.

Test your knowledge →