Configure Multi-Instance GPU (MIG): Worked Example — Administration (NVIDIA-Certified Professional: AI Operations)

Configuring Multi-Instance GPU (MIG): A Worked Example for NVIDIA-Certified Professional: AI Operations

Multi-Instance GPU (MIG) technology enables partitioning a single NVIDIA A100 or later GPU into multiple isolated instances, each with dedicated resources. This capability is essential for optimizing GPU utilization in AI workloads, allowing multiple users or processes to share a GPU securely and efficiently.

Scenario Overview

Imagine you are an AI Operations engineer tasked with configuring MIG on an NVIDIA A100 GPU within a Slurm cluster environment to support multiple concurrent AI training jobs. The goal is to partition the GPU into four equal instances, each isolated and ready for deployment in Kubernetes pods.

Step 1: Verify GPU and Driver Compatibility

• Ensure the GPU is an NVIDIA A100 or newer model supporting MIG.
• Check that the NVIDIA driver version is 450.80.02 or later, which supports MIG functionality.
• Run nvidia-smi to confirm GPU details and driver version.

Command

nvidia-smi

Expected Output

Displays GPU model, driver version, and current MIG mode status.

Step 2: Enable MIG Mode on the GPU

MIG mode must be enabled to partition the GPU. This requires the GPU to be in a clean state with no active processes.

• Run nvidia-smi -i 0 -mig 1 to enable MIG mode on GPU 0.
• Verify the change by running nvidia-smi again; the GPU should now show MIG mode enabled.

Commands

sudo nvidia-smi -i 0 -mig 1nvidia-smi

Step 3: Create MIG Instances

Decide the profile for each MIG instance. For four equal partitions, use the 1g.5gb profile, which allocates one GPU instance with 5GB memory.

• Use nvidia-smi mig -cgi 19,19,19,19 -i 0 to create four instances with profile ID 19 (1g.5gb).
• Confirm instances creation with nvidia-smi mig -lgi -i 0.

Commands

sudo nvidia-smi mig -cgi 19,19,19,19 -i 0nvidia-smi mig -lgi -i 0

Step 4: Assign MIG Instances to Kubernetes Pods

Each MIG instance appears as a separate GPU device, which Kubernetes can schedule independently.

• Ensure the NVIDIA device plugin for Kubernetes is installed and configured to support MIG.
• In the pod specification, request the specific MIG device by its UUID or device name.
• Deploy pods with resource requests matching the MIG instances.

Pod Spec Snippet

resources: limits: nvidia.com/mig-1g.5gb: 1

Step 5: Monitor and Troubleshoot

• Use nvidia-smi and nvidia-smi mig -lgi to monitor MIG instance health and utilization.
• If an instance is not functioning, consider resetting the GPU or re-creating the MIG instances.

Summary

This worked example demonstrated configuring Multi-Instance GPU (MIG) on an NVIDIA A100 GPU by enabling MIG mode, creating four equal GPU instances, and preparing them for Kubernetes deployment. Mastery of these steps is critical for AI Operations professionals managing NVIDIA AI infrastructure to optimize resource utilization and workload isolation.

For further details, refer to the official NVIDIA MIG documentation: NVIDIA MIG User Guide.

More in this topic

Configure Multi-Instance GPU (MIG) — Administration (NVIDIA-Certified Professional: AI Operations)Describe datacenter architecture for AI workloads — Administration (NVIDIA-Certified Professional: AI Operations)Configure Multi-Instance GPU (MIG): Practice Questions — Administration (NVIDIA-Certified Professional: AI Operations)Administer Slurm clusters: Quick Reference — Administration (NVIDIA-Certified Professional: AI Operations)Administer Run:ai platforms — Administration (NVIDIA-Certified Professional: AI Operations)Administer Slurm clusters: Practice Questions — Administration (NVIDIA-Certified Professional: AI Operations)Configure Multi-Instance GPU (MIG): Quick Reference — Administration (NVIDIA-Certified Professional: AI Operations)Administer Slurm clusters: Common Mistakes — Administration (NVIDIA-Certified Professional: AI Operations)Administer Slurm clusters — Administration (NVIDIA-Certified Professional: AI Operations)Administration — NVIDIA-Certified Professional: AI OperationsConfigure Multi-Instance GPU (MIG): Common Mistakes — Administration (NVIDIA-Certified Professional: AI Operations)Administer Slurm clusters: Worked Example — Administration (NVIDIA-Certified Professional: AI Operations)Administer Kubernetes environments — Administration (NVIDIA-Certified Professional: AI Operations)