Cluster installation with Slurm, Enroot, and Pyxis: Worked Example — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)

Cluster Installation with Slurm, Enroot, and Pyxis: Worked Example This worked example demonstrates the step-by-step process of installing and...

Cluster Installation with Slurm, Enroot, and Pyxis: Worked Example

This worked example demonstrates the step-by-step process of installing and configuring a GPU-accelerated AI compute cluster using Slurm as the workload manager, Enroot for container runtime, and Pyxis for GPU resource integration. These components are essential for efficient job scheduling and containerized AI workloads on NVIDIA AI infrastructure.

Scenario

You are tasked with setting up a small AI cluster of three nodes, each equipped with NVIDIA GPUs. The goal is to enable users to submit GPU-accelerated containerized jobs managed by Slurm, leveraging Enroot as the container runtime and Pyxis to expose GPUs inside containers.

Step 1: Prepare the Nodes

Ensure all nodes run a supported Linux distribution (e.g., Ubuntu 20.04 or CentOS 8).
Install the required NVIDIA GPU drivers and DOCA drivers to support GPU and networking capabilities.
Verify GPU visibility using nvidia-smi on each node.

Step 2: Install Slurm

On all nodes, install Slurm components: slurmctld on the controller node and slurmd on compute nodes.
Configure slurm.conf to define the cluster nodes, partitions, and GPU resources. For example, specify GresTypes=gpu and define GPUs per node.
Start and enable Slurm services, verifying communication between controller and compute nodes.

Step 3: Install Enroot

Download and install Enroot from the official NVIDIA repository on all nodes.
Verify installation by running a test container, e.g., enroot start ubuntu.
Configure Enroot to allow GPU access by enabling NVIDIA container toolkit integration.

Step 4: Install Pyxis

Install the Pyxis Slurm plugin on all nodes to enable GPU resource management inside Enroot containers.
Modify Slurm configuration to include Pyxis plugin by adding PrologFlags=Contain and TaskPlugin=task/affinity,task/pyxis.
Restart Slurm services to apply changes.

Step 5: Test GPU Access in Containerized Jobs

Create a Slurm job script that launches an Enroot container with GPU support via Pyxis.
Example job script snippet:

Example Slurm Job Script

SBATCH --gres=gpu:1

SBATCH --job-name=test_gpu

srun --container-image=nvcr.io/nvidia/cuda:11.4-base \ --container-mounts=/data:/data \ nvidia-smi

This script requests one GPU and runs nvidia-smi inside the container to verify GPU visibility.

Step 6: Verify High Availability and Cluster Functionality

Submit multiple jobs to Slurm and monitor scheduling and GPU allocation.
Check logs for any errors related to Enroot or Pyxis integration.
Confirm that GPU metrics are correctly reported inside containers.

Summary

This example outlines the concrete steps to install and configure a GPU-enabled AI cluster using Slurm, Enroot, and Pyxis. Mastery of these steps is critical for the Control Plane Installation and Configuration section of the NVIDIA-Certified Professional: AI Infrastructure exam, representing 19% of the assessment.

For detailed official guidance, refer to NVIDIA's documentation on Slurm with Enroot and Pyxis.

More in this topic

Ready to test your knowledge?

Put what you've learned into practice with a quick quiz and track your progress.

Test your knowledge →