Install Run:ai and Slurm: Worked Example — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)

Installing Run:ai and Slurm on an NVIDIA AI Operations Cluster: A Worked Example This worked example walks through the step-by-step process of...

Installing Run:ai and Slurm on an NVIDIA AI Operations Cluster: A Worked Example

This worked example walks through the step-by-step process of installing Run:ai and Slurm on an NVIDIA AI cluster, a critical skill for the NVIDIA-Certified Professional: AI Operations certification. This installation enables efficient job scheduling and resource management, essential for optimizing AI workloads.

Scenario

You are tasked with deploying Run:ai and Slurm on a multi-node NVIDIA GPU cluster managed via Base Command Manager (BCM). The cluster runs on NVIDIA-certified hosts with Kubernetes already initialized. The goal is to enable advanced scheduling and resource allocation for AI training jobs.

Step 1: Prepare the Environment

Verify cluster readiness: Ensure Kubernetes is installed and operational on all nodes via BCM Base View.
Check network configuration: Confirm that cluster nodes, DPUs, and switches are properly configured and communicating.
Access permissions: Confirm you have administrative rights in BCM to install software and manage cluster resources.

Step 2: Install Slurm

Slurm is the open-source workload manager used for job scheduling. Follow these steps:

Download Slurm packages: Obtain the latest stable Slurm release compatible with your cluster OS.
Install dependencies: Install required libraries such as munge for authentication and other Slurm prerequisites.
Configure munge: Generate and distribute munge keys securely across all nodes to enable authentication.
Configure Slurm: Create the slurm.conf file defining cluster nodes, partitions, and scheduling policies.
Deploy Slurm daemons: Start slurmctld on the control node and slurmd on compute nodes.
Verify installation: Use sinfo and squeue commands to check cluster status.

Step 3: Install Run:ai

Run:ai provides AI workload orchestration on Kubernetes clusters. Installation involves:

Obtain Run:ai CLI: Download the Run:ai command-line interface tool from the official repository.
Authenticate with Run:ai platform: Use your credentials to log in and connect to your cluster.
Install Run:ai operator: Deploy the Run:ai Kubernetes operator using Helm charts or manifests to manage AI workloads.
Configure resource pools: Define GPU resource pools and scheduling policies within Run:ai to optimize utilization.
Integrate with Slurm: Configure Run:ai to submit jobs through Slurm, enabling hybrid scheduling.
Validate installation: Submit a test AI training job and monitor its execution via Run:ai dashboard and Slurm commands.

Step 4: Post-Installation Checks and Troubleshooting

Monitor cluster health: Use BCM Base View and Run:ai dashboards to verify node status and job queues.
Check logs: Review Slurm and Run:ai logs for errors or warnings.
Test job scheduling: Submit multiple jobs to ensure fair scheduling and resource allocation.
Network diagnostics: Confirm no connectivity issues between Kubernetes nodes, DPUs, and switches.

Worked Example Summary

Problem: Deploy Run:ai and Slurm on a 5-node NVIDIA GPU cluster with Kubernetes initialized.

Solution:

Verified Kubernetes and network setup via BCM.
Installed and configured munge authentication for Slurm.
Created slurm.conf defining nodes and partitions.
Started Slurm daemons and confirmed cluster status.
Downloaded Run:ai CLI, authenticated, and deployed the Run:ai operator.
Configured GPU resource pools and integrated Run:ai with Slurm.
Submitted test AI jobs and monitored successful execution.

This stepwise approach ensures a robust installation, enabling efficient AI workload orchestration and scheduling on NVIDIA AI infrastructure.

More in this topic

Ready to test your knowledge?

Put what you've learned into practice with a quick quiz and track your progress.

Test your knowledge →