Manage job scheduling with Slurm or Kubernetes: Worked Example — Installation and Deployment (NVIDIA-Certified Professional: AI Operations)
Managing Job Scheduling with Slurm and Kubernetes: A Worked Example Effective job scheduling is critical for optimising resource utilisation and...
Managing Job Scheduling with Slurm and Kubernetes: A Worked Example
Effective job scheduling is critical for optimising resource utilisation and workload management in NVIDIA AI infrastructure. This worked example demonstrates how to manage job scheduling using Slurm and Kubernetes within the context of the NVIDIA-Certified Professional: AI Operations certification.
Scenario
An AI research team needs to schedule multiple GPU-accelerated training jobs on a shared cluster. The cluster uses Slurm for batch job scheduling and Kubernetes for container orchestration. The goal is to submit, monitor, and manage jobs efficiently, ensuring fair resource allocation and minimal wait times.
Step 1: Preparing the Environment
Ensure the cluster nodes have Slurm installed and configured with GPU resource tracking.
Verify Kubernetes is installed on NVIDIA hosts using Base Command Manager (BCM) and that the cluster is operational.
Confirm that the necessary NVIDIA drivers, CUDA, and DOCA services are deployed on DPUs.
Step 2: Submitting a Job with Slurm
Slurm uses job scripts to define resource requirements and execution commands.
restartPolicy: Never ensures the job does not restart automatically on failure.
backoffLimit: 4 allows up to 4 retries.
Deploy the job: kubectl apply -f job.yaml
Monitor job status: Use kubectl get jobs and kubectl logs JOB_POD_NAME.
Step 4: Handling Job Scheduling Conflicts and Priorities
Both Slurm and Kubernetes support priority and fair-share scheduling:
In Slurm, configure PriorityWeight and FairShare policies in slurm.conf.
In Kubernetes, use PriorityClass objects and ResourceQuota to manage resource allocation.
Step 5: Troubleshooting Common Issues
Job stuck in queue: Check node availability and GPU health with sinfo and nvidia-smi.
Kubernetes pod pending: Inspect resource requests vs. cluster capacity and node taints with kubectl describe pod POD_NAME.
Permission errors: Verify user roles and permissions in BCM for job submission.
Summary
This example illustrates the practical steps to manage job scheduling on NVIDIA AI clusters using Slurm and Kubernetes. Understanding these processes is essential for optimising workload throughput and resource utilisation, key competencies for the NVIDIA-Certified Professional: AI Operations certification.