Deploy containers from NGC: Worked Example — Workload Management (NVIDIA-Certified Professional: AI Operations)
{ "title": "NVIDIA-Certified Professional: AI Operations - Deploy Containers from NGC: Worked Example", "category": "NVIDIA-Certified Professional...
{ "title": "NVIDIA-Certified Professional: AI Operations - Deploy Containers from NGC: Worked Example", "category": "NVIDIA-Certified Professional: AI Operations", "hashtags": "NVIDIA, AIOperations, NGC, container-deployment, workload-management", "content": "
Deploy Containers from NGC: A Worked Example
In the NVIDIA-Certified Professional: AI Operations exam, understanding how to deploy containers from the NVIDIA GPU Cloud (NGC) is essential for efficient workload management. This worked example walks through a realistic scenario to demonstrate the step-by-step process of deploying an AI inference container from NGC on a Kubernetes cluster.
Scenario
Your AI operations team needs to deploy an optimized TensorRT inference container from NGC to serve a computer vision model on your Kubernetes cluster. The goal is to ensure GPU acceleration is leveraged, and the deployment is scalable and manageable.
Step 1: Prepare the Kubernetes Cluster
Ensure your Kubernetes cluster has the NVIDIA device plugin installed to expose GPUs to pods. This is critical for GPU scheduling.
- Run the following command to deploy the NVIDIA device plugin:
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.11.0/nvidia-device-plugin.yml
This enables Kubernetes to recognize and schedule GPU resources.
Step 2: Authenticate to NGC
To pull containers from NGC, authenticate using your NGC API key.
- Create a Kubernetes secret with your NGC API key:
kubectl create secret docker-registry ngc-secret \ --docker-server=nvcr.io \ --docker-username='$oauthtoken' \ --docker-password='YOUR_NGC_API_KEY' \ --docker-email='user@example.com'
This secret allows Kubernetes to pull private containers from NGC.
Step 3: Define the Deployment YAML
Create a deployment manifest that specifies the NGC container image, GPU resource requests, and the image pull secret.
apiVersion: apps/v1 kind: Deployment metadata: name: tensorrt-inference spec: replicas: 2 selector: matchLabels: app: tensorrt-inference template: metadata: labels: app: tensorrt-inference spec: containers: - name: tensorrt-container image: nvcr.io/nvidia/tensorrt:22.09-py3 resources: limits: nvidia.com/gpu: 1 ports: - containerPort: 8000 imagePullSecrets: - name: ngc-secretThis manifest deploys two replicas of the TensorRT inference container, each requesting one GPU.
Step 4: Deploy the Container
Apply the deployment manifest to your cluster:
kubectl apply -f tensorrt-deployment.yaml
Verify the pods are running and have GPU assigned:
kubectl get pods -l app=tensorrt-inference
Use kubectl describe pod [pod-name] to confirm GPU resource allocation.
Step 5: Expose the Service
Create a service to expose the inference pods:
apiVersion: v1 kind: Service metadata: name: tensorrt-service spec: selector: app: tensorrt-inference ports: - protocol: TCP port: 80 targetPort: 8000 type: LoadBalancer
Apply the service manifest:
kubectl apply -f tensorrt-service.yaml
This exposes the inference service externally for client requests.
Step 6: Monitor and Troubleshoot
Use kubectl logs to check container logs and kubectl describe pod to troubleshoot resource issues. System tools like nvidia-smi inside the container can verify GPU utilization.
Summary
This example demonstrated deploying an NGC container for AI inference on Kubernetes by:
- Installing the NVIDIA device plugin
- Creating an image pull secret for NGC authentication
- Defining a deployment with GPU resource requests
- Deploying and exposing the containerized service
- Monitoring the deployment for troubleshooting
Mastering these steps is critical for effective workload management in NVIDIA AI Operations environments.
" }
More in this topic
Ready to test your knowledge?
Put what you've learned into practice with a quick quiz and track your progress.
Test your knowledge →