Base Command Manager installation, configuration, and HA verification: Worked Example — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)

Base Command Manager Installation, Configuration, and HA Verification: Worked Example

The Command Manager is a critical component in NVIDIA's AI infrastructure, providing centralized control and orchestration for GPU clusters. This worked example guides you through the step-by-step process of installing, configuring, and verifying High Availability (HA) for the Command Manager in a realistic enterprise AI deployment scenario.

Scenario Overview

Your organization is deploying an AI compute cluster using NVIDIA's AI infrastructure stack. The goal is to install the Command Manager on two nodes to ensure HA, configure it for cluster management, and verify that the HA setup is functioning correctly.

Step 1: Prepare the Environment

• Ensure both nodes (Node A and Node B) have compatible OS versions and network connectivity.
• Verify that required dependencies such as Docker and NVIDIA GPU drivers are installed and operational.
• Set up SSH key-based authentication between the nodes for seamless communication.

Step 2: Install Base Command Manager on Node A

1. Download the Command Manager installer: Obtain the latest Command Manager installation package from NVIDIA's official repository or internal distribution.

2. Run the installer: Execute the installation script with administrative privileges.

   Example Command

   sudo ./command-manager-install.sh --node=nodeA --role=primary

3. Verify installation: Confirm that the Command Manager service is running.

   • systemctl status command-manager

Step 3: Configure Command Manager on Node A

• Edit the configuration file /etc/command-manager/config.yaml to define cluster parameters such as cluster name, node roles, and network settings.
• Set the primary node role and specify the secondary node IP for HA failover.
• Apply configuration changes by restarting the Command Manager service.

Step 4: Install Command Manager on Node B for HA

1. Repeat the installation process on Node B, this time specifying the secondary role.
2. Example command:

sudo ./command-manager-install.sh --node=nodeB --role=secondary

1. Ensure Node B is configured to monitor the health of Node A and be ready to take over if the primary fails.

Step 5: Configure HA Verification

• Use built-in Command Manager HA tools or scripts to simulate failover scenarios.
• Check logs on both nodes to confirm heartbeat signals and failover events.
• Example command to check HA status:

command-manager-ha-status --cluster=myCluster

Step 6: Validate HA Functionality

1. Manually stop the Command Manager service on Node A:

sudo systemctl stop command-manager

1. Observe Node B taking over as the primary node automatically.
2. Verify cluster operations continue without interruption.
3. Restart the service on Node A and confirm it resumes the primary role or becomes secondary as per configuration.

Summary

This worked example demonstrated the detailed process of installing the base Command Manager on two nodes, configuring it for HA, and verifying failover functionality. Mastery of these steps is essential for deploying resilient NVIDIA AI infrastructure clusters that maintain continuous operation even during node failures.

For further details and official documentation, refer to NVIDIA's AI Infrastructure resources at NVIDIA AI Infrastructure.

More in this topic

Base Command Manager installation, configuration, and HA verification: Quick Reference — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)NVIDIA container toolkit installation and GPU use with Docker — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)Base Command Manager installation, configuration, and HA verification: Practice Questions — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)NGC CLI installation — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)Control Plane Installation and Configuration — NVIDIA-Certified Professional: AI InfrastructureBase Command Manager installation, configuration, and HA verification: Common Mistakes — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)Base Command Manager installation, configuration, and HA verification — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)GPU and DOCA driver installation — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)Cluster installation with Slurm, Enroot, and Pyxis — Control Plane Installation and Configuration (NVIDIA-Certified Professional: AI Infrastructure)