Single- and multi-GPU training: Practice Questions — Machine Learning (NVIDIA-Certified Professional: Accelerated Data Science)
Practice Questions: Single- and Multi-GPU Training for NVIDIA-Certified Professional: Accelerated Data Science This set of multiple-choice questions...
Practice Questions: Single- and Multi-GPU Training for NVIDIA-Certified Professional: Accelerated Data Science
This set of multiple-choice questions is designed to help candidates prepare for the Single- and Multi-GPU Training portion of the Machine Learning topic (15% of the exam) in the NVIDIA-Certified Professional: Accelerated Data Science certification. Each question includes four options, the correct answer, and a brief explanation.
What is the primary advantage of using multi-GPU training over single-GPU training?
- A) Reduces the need for feature engineering
- B) Enables larger batch sizes and faster training times
- C) Eliminates the need for hyperparameter tuning
- D) Guarantees higher model accuracy without additional effort
Correct Answer: B
Explanation: Multi-GPU training allows the distribution of data and computations across multiple GPUs, enabling larger batch sizes and significantly reducing training time. It does not eliminate the need for feature engineering or hyperparameter tuning, nor does it guarantee higher accuracy automatically.
Which technique is commonly used to synchronize model updates across GPUs during distributed training?
- A) Gradient accumulation
- B) Batch normalization
- C) All-reduce communication
- D) Dropout regularization
Correct Answer: C
Explanation: All-reduce is a communication operation used to aggregate gradients from multiple GPUs to ensure consistent model updates during distributed training. Gradient accumulation is related but distinct, batch normalization and dropout are regularization techniques.
In single-GPU training, which method helps to maximize GPU memory utilization without increasing memory capacity?
- A) Using mixed precision training
- B) Increasing the learning rate
- C) Reducing batch size
- D) Disabling data augmentation
Correct Answer: A
Explanation: Mixed precision training uses lower-precision (e.g., FP16) arithmetic where possible, reducing memory usage and speeding up computation, thus maximizing GPU memory utilization. Increasing learning rate or disabling data augmentation does not affect memory usage directly; reducing batch size decreases memory usage but does not maximize utilization.
Which of the following is a common challenge when scaling training from a single GPU to multiple GPUs?
- A) Overfitting due to larger datasets
- B) Communication overhead between GPUs
- C) Loss of model interpretability
- D) Inability to use mixed precision
Correct Answer: B
Explanation: Communication overhead during synchronization of gradients and parameters between GPUs can limit scalability and reduce training efficiency. Overfitting and interpretability are unrelated to GPU scaling, and mixed precision is supported on multi-GPU setups.
During multi-GPU training, which strategy can help maintain model convergence when increasing batch size?
- A) Decreasing the number of epochs
- B) Using learning rate warm-up
- C) Eliminating dropout layers
- D) Reducing dataset size
Correct Answer: B
Explanation: Learning rate warm-up gradually increases the learning rate at the start of training to help stabilize convergence when using large batch sizes common in multi-GPU training. Decreasing epochs or reducing dataset size can harm model performance, and eliminating dropout is unrelated to batch size scaling.
What is the role of gradient accumulation in single-GPU training?
- A) To split the model across multiple GPUs
- B) To simulate larger batch sizes by accumulating gradients over multiple smaller batches
- C) To reduce GPU memory usage by lowering precision
- D) To synchronize gradients across GPUs
Correct Answer: B
Explanation: Gradient accumulation allows training with effectively larger batch sizes than the GPU memory would normally permit by accumulating gradients over several smaller mini-batches before performing a weight update. It does not split models or synchronize gradients across GPUs.
Which NVIDIA tool or library is specifically designed to optimize multi-GPU training performance?
- A) cuDNN
- B) NCCL (NVIDIA Collective Communications Library)
- C) TensorRT
- D) CUDA Graphs
Correct Answer: B
Explanation: NCCL provides highly optimized communication primitives for multi-GPU and multi-node training, enabling efficient collective operations like all-reduce. cuDNN accelerates deep learning primitives, TensorRT focuses on inference optimization, and CUDA Graphs optimize kernel launches.
Which of the following best describes the impact of batch size on GPU memory usage during training?
- A) Larger batch sizes decrease GPU memory usage
- B) Batch size does not affect GPU memory usage
- C) Larger batch sizes increase GPU memory usage
- D) Smaller batch sizes increase GPU memory usage
Correct Answer: C
Explanation: Increasing batch size increases the number of samples processed simultaneously, which requires more GPU memory to store inputs, activations, and gradients during training.