Parallelism techniques: Practice Questions — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)

Practice Questions: Parallelism Techniques for GPU Acceleration and Optimization

This set of multiple-choice questions is designed to help candidates prepare for the NVIDIA-Certified Professional: Generative AI LLMs exam, focusing specifically on parallelism techniques used in multi-GPU and distributed training setups.

1. Which type of parallelism involves splitting a neural network's layers across multiple GPUs, where each GPU processes a different subset of layers sequentially?

   • A. Data Parallelism
   • B. Model Parallelism
   • C. Pipeline Parallelism
   • D. Tensor Parallelism

   Correct answer: C

   Explanation: Pipeline parallelism partitions the model into stages (layers or groups of layers) assigned to different GPUs, which process data sequentially through these stages, enabling efficient utilization of multiple GPUs.

2. In data parallelism, what is the primary operation performed after each GPU computes gradients on its mini-batch?

   • A. Gradients are averaged across GPUs to synchronize model updates.
   • B. Each GPU updates its model independently without communication.
   • C. Model parameters are split and assigned to different GPUs.
   • D. The batch size is reduced to fit GPU memory.

   Correct answer: A

   Explanation: Data parallelism requires synchronizing gradients by averaging them across GPUs to ensure consistent model updates during training.

3. Which parallelism technique is best suited for very large models that cannot fit into a single GPU’s memory by splitting tensor operations across GPUs?

   • A. Data Parallelism
   • B. Pipeline Parallelism
   • C. Tensor Parallelism
   • D. Model Parallelism

   Correct answer: C

   Explanation: Tensor parallelism divides individual tensor operations (e.g., matrix multiplications) across multiple GPUs, enabling training of very large models that exceed single GPU memory limits.

4. What is a common challenge when implementing pipeline parallelism in distributed training?

   • A. Increased communication overhead due to gradient synchronization
   • B. Load imbalance causing some GPUs to be idle while others are busy
   • C. Difficulty in splitting mini-batches across GPUs
   • D. Reduced batch size per GPU

   Correct answer: B

   Explanation: Pipeline parallelism can suffer from load imbalance and pipeline bubbles, where some GPUs wait idle for data to arrive, reducing overall efficiency.

5. Which of the following is NOT a benefit of combining data parallelism with model or tensor parallelism?

   • A. Improved scalability across many GPUs
   • B. Reduced memory footprint per GPU
   • C. Elimination of all communication overhead
   • D. Ability to train larger models efficiently

   Correct answer: C

   Explanation: While combining parallelism techniques improves scalability and memory usage, communication overhead cannot be completely eliminated; it is managed and optimized.

6. In the context of parallelism, what does 'micro-batching' refer to?

   • A. Splitting a mini-batch into smaller chunks to improve pipeline utilization
   • B. Reducing the overall batch size to fit GPU memory
   • C. Assigning each GPU a different mini-batch
   • D. Combining multiple batches into a larger batch for better throughput

   Correct answer: A

   Explanation: Micro-batching divides a mini-batch into smaller micro-batches to keep all pipeline stages busy and reduce idle time in pipeline parallelism.

7. Which profiling tool is commonly used to identify bottlenecks in GPU parallelism and optimize performance?

   • A. NVIDIA Nsight Systems
   • B. TensorBoard
   • C. PyTorch Profiler
   • D. All of the above

   Correct answer: D

   Explanation: All these tools can be used to profile GPU workloads, identify bottlenecks, and optimize parallelism strategies in training large language models.

8. What is the main advantage of using distributed data parallel (DDP) training over single-GPU training?

   • A. It reduces the total training time by parallelizing workload across GPUs.
   • B. It eliminates the need for gradient synchronization.
   • C. It allows training without any communication between GPUs.
   • D. It decreases model size.

   Correct answer: A

   Explanation: DDP accelerates training by distributing mini-batches across GPUs and synchronizing gradients, thus reducing overall training time.

More in this topic

Memory and batch optimization — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)Parallelism techniques: Common Mistakes — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)Parallelism techniques: Quick Reference — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)Multi-GPU and distributed setups — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)Performance profiling and troubleshooting — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)GPU Acceleration and Optimization — NVIDIA-Certified Professional: Generative AI LLMsParallelism techniques: Worked Example — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)Parallelism techniques — GPU Acceleration and Optimization (NVIDIA-Certified Professional: Generative AI LLMs)