Quantitative and qualitative LLM metrics: Practice Questions — Evaluation (NVIDIA-Certified Professional: Generative AI LLMs)

Practice Questions: Quantitative and Qualitative LLM Metrics

These multiple-choice questions are designed to help candidates prepare for the NVIDIA-Certified Professional: Generative AI LLMs exam, focusing on the evaluation of large language models (LLMs) through quantitative and qualitative metrics.

1. Which metric is commonly used to measure the fluency and grammatical correctness of generated text by an LLM?

   • A. BLEU
   • B. Perplexity
   • C. ROUGE
   • D. F1 Score

   Correct Answer: B. Perplexity

   Explanation: Perplexity measures how well a language model predicts a sample. Lower perplexity indicates better fluency and grammatical correctness in generated text.

2. What does the BLEU score primarily evaluate in LLM outputs?

   • A. Semantic similarity
   • B. Lexical overlap with reference text
   • C. Model training speed
   • D. Error rate in token prediction

   Correct Answer: B. Lexical overlap with reference text

   Explanation: BLEU (Bilingual Evaluation Understudy) score measures the n-gram overlap between generated text and reference text, focusing on lexical similarity rather than semantic meaning.

3. Which qualitative evaluation method involves human judges rating LLM outputs for coherence, relevance, and creativity?

   • A. Automated token-level accuracy
   • B. Human annotation and scoring
   • C. Perplexity measurement
   • D. Cross-entropy loss calculation

   Correct Answer: B. Human annotation and scoring

   Explanation: Human evaluation is essential for assessing qualitative aspects such as coherence, relevance, and creativity, which automated metrics may not fully capture.

4. In the context of LLM evaluation, what does the ROUGE metric primarily assess?

   • A. Recall of overlapping n-grams between generated and reference summaries
   • B. Precision of token predictions
   • C. Model inference latency
   • D. Training data diversity

   Correct Answer: A. Recall of overlapping n-grams between generated and reference summaries

   Explanation: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures recall-based overlap of n-grams, commonly used for evaluating summarization quality.

5. Which metric is best suited for evaluating the factual accuracy of an LLM's generated content?

   • A. Perplexity
   • B. Human fact-checking
   • C. BLEU score
   • D. Cross-entropy loss

   Correct Answer: B. Human fact-checking

   Explanation: Factual accuracy is difficult to measure automatically; human fact-checking is currently the most reliable qualitative method.

6. What is a key limitation of relying solely on quantitative metrics like BLEU or ROUGE for LLM evaluation?

   • A. They require extensive human input
   • B. They do not measure semantic meaning or creativity
   • C. They are computationally expensive
   • D. They cannot be automated

   Correct Answer: B. They do not measure semantic meaning or creativity

   Explanation: Quantitative metrics focus on surface-level text similarity and often miss deeper semantic understanding and creative aspects of language generation.

7. Which of the following best describes the purpose of error analysis in LLM evaluation?

   • A. To optimize training speed
   • B. To identify and categorize model output mistakes for improvement
   • C. To calculate BLEU scores
   • D. To measure model size and complexity

   Correct Answer: B. To identify and categorize model output mistakes for improvement

   Explanation: Error analysis helps developers understand specific failure modes of the model, guiding targeted improvements.

More in this topic

Quantitative and qualitative LLM metrics: Common Mistakes — Evaluation (NVIDIA-Certified Professional: Generative AI LLMs)Quantitative and qualitative LLM metrics: Quick Reference — Evaluation (NVIDIA-Certified Professional: Generative AI LLMs)Quantitative and qualitative LLM metrics: Worked Example — Evaluation (NVIDIA-Certified Professional: Generative AI LLMs)Evaluation — NVIDIA-Certified Professional: Generative AI LLMsError analysis — Evaluation (NVIDIA-Certified Professional: Generative AI LLMs)Quantitative and qualitative LLM metrics — Evaluation (NVIDIA-Certified Professional: Generative AI LLMs)Benchmarking and framework design — Evaluation (NVIDIA-Certified Professional: Generative AI LLMs)