Notes

Methods of Evaluation

Real User Feedback

• Gold standard
• How do real people using your LLM find the output?
• Tactics
  • Explicit feedback - Thumbs up/down, stars, etc.
  • Implicit feedback - Observe user behavior

Human Evaluation

• Have human evaluators examine output
• Build a test dataset

Reference

• Compare against ground truth
• Verify ground truth, could be misleading otherwise

Scoring

• Give a score (0 to 10 for example)
• No ground truth reference
• Specialized tasks can check for specific output/rules

A/B Testing

• Challenging to scale humans
• Humans evaluation is subjective