Deployment

• Easy if just calling an API
• Complicated if doing prompt or chain construction

Gathering Feedback

• Make it part of the workflow
  • Accept changes
  • Thumbs up/down
  • Long-form feedback
• Use feedback to make prompts better
  • Look for themes
  • Adjust prompts
  • Test, re-test, deploy
  • Look for opportunities to automate

Issues

• Latency
• UI issues
• Incorrect answers
• Answers are too long
• Answers not given (I’m just a language model)
• Prompt injection
• Toxicity, profanity

Measurement

• Just because your examples look better, they might not be
• Can improve on one aspect, but make another worse
• Understand what metrics work
• Old-school way (NLU)
  • Measure for overfitting
  • Domain shift
  • Drift