Notes

Pre-training - Predicts the output of a sentence
Uses Stochastic Gradient Descent
NN needs to learn about objects and how they interact with each other
This is a form of compression
LM Fine-Tuning - Giving data to a model (Common Crawl, Wikipedia, etc.). These are documents that should be close to what we want the model to do.
Classifier Fine-Tuning - End task we want the model to do.
Instruction Tuning - Give the model examples for how to respond.
- Prompt | Question | Answer
- OpenOrca

Links