Pretraining

Just like people, LLMs don’t start out knowing everything - even though ChatGPT seems like it does. Nope. Even ChatGPT had to learn.

The first phase of learning is pretraining.

Think of it like going to school.

In school, you learn a number of subjects. You read a bunch of books. You practice math, history, science, and more. The goal is to get a well rounded education.

By getting this well rounded education, you learn how to learn. You pick up patterns and skills that help you learn ever more.

ChatGPT was trained on internet archives, thousands of books and other content. This helped the models learn how to understand how language works.

Just like us, the next step for the model is to move beyond general education. It’s time to learn how to specialize.

That’s where fine-tuning comes in.