Pre-training
The initial large-scale training phase of a language model in which it learns to predict the next token (or masked tokens) over a massive text corpus, establishing general language understanding before any task-specific fine-tuning.
Pre-training is the most computationally expensive phase of LLM development, consuming tens of thousands of GPU-hours and petabytes of training data. The objective is self-supervised: the model predicts the next token (causal language modeling) or reconstructs masked tokens (masked language modeling, used in BERT). No human-labeled data is required; the supervision signal comes from the data itself.
The resulting pre-trained model has broad language capabilities—it can complete sentences, continue code, summarize text—but is not aligned to follow instructions or behave safely. This is the "base model" stage. GPT-4 base, Claude base (before RLHF), and Llama 3 base are examples of pre-trained-only models, generally not deployed directly to users.
Pre-training quality depends on dataset curation (deduplication, quality filtering, domain mix), compute budget, and architecture. Chinchilla scaling laws (Hoffmann et al., 2022) established that training longer on more data with a smaller model often outperforms training shorter with a larger model for the same compute budget, reshaping how frontier labs allocate their training runs.
## Related Terms
- [[glossary/instruction-tuning|Instruction Tuning]] — the fine-tuning stage that teaches a model to follow natural-language instructions - [[glossary/fine-tuning|Fine-tuning]] — the weight-update process that adapts a pre-trained base model for downstream tasks - [[glossary/scaling-laws|Scaling Laws]] — the empirical relationships between model size, compute, data, and performance - [[glossary/transformer|Transformer]] — the neural architecture underlying virtually all modern LLMs - [[courses/gemini-enterprise-agents|Course: Gemini Enterprise Agents]] — hands-on practice with the concepts covered in this entry