Pretraining is the initial training phase in which a model learns general patterns from a large dataset before later adaptation or fine-tuning for specific tasks.
Pretraining is common in foundation models and language models because it creates reusable representations that can support many downstream applications.