This paper was accepted to the ACL 2025 foremost convention as an oral presentation.
This paper was accepted on the Scalable Continuous Studying for Lifelong Basis Fashions (SCLLFM) Workshop at NeurIPS 2024.
Giant Language Fashions (LLMs) educated on historic internet knowledge inevitably turn into outdated. We examine analysis methods and replace strategies for LLMs as new knowledge turns into out there. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Frequent Crawl (CC) – orders of magnitude bigger than earlier continuous language modeling benchmarks. We additionally design time-stratified evaluations throughout each normal CC knowledge and particular domains (Wikipedia, StackExchange, and code documentation) to evaluate how nicely varied continuous studying strategies adapt to new knowledge whereas retaining previous information. Our findings exhibit that, on normal CC knowledge, autoregressive meta-schedules mixed with a fixed-ratio replay of older knowledge can obtain comparable held-out loss to re-training from scratch, whereas requiring considerably much less computation (2.6x). Nonetheless, the optimum stability between incorporating new knowledge and replaying outdated knowledge differs as replay is essential to keep away from forgetting on generic internet knowledge however much less so on particular domains.
- * Work performed throughout an internship at Apple
- ° Work performed whereas at Apple
- † Equal contribution
- ‡ Challenge lead
- § College of Washington
