Upload README.md with huggingface_hub

40ac7a3 verified 16 days ago

34.5 kB

	---
	license: apache-2.0
	datasets:
	- allenai/datadecide
	language:
	- en
	---
	![image/png](https://cdn-uploads.huggingface.co/production/uploads/62bddd0b1e22ec8427a0f27e/MwddQs_8OaU4128VYrwoU.png)

	More than one training run goes into making a large language model, but developers rarely release the small models and datasets they experiment with during the development process. How do they decide what dataset to use for pretraining or which benchmarks to hill climb on? To empower open exploration of these questions, we release [DataDecide](allenai.org/paper/datadecide)—a suite of models we pretrain on 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, over 14 different model sizes ranging from 4M parameters up to 1B parameters (more than 30k model checkpoints in total).

	## 350 Models over Differences in Data in Scale
	For each of our 25 datasets and 14 model sizes, we train a model linked below. Each has intermediate checkpoints (uploading after initial release), runs over 3 random seeds. All models finish training at a token to parameter ratio of 100 (e.g., 1B parameters -> 100B tokens).
	\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|
	\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|------\|------\|------\|------\|-----\|
	\| Dolma1.7 \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-1B) \|
	\| Dolma1.7 (no code) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-1B) \|
	\| Dolma1.7 (no math, code) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-1B) \|
	\| Dolma1.7 (no Reddit) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-1B) \|
	\| Dolma1.7 (no Flan) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-1B) \|
	\| Dolma1.6++ \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_6plus-1B) \|
	\| C4 \| [4M](https://huggingface.co/allenai/DataDecide-c4-4M) \| [6M](https://huggingface.co/allenai/DataDecide-c4-6M) \| [8M](https://huggingface.co/allenai/DataDecide-c4-8M) \| [10M](https://huggingface.co/allenai/DataDecide-c4-10M) \| [14M](https://huggingface.co/allenai/DataDecide-c4-14M) \| [16M](https://huggingface.co/allenai/DataDecide-c4-16M) \| [20M](https://huggingface.co/allenai/DataDecide-c4-20M) \| [60M](https://huggingface.co/allenai/DataDecide-c4-60M) \| [90M](https://huggingface.co/allenai/DataDecide-c4-90M) \| [150M](https://huggingface.co/allenai/DataDecide-c4-150M) \| [300M](https://huggingface.co/allenai/DataDecide-c4-300M) \| [530M](https://huggingface.co/allenai/DataDecide-c4-530M) \| [750M](https://huggingface.co/allenai/DataDecide-c4-750M) \| [1B](https://huggingface.co/allenai/DataDecide-c4-1B) \|
	\| FineWeb-Pro \| [4M](https://huggingface.co/allenai/DataDecide-fineweb-pro-4M) \| [6M](https://huggingface.co/allenai/DataDecide-fineweb-pro-6M) \| [8M](https://huggingface.co/allenai/DataDecide-fineweb-pro-8M) \| [10M](https://huggingface.co/allenai/DataDecide-fineweb-pro-10M) \| [14M](https://huggingface.co/allenai/DataDecide-fineweb-pro-14M) \| [16M](https://huggingface.co/allenai/DataDecide-fineweb-pro-16M) \| [20M](https://huggingface.co/allenai/DataDecide-fineweb-pro-20M) \| [60M](https://huggingface.co/allenai/DataDecide-fineweb-pro-60M) \| [90M](https://huggingface.co/allenai/DataDecide-fineweb-pro-90M) \| [150M](https://huggingface.co/allenai/DataDecide-fineweb-pro-150M) \| [300M](https://huggingface.co/allenai/DataDecide-fineweb-pro-300M) \| [530M](https://huggingface.co/allenai/DataDecide-fineweb-pro-530M) \| [750M](https://huggingface.co/allenai/DataDecide-fineweb-pro-750M) \| [1B](https://huggingface.co/allenai/DataDecide-fineweb-pro-1B) \|
	\| FineWeb-Edu \| [4M](https://huggingface.co/allenai/DataDecide-fineweb-edu-4M) \| [6M](https://huggingface.co/allenai/DataDecide-fineweb-edu-6M) \| [8M](https://huggingface.co/allenai/DataDecide-fineweb-edu-8M) \| [10M](https://huggingface.co/allenai/DataDecide-fineweb-edu-10M) \| [14M](https://huggingface.co/allenai/DataDecide-fineweb-edu-14M) \| [16M](https://huggingface.co/allenai/DataDecide-fineweb-edu-16M) \| [20M](https://huggingface.co/allenai/DataDecide-fineweb-edu-20M) \| [60M](https://huggingface.co/allenai/DataDecide-fineweb-edu-60M) \| [90M](https://huggingface.co/allenai/DataDecide-fineweb-edu-90M) \| [150M](https://huggingface.co/allenai/DataDecide-fineweb-edu-150M) \| [300M](https://huggingface.co/allenai/DataDecide-fineweb-edu-300M) \| [530M](https://huggingface.co/allenai/DataDecide-fineweb-edu-530M) \| [750M](https://huggingface.co/allenai/DataDecide-fineweb-edu-750M) \| [1B](https://huggingface.co/allenai/DataDecide-fineweb-edu-1B) \|
	\| Falcon \| [4M](https://huggingface.co/allenai/DataDecide-falcon-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-1B) \|
	\| Falcon+CC \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-1B) \|
	\| Falcon+CC (QC 10%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-1B) \|
	\| Falcon+CC (QC 20%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-1B) \|
	\| Falcon+CC (QC Orig 10%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-1B) \|
	\| Falcon+CC (QC Tulu 10%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-1B) \|
	\| DCLM-Baseline \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-1B) \|
	\| DCLM-Baseline (QC 7%, FW2) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-1B) \|
	\| DCLM-Baseline (QC 7%, FW3) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-1B) \|
	\| DCLM-Baseline (QC FW 3%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-1B) \|
	\| DCLM-Baseline (QC FW 10%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-1B) \|
	\| DCLM-Baseline (QC 10%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-1B) \|
	\| DCLM-Baseline (QC 20%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-1B) \|
	\| DCLM-Baseline 25% / Dolma 75% \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-1B) \|
	\| DCLM-Baseline 50% / Dolma 50% \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-1B) \|
	\| DCLM-Baseline 75% / Dolma 25% \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-1B) \|

	## Load a Model

	To load a specific model with HuggingFace:

	```
	from hf_olmo import OLMoForCausalLM # pip install ai2-olmo

	olmo = OLMoForCausalLM.from_pretrained("allenai/DataDecide-dolma1_7-1B", revision="step69369-seed-default")
	```

	### Model Description

	<!-- Provide a longer summary of what this model is. -->



	- Developed by: Allen Institute for AI (Ai2)
	- Model type: a Transformer style autoregressive language model.
	- Language(s) (NLP): English
	- License: The code and model are released under Apache 2.0.
	- Contact: Technical inquiries: `ianmag@cs.washington.edu`. Press: `press@allenai.org`

	### Model Sources

	<!-- Provide the basic links for the model. -->

	- Repository: [https://github.com/allenai/DataDecide](https://github.com/allenai/DataDecide)
	- Paper: [https:/allenai.org/paper/datadecide](https:/allenai.org/paper/datadecide)
	- Data: [https://huggingface.co/datasets/allenai/datadecide](https://huggingface.co/datasets/allenai/datadecide)

	## Data

	\| Source / Recipe \| Description \|
	\|----------------------------------------\|-------------\|
	\| Dolma1.7 Original, No code, No math/code, No Reddit, No Flan \| A 2.3T-token corpus (Dolma; 1.7 [Soldaini et al., 2024](https://arxiv.org/abs/2402.00159)) sampling common LM sources for open research. We ablate code, math/code, Reddit, or Flan subsets. \|
	\| Dolma1.6++ Original \| Dolma 1.6 plus additional sources from Dolma 1.7: RedPajama’s arxiv subset, openwebmath, algebraic stack, flan, starcoder, falcon. \|
	\| C4 Original \| The C4 dataset ([Raffel et al., 2019](https://arxiv.org/abs/1910.10683)) as prepared in Dolma 1.7, heuristically filtered from the April 2019 Common Crawl. \|
	\| FineWeb-Pro Original \| The FineWeb Pro corpus ([Zhou et al., 2024](https://arxiv.org/abs/2409.17115)), featuring model-driven data cleaning on FineWeb. \|
	\| FineWeb-Edu Original \| The deduplicated FineWeb-Edu subset of SmoLLM-Corpus ([Ben Allal et al., 2024](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)), focused on educational web pages. \|
	\| Falcon Original \| The Falcon RefinedWeb corpus ([Penedo et al., 2023](https://api.semanticscholar.org/CorpusID:259063761)) in Dolma 1.7, derived from Common Crawl through June 2023 and more aggressively filtered/deduplicated than C4. \|
	\| Falcon+CC Original, QC 10%, QC 20%, QC Orig 10%, QC Tulu 10% \| Falcon and Dolma 1.7’s Common Crawl. We quality filter to top 10% or 20% documents with reproduced or original [Li et al. (2024)](https://arxiv.org/abs/2406.11794) filter or retrain filter on pre-release version of Tulu-v3 ([Lambert et al., 2024](https://arxiv.org/abs/2411.15124)). \|
	\| DCLM-Baseline Original, QC 7% FW2, QC 7% FW3, QC FW 10%, QC 10%, QC 20% \| A SOTA Common Crawl corpus using best ablated deduplication, cleaning heuristics, and quality filter. We quality filter to top 7% of DCLM classified documents and further take 2+ or 3+ scores with FineWeb-edu classifier; or filter to top 3% or 10% with FineWeb-edu classifier; or take top 10% or 20% with reproduced DCLM classifier. \|
	\| λ% DCLM-Baseline + 1 – λ% Dolma1.7 \| Fractional combinations of Dolma1.7 and DCLM-Baseline mixing different proportions of the two datasets for λ ∈ {25%, 50%, 75%}. \|


	## Evaluation

	We evaluate all checkpoints over OLMES suite of 10 multiple choice question answering benchmarks
	([Gu et al., 2024](https://arxiv.org/abs/2406.08446)):

	- [MMLU (Hendrycks et al., 2021)](https://arxiv.org/abs/2009.03300)
	- [HellaSwag (Zellers et al., 2019)](https://arxiv.org/abs/1905.07830)
	- [ARC-Challenge (Clark et al., 2018)](https://arxiv.org/abs/1803.05457)
	- [ARC-Easy (Clark et al., 2018)](https://arxiv.org/abs/1803.05457)
	- [PIQA (Bisk et al., 2020)](https://arxiv.org/abs/1911.11641)
	- [CommonsenseQA (Talmor et al., 2019)](https://arxiv.org/abs/1811.00937)
	- [Social IQa (Sap et al., 2019)](https://arxiv.org/abs/1904.09728)
	- [OpenBookQA (Mihaylov et al., 2018)](https://arxiv.org/abs/1809.02789)
	- [BoolQ (Clark et al., 2019)](https://arxiv.org/abs/1905.10044)
	- [Winogrande (Sakaguchi et al., 2020)](https://arxiv.org/abs/1907.10641)

	We release all these evaluations:
	- for task-level metric results: [https://huggingface.co/datasets/allenai/DataDecide-eval-results](https://huggingface.co/datasets/allenai/DataDecide-eval-results)
	- for instance-level results: [https://huggingface.co/datasets/allenai/DataDecide-eval-instances](https://huggingface.co/datasets/allenai/DataDecide-eval-instances)


	## Hyperparameters

	\| Name \| Batch Size \| Hidden Dim. \| LR \| Model size \| Heads \| Layers \| Training steps \| Tokens trained \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| 4M \| 32 \| 64 \| 1.4e-02 \| 3.7M \| 8 \| 8 \| 5,725 \| 0.4B \|
	\| 6M \| 32 \| 96 \| 1.2e-02 \| 6.0M \| 8 \| 8 \| 9,182 \| 0.6B \|
	\| 8M \| 32 \| 128 \| 1.1e-02 \| 8.5M \| 8 \| 8 \| 13,039 \| 0.9B \|
	\| 10M \| 32 \| 144 \| 1.0e-02 \| 9.9M \| 8 \| 8 \| 15,117 \| 1.0B \|
	\| 14M \| 32 \| 192 \| 9.2e-03 \| 14.4M \| 8 \| 8 \| 21,953 \| 1.4B \|
	\| 16M \| 32 \| 208 \| 8.9e-03 \| 16.0M \| 8 \| 8 \| 24,432 \| 1.6B \|
	\| 20M \| 64 \| 192 \| 8.4e-03 \| 19.1M \| 8 \| 16 \| 14,584 \| 1.9B \|
	\| 60M \| 96 \| 384 \| 5.8e-03 \| 57.1M \| 12 \| 16 \| 29,042 \| 5.7B \|
	\| 90M \| 160 \| 528 \| 4.9e-03 \| 97.9M \| 12 \| 16 \| 29,901 \| 9.8B \|
	\| 150M \| 192 \| 768 \| 4.2e-03 \| 151.9M \| 12 \| 12 \| 38,157 \| 15.0B \|
	\| 300M \| 320 \| 1,024 \| 3.3e-03 \| 320.0M \| 16 \| 16 \| 45,787 \| 30.0B \|
	\| 530M \| 448 \| 1,344 \| 2.8e-03 \| 530.1M \| 16 \| 16 \| 57,786 \| 53.0B \|
	\| 750M \| 576 \| 1,536 \| 2.5e-03 \| 681.3M \| 16 \| 16 \| 63,589 \| 75.0B \|
	\| 1B \| 704 \| 2,048 \| 2.1e-03 \| 1176.8M \| 16 \| 16 \| 69,369 \| 100.0B \|


	## Bias, Risks, and Limitations

	Like any base or fine-tuned language model, AI can be prompted by users to generate harmful and sensitive content. Such content may also be produced unintentionally, especially in cases involving bias, so we recommend that users consider the risks when applying this technology. Additionally, many statements from any LLM are often inaccurate, so facts should be verified.


	## Citation

	BibTeX:

	```
	@article{MagnussonDataDecide2025,
	title={{DataDecide: How to Predict Best Pretraining Data with Small Experiments}},
	author={Ian Magnusson and Nguyen Tai and Ben Bogin and David Heineman and Jena Hwang and Luca Soldaini and Akshita Bhagia and Jiacheng Liu and Dirk Groeneveld and Oyvind Tafjord and Noah A. Smith and Pang Wei Koh and Jesse Dodge},
	year={2025},
	journal={arXiv preprint},
	}
	```

	## Model Card Contact

	For errors in this model card, contact ianmag@cs.washington.edu

	---
	license: apache-2.0
	datasets:
	- allenai/datadecide
	language:
	- en
	---
	![image/png](https://cdn-uploads.huggingface.co/production/uploads/62bddd0b1e22ec8427a0f27e/MwddQs_8OaU4128VYrwoU.png)

	More than one training run goes into making a large language model, but developers rarely release the small models and datasets they experiment with during the development process. How do they decide what dataset to use for pretraining or which benchmarks to hill climb on? To empower open exploration of these questions, we release [DataDecide](allenai.org/paper/datadecide)—a suite of models we pretrain on 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, over 14 different model sizes ranging from 4M parameters up to 1B parameters (more than 30k model checkpoints in total).

	## 350 Models over Differences in Data in Scale
	For each of our 25 datasets and 14 model sizes, we train a model linked below. Each has intermediate checkpoints (uploading after initial release), runs over 3 random seeds. All models finish training at a token to parameter ratio of 100 (e.g., 1B parameters -> 100B tokens).
	\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|
	\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|-----\|------\|------\|------\|------\|-----\|
	\| Dolma1.7 \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-1B) \|
	\| Dolma1.7 (no code) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-code-1B) \|
	\| Dolma1.7 (no math, code) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-math-code-1B) \|
	\| Dolma1.7 (no Reddit) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-reddit-1B) \|
	\| Dolma1.7 (no Flan) \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_7-no-flan-1B) \|
	\| Dolma1.6++ \| [4M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dolma1_6plus-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dolma1_6plus-1B) \|
	\| C4 \| [4M](https://huggingface.co/allenai/DataDecide-c4-4M) \| [6M](https://huggingface.co/allenai/DataDecide-c4-6M) \| [8M](https://huggingface.co/allenai/DataDecide-c4-8M) \| [10M](https://huggingface.co/allenai/DataDecide-c4-10M) \| [14M](https://huggingface.co/allenai/DataDecide-c4-14M) \| [16M](https://huggingface.co/allenai/DataDecide-c4-16M) \| [20M](https://huggingface.co/allenai/DataDecide-c4-20M) \| [60M](https://huggingface.co/allenai/DataDecide-c4-60M) \| [90M](https://huggingface.co/allenai/DataDecide-c4-90M) \| [150M](https://huggingface.co/allenai/DataDecide-c4-150M) \| [300M](https://huggingface.co/allenai/DataDecide-c4-300M) \| [530M](https://huggingface.co/allenai/DataDecide-c4-530M) \| [750M](https://huggingface.co/allenai/DataDecide-c4-750M) \| [1B](https://huggingface.co/allenai/DataDecide-c4-1B) \|
	\| FineWeb-Pro \| [4M](https://huggingface.co/allenai/DataDecide-fineweb-pro-4M) \| [6M](https://huggingface.co/allenai/DataDecide-fineweb-pro-6M) \| [8M](https://huggingface.co/allenai/DataDecide-fineweb-pro-8M) \| [10M](https://huggingface.co/allenai/DataDecide-fineweb-pro-10M) \| [14M](https://huggingface.co/allenai/DataDecide-fineweb-pro-14M) \| [16M](https://huggingface.co/allenai/DataDecide-fineweb-pro-16M) \| [20M](https://huggingface.co/allenai/DataDecide-fineweb-pro-20M) \| [60M](https://huggingface.co/allenai/DataDecide-fineweb-pro-60M) \| [90M](https://huggingface.co/allenai/DataDecide-fineweb-pro-90M) \| [150M](https://huggingface.co/allenai/DataDecide-fineweb-pro-150M) \| [300M](https://huggingface.co/allenai/DataDecide-fineweb-pro-300M) \| [530M](https://huggingface.co/allenai/DataDecide-fineweb-pro-530M) \| [750M](https://huggingface.co/allenai/DataDecide-fineweb-pro-750M) \| [1B](https://huggingface.co/allenai/DataDecide-fineweb-pro-1B) \|
	\| FineWeb-Edu \| [4M](https://huggingface.co/allenai/DataDecide-fineweb-edu-4M) \| [6M](https://huggingface.co/allenai/DataDecide-fineweb-edu-6M) \| [8M](https://huggingface.co/allenai/DataDecide-fineweb-edu-8M) \| [10M](https://huggingface.co/allenai/DataDecide-fineweb-edu-10M) \| [14M](https://huggingface.co/allenai/DataDecide-fineweb-edu-14M) \| [16M](https://huggingface.co/allenai/DataDecide-fineweb-edu-16M) \| [20M](https://huggingface.co/allenai/DataDecide-fineweb-edu-20M) \| [60M](https://huggingface.co/allenai/DataDecide-fineweb-edu-60M) \| [90M](https://huggingface.co/allenai/DataDecide-fineweb-edu-90M) \| [150M](https://huggingface.co/allenai/DataDecide-fineweb-edu-150M) \| [300M](https://huggingface.co/allenai/DataDecide-fineweb-edu-300M) \| [530M](https://huggingface.co/allenai/DataDecide-fineweb-edu-530M) \| [750M](https://huggingface.co/allenai/DataDecide-fineweb-edu-750M) \| [1B](https://huggingface.co/allenai/DataDecide-fineweb-edu-1B) \|
	\| Falcon \| [4M](https://huggingface.co/allenai/DataDecide-falcon-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-1B) \|
	\| Falcon+CC \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-1B) \|
	\| Falcon+CC (QC 10%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-10p-1B) \|
	\| Falcon+CC (QC 20%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-20p-1B) \|
	\| Falcon+CC (QC Orig 10%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-orig-10p-1B) \|
	\| Falcon+CC (QC Tulu 10%) \| [4M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-falcon-and-cc-qc-tulu-10p-1B) \|
	\| DCLM-Baseline \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-1B) \|
	\| DCLM-Baseline (QC 7%, FW2) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw2-1B) \|
	\| DCLM-Baseline (QC 7%, FW3) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-7p-fw3-1B) \|
	\| DCLM-Baseline (QC FW 3%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-3p-1B) \|
	\| DCLM-Baseline (QC FW 10%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-fw-10p-1B) \|
	\| DCLM-Baseline (QC 10%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-10p-1B) \|
	\| DCLM-Baseline (QC 20%) \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-qc-20p-1B) \|
	\| DCLM-Baseline 25% / Dolma 75% \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-25p-dolma1.7-75p-1B) \|
	\| DCLM-Baseline 50% / Dolma 50% \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-50p-dolma1.7-50p-1B) \|
	\| DCLM-Baseline 75% / Dolma 25% \| [4M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-4M) \| [6M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-6M) \| [8M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-8M) \| [10M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-10M) \| [14M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-14M) \| [16M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-16M) \| [20M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-20M) \| [60M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-60M) \| [90M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-90M) \| [150M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-150M) \| [300M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-300M) \| [530M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-530M) \| [750M](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-750M) \| [1B](https://huggingface.co/allenai/DataDecide-dclm-baseline-75p-dolma1.7-25p-1B) \|

	## Load a Model

	To load a specific model with HuggingFace:

	```
	from hf_olmo import OLMoForCausalLM # pip install ai2-olmo

	olmo = OLMoForCausalLM.from_pretrained("allenai/DataDecide-dolma1_7-1B", revision="step69369-seed-default")
	```

	### Model Description

	<!-- Provide a longer summary of what this model is. -->



	- Developed by: Allen Institute for AI (Ai2)
	- Model type: a Transformer style autoregressive language model.
	- Language(s) (NLP): English
	- License: The code and model are released under Apache 2.0.
	- Contact: Technical inquiries: `ianmag@cs.washington.edu`. Press: `press@allenai.org`

	### Model Sources

	<!-- Provide the basic links for the model. -->

	- Repository: [https://github.com/allenai/DataDecide](https://github.com/allenai/DataDecide)
	- Paper: [https:/allenai.org/paper/datadecide](https:/allenai.org/paper/datadecide)
	- Data: [https://huggingface.co/datasets/allenai/datadecide](https://huggingface.co/datasets/allenai/datadecide)

	## Data

	\| Source / Recipe \| Description \|
	\|----------------------------------------\|-------------\|
	\| Dolma1.7 Original, No code, No math/code, No Reddit, No Flan \| A 2.3T-token corpus (Dolma; 1.7 [Soldaini et al., 2024](https://arxiv.org/abs/2402.00159)) sampling common LM sources for open research. We ablate code, math/code, Reddit, or Flan subsets. \|
	\| Dolma1.6++ Original \| Dolma 1.6 plus additional sources from Dolma 1.7: RedPajama’s arxiv subset, openwebmath, algebraic stack, flan, starcoder, falcon. \|
	\| C4 Original \| The C4 dataset ([Raffel et al., 2019](https://arxiv.org/abs/1910.10683)) as prepared in Dolma 1.7, heuristically filtered from the April 2019 Common Crawl. \|
	\| FineWeb-Pro Original \| The FineWeb Pro corpus ([Zhou et al., 2024](https://arxiv.org/abs/2409.17115)), featuring model-driven data cleaning on FineWeb. \|
	\| FineWeb-Edu Original \| The deduplicated FineWeb-Edu subset of SmoLLM-Corpus ([Ben Allal et al., 2024](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)), focused on educational web pages. \|
	\| Falcon Original \| The Falcon RefinedWeb corpus ([Penedo et al., 2023](https://api.semanticscholar.org/CorpusID:259063761)) in Dolma 1.7, derived from Common Crawl through June 2023 and more aggressively filtered/deduplicated than C4. \|
	\| Falcon+CC Original, QC 10%, QC 20%, QC Orig 10%, QC Tulu 10% \| Falcon and Dolma 1.7’s Common Crawl. We quality filter to top 10% or 20% documents with reproduced or original [Li et al. (2024)](https://arxiv.org/abs/2406.11794) filter or retrain filter on pre-release version of Tulu-v3 ([Lambert et al., 2024](https://arxiv.org/abs/2411.15124)). \|
	\| DCLM-Baseline Original, QC 7% FW2, QC 7% FW3, QC FW 10%, QC 10%, QC 20% \| A SOTA Common Crawl corpus using best ablated deduplication, cleaning heuristics, and quality filter. We quality filter to top 7% of DCLM classified documents and further take 2+ or 3+ scores with FineWeb-edu classifier; or filter to top 3% or 10% with FineWeb-edu classifier; or take top 10% or 20% with reproduced DCLM classifier. \|
	\| λ% DCLM-Baseline + 1 – λ% Dolma1.7 \| Fractional combinations of Dolma1.7 and DCLM-Baseline mixing different proportions of the two datasets for λ ∈ {25%, 50%, 75%}. \|


	## Evaluation

	We evaluate all checkpoints over OLMES suite of 10 multiple choice question answering benchmarks
	([Gu et al., 2024](https://arxiv.org/abs/2406.08446)):

	- [MMLU (Hendrycks et al., 2021)](https://arxiv.org/abs/2009.03300)
	- [HellaSwag (Zellers et al., 2019)](https://arxiv.org/abs/1905.07830)
	- [ARC-Challenge (Clark et al., 2018)](https://arxiv.org/abs/1803.05457)
	- [ARC-Easy (Clark et al., 2018)](https://arxiv.org/abs/1803.05457)
	- [PIQA (Bisk et al., 2020)](https://arxiv.org/abs/1911.11641)
	- [CommonsenseQA (Talmor et al., 2019)](https://arxiv.org/abs/1811.00937)
	- [Social IQa (Sap et al., 2019)](https://arxiv.org/abs/1904.09728)
	- [OpenBookQA (Mihaylov et al., 2018)](https://arxiv.org/abs/1809.02789)
	- [BoolQ (Clark et al., 2019)](https://arxiv.org/abs/1905.10044)
	- [Winogrande (Sakaguchi et al., 2020)](https://arxiv.org/abs/1907.10641)

	We release all these evaluations:
	- for task-level metric results: [https://huggingface.co/datasets/allenai/DataDecide-eval-results](https://huggingface.co/datasets/allenai/DataDecide-eval-results)
	- for instance-level results: [https://huggingface.co/datasets/allenai/DataDecide-eval-instances](https://huggingface.co/datasets/allenai/DataDecide-eval-instances)


	## Hyperparameters

	\| Name \| Batch Size \| Hidden Dim. \| LR \| Model size \| Heads \| Layers \| Training steps \| Tokens trained \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| 4M \| 32 \| 64 \| 1.4e-02 \| 3.7M \| 8 \| 8 \| 5,725 \| 0.4B \|
	\| 6M \| 32 \| 96 \| 1.2e-02 \| 6.0M \| 8 \| 8 \| 9,182 \| 0.6B \|
	\| 8M \| 32 \| 128 \| 1.1e-02 \| 8.5M \| 8 \| 8 \| 13,039 \| 0.9B \|
	\| 10M \| 32 \| 144 \| 1.0e-02 \| 9.9M \| 8 \| 8 \| 15,117 \| 1.0B \|
	\| 14M \| 32 \| 192 \| 9.2e-03 \| 14.4M \| 8 \| 8 \| 21,953 \| 1.4B \|
	\| 16M \| 32 \| 208 \| 8.9e-03 \| 16.0M \| 8 \| 8 \| 24,432 \| 1.6B \|
	\| 20M \| 64 \| 192 \| 8.4e-03 \| 19.1M \| 8 \| 16 \| 14,584 \| 1.9B \|
	\| 60M \| 96 \| 384 \| 5.8e-03 \| 57.1M \| 12 \| 16 \| 29,042 \| 5.7B \|
	\| 90M \| 160 \| 528 \| 4.9e-03 \| 97.9M \| 12 \| 16 \| 29,901 \| 9.8B \|
	\| 150M \| 192 \| 768 \| 4.2e-03 \| 151.9M \| 12 \| 12 \| 38,157 \| 15.0B \|
	\| 300M \| 320 \| 1,024 \| 3.3e-03 \| 320.0M \| 16 \| 16 \| 45,787 \| 30.0B \|
	\| 530M \| 448 \| 1,344 \| 2.8e-03 \| 530.1M \| 16 \| 16 \| 57,786 \| 53.0B \|
	\| 750M \| 576 \| 1,536 \| 2.5e-03 \| 681.3M \| 16 \| 16 \| 63,589 \| 75.0B \|
	\| 1B \| 704 \| 2,048 \| 2.1e-03 \| 1176.8M \| 16 \| 16 \| 69,369 \| 100.0B \|


	## Bias, Risks, and Limitations

	Like any base or fine-tuned language model, AI can be prompted by users to generate harmful and sensitive content. Such content may also be produced unintentionally, especially in cases involving bias, so we recommend that users consider the risks when applying this technology. Additionally, many statements from any LLM are often inaccurate, so facts should be verified.


	## Citation

	BibTeX:

	```
	@article{MagnussonDataDecide2025,
	title={{DataDecide: How to Predict Best Pretraining Data with Small Experiments}},
	author={Ian Magnusson and Nguyen Tai and Ben Bogin and David Heineman and Jena Hwang and Luca Soldaini and Akshita Bhagia and Jiacheng Liu and Dirk Groeneveld and Oyvind Tafjord and Noah A. Smith and Pang Wei Koh and Jesse Dodge},
	year={2025},
	journal={arXiv preprint},
	}
	```

	## Model Card Contact

	For errors in this model card, contact ianmag@cs.washington.edu