Oriel — AI & technology worth noticing, every day.

HuggingFaceFW/fineweb-2

Dataset · https://huggingface.co/datasets/HuggingFaceFW/fineweb-2

🥂 FineWeb2 A sparkling update with 1000s of languages What is it? This is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages. The 🥂 FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments. In particular, on the set of 9 diverse languages we used to guide our processing decisions, 🥂… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2.

First seen by Oriel on .

AI summary

FineWeb2 is a reproducible, multilingual pretraining dataset providing filtered CommonCrawl data for over 1,000 languages and 1,868 language-script pairs.

  • The dataset contains 5,018,505,566 documents, 3.34 trillion words, 20.78TB of UTF-8 data, and 8.58TB of disk storage.
  • Data comes from 96 CommonCrawl snapshots spanning summer 2013 through April 2024, processed with datatrove and globally deduplicated per language.
  • Filtered and removed subsets are both provided, enabling users to apply alternative filtering without repeating global deduplication.
  • Processing uses GlotLID, language-specific confidence thresholds, filters, stopwords, PII anonymization, and per-language deduplication with rehydration metadata.
  • FineWeb2 is released under the ODC-By 1.0 license and was evaluated through hundreds of ablations across nine languages and diverse FineTasks tasks.

#multilingual datasets #commoncrawl #datatrove #pretraining data

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Likes

Total likes reported by the source.

  • Hugging Face Datasets: 929, latest observation ; +2 over observed period since

Downloads (30d)

Rolling 30-day downloads. A change in this value is not new downloads during the observed period.

  • Hugging Face Datasets: 146,735, latest observation ; +8,235 over observed period since

Aliases

SourceExternal IDURL
Hugging Face DatasetsHuggingFaceFW/fineweb-2https://huggingface.co/datasets/HuggingFaceFW/fineweb-2 Read discussion