somast/somali-nlp-corpus
Dataset · https://huggingface.co/datasets/somast/somali-nlp-corpus
Somali NLP Corpus A cleaned, deduplicated Somali text corpus for NLP and language-model research: 7,913,008 documents, 826,973,689 words, 1,129,084,663 tokens (v2 ByteLevel BPE, 1.365 tokens per word) from 14 public sources. Two license tiers There is no single license for this corpus. Each record keeps the license of the dataset it came from, in its license field, and records are split into two configurations: Config Records Use open 6,955,948 Licenses… See the full description on the dataset page: https://huggingface.co/datasets/somast/somali-nlp-corpus.
First seen by Oriel on .
AI summary
somast/somali-nlp-corpus is a cleaned, deduplicated Somali text corpus for NLP and language-model research, containing 7.9 million documents from 14 public sources under record-specific licenses.
- The corpus contains 826,973,689 words and 1,129,084,663 tokens, with 6,955,948 open records and 957,060 non-commercial records.
- Records retain source and license metadata; the dataset provides fields for language-detector confidence, content hashes, and quality flags.
- A reproducible Rust pipeline applies normalization, language identification, exact and near deduplication, and masking of URLs, emails, and common phone formats.
- Language labels can be inaccurate, personal data masking is best-effort, and quality, dialect, and register were not assessed through human review.
- Users must follow each record's license, avoid identifying private individuals, and honor takedown requests.
#somali nlp #language datasets #data deduplication
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Likes
Total likes reported by the source.
- Hugging Face Datasets: 15, latest observation ; +4 over observed period since
Downloads (30d)
Rolling 30-day downloads. A change in this value is not new downloads during the observed period.
- Hugging Face Datasets: 61, latest observation ; +50 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Datasets | somast/somali-nlp-corpus | https://huggingface.co/datasets/somast/somali-nlp-corpus Read discussion |