Oriel — AI & technology worth noticing, every day.

somast/somali-nlp-corpus

Dataset · https://huggingface.co/datasets/somast/somali-nlp-corpus

Somali NLP Corpus A cleaned, deduplicated Somali text corpus for NLP and language-model research: 7,913,008 documents, 826,973,689 words, 1,129,084,663 tokens (v2 ByteLevel BPE, 1.365 tokens per word) from 14 public sources. Two license tiers There is no single license for this corpus. Each record keeps the license of the dataset it came from, in its license field, and records are split into two configurations: Config Records Use open 6,955,948 Licenses… See the full description on the dataset page: https://huggingface.co/datasets/somast/somali-nlp-corpus.

First seen by Oriel on .

AI summary

somast/somali-nlp-corpus is a cleaned, deduplicated Somali text corpus for NLP and language-model research, containing 7.9 million documents from 14 public sources under record-specific licenses.

  • The corpus contains 826,973,689 words and 1,129,084,663 tokens, with 6,955,948 open records and 957,060 non-commercial records.
  • Records retain source and license metadata; the dataset provides fields for language-detector confidence, content hashes, and quality flags.
  • A reproducible Rust pipeline applies normalization, language identification, exact and near deduplication, and masking of URLs, emails, and common phone formats.
  • Language labels can be inaccurate, personal data masking is best-effort, and quality, dialect, and register were not assessed through human review.
  • Users must follow each record's license, avoid identifying private individuals, and honor takedown requests.

#somali nlp #language datasets #data deduplication

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Likes

Total likes reported by the source.

  • Hugging Face Datasets: 15, latest observation ; +4 over observed period since

Downloads (30d)

Rolling 30-day downloads. A change in this value is not new downloads during the observed period.

  • Hugging Face Datasets: 61, latest observation ; +50 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Datasetssomast/somali-nlp-corpushttps://huggingface.co/datasets/somast/somali-nlp-corpus Read discussion