Oriel — AI & technology worth noticing, every day.

open-index/hacker-news

Dataset · https://huggingface.co/datasets/open-index/hacker-news

Hacker News - Complete Archive Every Hacker News item since 2006, live-updated every 5 minutes What is it? This dataset contains the complete Hacker News archive: every story, comment, Ask HN, Show HN, job posting, and poll ever submitted to the site. Hacker News is one of the longest-running and most influential technology communities on the internet, operated by Y Combinator since 2007. It has become the de facto gathering place for founders, engineers… See the full description on the dataset page: https://huggingface.co/datasets/open-index/hacker-news.

First seen by Oriel on .

AI summary

This Hugging Face dataset provides a complete, live-updated Hacker News archive in monthly and five-minute Parquet files for research, analysis, and machine learning.

  • It includes stories, comments, Ask HN and Show HN posts, jobs, polls, and poll options, with every Parquet file matching the HN API schema.
  • Historical months are sourced from ClickHouse, while new items are fetched from the HN Firebase API every five minutes and committed as live Parquet blocks.
  • At midnight UTC, the current month is refetched authoritatively and live blocks are removed, ensuring monthly files are complete and deduplicated.
  • DuckDB can query files directly through hf:// paths, while datasets, pandas, huggingface_hub, and the CLI support streaming or selective downloads.
  • The archive contains 43,062,301 comments and 6,265,054 stories; GitHub is the most-linked domain with 213,999 stories.

#hacker news #parquet #duckdb #datasets

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Likes

Total likes reported by the source.

  • Hugging Face Datasets: 590, latest observation ; +226 over observed period since

Downloads (30d)

Rolling 30-day downloads. A change in this value is not new downloads during the observed period.

  • Hugging Face Datasets: 402,911, latest observation ; -31,414 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Datasetsopen-index/hacker-newshttps://huggingface.co/datasets/open-index/hacker-news Read discussion