open-index/hacker-news
Dataset · https://huggingface.co/datasets/open-index/hacker-news
Hacker News - Complete Archive Every Hacker News item since 2006, live-updated every 5 minutes What is it? This dataset contains the complete Hacker News archive: every story, comment, Ask HN, Show HN, job posting, and poll ever submitted to the site. Hacker News is one of the longest-running and most influential technology communities on the internet, operated by Y Combinator since 2007. It has become the de facto gathering place for founders, engineers… See the full description on the dataset page: https://huggingface.co/datasets/open-index/hacker-news.
First seen by Oriel on .
AI summary
This Hugging Face dataset provides a complete, live-updated Hacker News archive in monthly and five-minute Parquet files for research, analysis, and machine learning.
- It includes stories, comments, Ask HN and Show HN posts, jobs, polls, and poll options, with every Parquet file matching the HN API schema.
- Historical months are sourced from ClickHouse, while new items are fetched from the HN Firebase API every five minutes and committed as live Parquet blocks.
- At midnight UTC, the current month is refetched authoritatively and live blocks are removed, ensuring monthly files are complete and deduplicated.
- DuckDB can query files directly through hf:// paths, while datasets, pandas, huggingface_hub, and the CLI support streaming or selective downloads.
- The archive contains 43,062,301 comments and 6,265,054 stories; GitHub is the most-linked domain with 213,999 stories.
#hacker news #parquet #duckdb #datasets
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Likes
Total likes reported by the source.
- Hugging Face Datasets: 590, latest observation ; +226 over observed period since
Downloads (30d)
Rolling 30-day downloads. A change in this value is not new downloads during the observed period.
- Hugging Face Datasets: 402,911, latest observation ; -31,414 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Datasets | open-index/hacker-news | https://huggingface.co/datasets/open-index/hacker-news Read discussion |