Probabilistic Data Structures in Go
Web page · https://antonz.org/probabilistic-data-structures
First seen by Oriel on .
AI summary
This post explains approximate data structures and introduces unsure, a dependency-free Go package designed to reduce memory usage while providing explicit accuracy guarantees.
- A Bloom filter uses 1.24 MB for 1,000,000 distinct 8-byte strings at a 1% false-positive rate, about 50 times less than a regular set.
- Unsure’s blocked Bloom filter accesses one 512-bit CPU-cache-sized block per operation, improving speed while using 3–15% more memory than a standard filter.
- A count-min sketch estimates frequencies without undercounting; its error is bounded relative to total observations, making rare-item estimates less reliable.
- Unsure’s count-min sketch uses conservative updates and 32-bit counters, achieving about half the memory, lower error, and faster operations than the comparison implementation.
- HyperLogLog estimates distinct-item counts and supports merging; with precision p=14, it provides approximately 0.81% standard error using a 16 KB sketch for one million items.
#go #probabilistic data structures #bloom filters #hyperloglog
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hacker News: 61, latest observation ; +56 over observed period since
- Lobsters: 66, latest observation ; +64 over observed period since
Comments
Comment count reported by the source.
- Hacker News: 7, latest observation ; +7 over observed period since
- Lobsters: 12, latest observation ; +12 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hacker News | 50496870 | https://antonz.org/probabilistic-data-structures/ Read discussion |
| Lobsters | xca2qj | https://antonz.org/probabilistic-data-structures/ Read discussion |