Oriel — AI & technology worth noticing, every day.

Probabilistic Data Structures in Go

Web page · https://antonz.org/probabilistic-data-structures

First seen by Oriel on .

AI summary

This post explains approximate data structures and introduces unsure, a dependency-free Go package designed to reduce memory usage while providing explicit accuracy guarantees.

  • A Bloom filter uses 1.24 MB for 1,000,000 distinct 8-byte strings at a 1% false-positive rate, about 50 times less than a regular set.
  • Unsure’s blocked Bloom filter accesses one 512-bit CPU-cache-sized block per operation, improving speed while using 3–15% more memory than a standard filter.
  • A count-min sketch estimates frequencies without undercounting; its error is bounded relative to total observations, making rare-item estimates less reliable.
  • Unsure’s count-min sketch uses conservative updates and 32-bit counters, achieving about half the memory, lower error, and faster operations than the comparison implementation.
  • HyperLogLog estimates distinct-item counts and supports merging; with precision p=14, it provides approximately 0.81% standard error using a 16 KB sketch for one million items.

#go #probabilistic data structures #bloom filters #hyperloglog

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hacker News: 61, latest observation ; +56 over observed period since
  • Lobsters: 66, latest observation ; +64 over observed period since

Comments

Comment count reported by the source.

  • Hacker News: 7, latest observation ; +7 over observed period since
  • Lobsters: 12, latest observation ; +12 over observed period since

Aliases

SourceExternal IDURL
Hacker News50496870https://antonz.org/probabilistic-data-structures/ Read discussion
Lobstersxca2qjhttps://antonz.org/probabilistic-data-structures/ Read discussion