Single-Hash Bloom Filters: From Theory to Production
Bloom filters enable efficient set membership testing by trading perfect accuracy for speed and minimal memory overhead. At Bluesky, they eliminate millions of unnecessary database queries daily, with typical false positive rates around 1%. This approach became essential to scaling the platform cost-effectively.
Traditional multi-hash implementations created production bottlenecks. Bluesky’s mutex-locked hash filter access extended deployments to 90 minutes before switching to atomic CPU instructions (atomic-bloom) cut that to 15 minutes. Further optimization gains faced diminishing returns, but research papers revealed an alternative: a single hash function paired with multiple bit array partitions of coprime sizes. Each partition generates one checking position from the same hash, achieving the same false positive guarantees with less computation.
The author implemented this single-hash approach as gloom in Go, adding cache-conscious optimizations: 64-byte block alignment to guarantee lookups fit in one CPU cache miss, multiplication instead of division for partition selection, and write sharding to reduce atomic operation contention. The implementation has run in Bluesky’s production backend for nine months, proving that theoretical refinements can deliver meaningful real-world performance gains when paired with systems-level engineering.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.