9 Comments
User's avatar
Some dude's avatar

You also mentioned HyperLogLog as a follow-up - not that anyone asked, but this is my favorite video explaining it: https://www.youtube.com/watch?v=lJYufx0bfpw

There are production-grade implementations of HyperLogLog and other sketches in Apache DataSketches (https://datasketches.apache.org/)

Teiva Harsanyi's avatar

Thanks! I wanted to make a dedicated post for HyperLogLog, I'll definitely add your video as a reference.

Some dude's avatar

This is referring to the CVM Algorithm, as depicted here, right?

https://www.youtube.com/shorts/47PCjlbxk74 (full vid: https://www.youtube.com/watch?v=MZI3aL1igP8)

Teiva Harsanyi's avatar

Thanks, I didn't know it was called this way. Funny enough, the first time I heard about it was before 2023, when the paper was released. I'm going to reference it.

Robin's avatar

Insanely beautiful

Teiva Harsanyi's avatar

I loved it as well :)

Gabriele Cimato's avatar

Thank you for this post! My initial thought was more oriented to UX with optimistic updates. That wouldn't have really solved the problem, but just hid it away. First time I hear about Probabilistic increments, pretty cool!

Nick Ang's avatar

Hey Teiva, thanks for this insightful post. Probabilistic increment - definitely an elegant solution to a problem that general SQL databases aren't optimised for!

Teiva Harsanyi's avatar

Thank you, glad you liked it.