Hacker News new | past | comments | ask | show | jobs | submit login

This was posted a while ago, and I have since implemented bitmaps myself. One thing I learned from the documentation[1] is that setting an initial bit at a very high-numbered place (like 2^30 - 1) takes a while to allocate (compared to Redis's normal speed) and blocks other operations in the process.

In my case, and it appears to be true for Spool too, I don't know what bit will be set first. It could be 12 or it could be 2938251, so to prevent a slowdown if the initial bit is a high place I use buckets of bitmaps, each holding around 8 million bits.

[1] Read Warning: http://redis.io/commands/setbit




There is an easy fix for this, that is also the recommended way to use bitmaps in Redis: split your bitmap among multiple keys.

For instance you want to set bit i, but you want k bits per every key, the you do:

    keyname = "bitmap:"+(i/k)
    keybit = i%k
k can be fairly large, like 128k bytes per key. It's still small but big enough for keys overhead to be negligible.


Wow, that's almost exactly how I implemented it:

   var bucketSize = 8190;

   ...

   var bucketNumber = Math.floor(userId / bucketSize),
       bitInBucket = userId % bucketSize;
...correction on my last comment, looks like I use ~8 thousand bits per bucket, not 8 million.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: