It is a blast. Although we had our concerns because you can't see what's inside a Beanstalkd queue (duh, because it's a queue), but building a system for monitoring the failures was not a big deal.

And of course the tubes rock. Now I can build a separate tube for each retry, I can set different priorities for the tubes (so the more fail prone tubes won't block all workers), I can set a deadline for a job to finish and if the worker dies in the middle the job will be returned for the other workers. And best of all, I can monitor all tubes nicely with Graphite.

Why do you feel that a queue is inherently unpeekable?

You can peek, but it's not built for looking through all the items in it. It is doable, but not very efficient.

Do you have a custom solution for monitoring via Graphite? If so, is it something you can share?

It's pretty straightforward. Mostly we have custom data not related to Beanstalkd to monitor, but basically there's two scenarios:

1. When doing something, increment a counter in Statsd, which will aggregate the values to Graphite. Works well for custom stuff.

2. Write a rake task to poll Beanstalkd with `stats` command, reduce the data and send it straight to Graphite.

I'm really planning to open source the queuing system we're using because it's so simple. It might need a prettier admin view and some nice way of plugging in more monitoring systems before I put it to public.

We also use beanstalk a ton where I work. We have put multiple billions of jobs through it over the past few years. It's rock solid and one of my favorite pieces of software.