You also could use an advisory exact lock, which would serialize the order of input and force everyone to wait in line, but the advantage here is that we are allowed shared concurrent reads while maintaining our lock. Thank you for the feedback. It was fun to revisit after 10 years. The initial piece came after a Columbus, OH Ruby meetup that Jose attended. It was quite fun to see him put it together as a whole and I ran with things a bit further than his first ideas with this.
SKIP LOCKED is amazing for this kind of thing. I used it to build a transaction outbox a few years ago.
One thing worth looking into if you do this in production is adding a way to add partitions such that each partition is single threaded. It’s the only way to guarantee ordering if your jobs are doing anything non-deterministic.
We have a system where each pod spins up around 30 scheduled job instances of one job each processing a "partition", then transaction outbox is queried with hash of identifier equating it to partition.
We increased partition counts on sale days and it works well for us.
Couple of gotchas we had were.
1) Using hashtext from postgres is sketchy.
2) Increasing partiton count is an orchestra which requires stopping the partition.
If you have any feedback or anything is unclear feel free to open an issue. I am thinking I am going to take this and expand the concepts to start as a beginners primer going through the primary concepts in brief akin to Elixir School, and then expand it into building this and a web service that is using it and offering some real time features.
I wrote this up after having written a tutorial on doing this in GenStage around a decade ago, and thought it was interesting to have the two of them side by side to consider. I linked the original in the document linked to here.
Overall, I'm wildly impressed at how this Elixir code held up, and it was a joy to revisit this.
Great write up, thanks for sharing! Nothing against Oban, but it is nice seeing someone in the Elixir community not just say, "run Oban" and drop the mic
I attended Nolen's recent talk at Recurse, and these absolutely mad but fundamentally funny and cool games as one-shots are so delightful.
It reminds me of the older internet, when people would make things just to be silly and have fun. I had some fun last night inspired by some of the stuff he has been posting making a very funny kind-of-quine that outputs its own source code of the page showing it, via BEAM decompilation and some other tricks. I wish I had the time to crank out things like this, and it makes me smile to know that there are people out there creating things in this manner that will just keep things interesting for all of us.
See my other comment. But, IMHO, you sit at a time where you will be able to make more cool things faster than anyone has ever been able to before. And that if you are a curious person and/or voracious absorber of information, you could become a skilled developer quite quickly and begin to use the tools in more meaningful ways and have a lot of fun and a cool career ahead of you.
I have been doing this about 12 years, and am quite comfortable in my position though. I can understand anxieties for new entrants in an industry that historically has been solid and reliable alongside high-paying. But fundamental shifts in how we work don't necessarily mean gloom. It means innovation on fronts that we haven't had to think about it for a while.
If making things excites you, its a great time to be doing it.
> I can understand anxieties for new entrants in an industry that historically has been solid and reliable alongside high-paying
I started in tech in the 90's. Between 1994-2014 I worked _way_ more than most people in most industries because IT (and management) is one of those exempt labor categorizes. High-paying, sure... but when you spend a decade or two working 50-55 hours a week the hourly rate ain't so good. :)
I guess it depends on how you come in. I was in a position where I came around 2012 with some self taught skills and managed to start out with a 50-60 hour grind but 70k wage and climb slowly into the 250+ range in the ~decade since.
However, I location-optimized. I started in rural Ohio ~2hrs from Detroit, and ended up in New York City after some bouncing to smaller tech cities climbing the ladder. But I think my hourly rate comes out pretty great, even historically to my career beginning, especially for a non-college-grad.
I consider my job to be being a skilled practitioner that develops solutions for complex problems using technology. New technology is available to allow me to do that for my employer and potentially many others. This does not worry me, but does demonstrate I have learning to do. I have set out on a journey to do that learning, and found it pleasant so far. I have enjoyed the changes it provides to my development workflow and find myself more productive.
Times always change, but being useful and providing answers will continue to provide value to business and society, and if one wants to focus on that over the minutiae of programming, they will probably have a much better time as things change more and faster.
Unrelated, but I would implore everyone who enjoys anything relating to the words "dark star" to listen to this compilation of the jams of every Dark Star the Grateful Dead played in 1972. It has so much continuity, and is a great bit to have playing while working. https://archive.org/details/DarkStar_1972
