I'm trying to be cautious yet open-minded and optimistic with Redis modules.
Redis is such a well-written C software that I can't help but think modules might become an antipattern to Redis. I like the idea that the option to use modules exists, but I hope it doesn't spiral into dependency hell as it is with the Ruby ecosystem. I hope library developers think very hard before employing and requiring a module.
Redis is fast, easy to use, and extremely flexible, and makes it easy to use any of the hosted options (Redis Labs, RedisToGo). Its allure is that Redis is batteries-included yet small enough like a Swiss knife. I wouldn't mind installing a module if it brought in significant optimization for a specific use case. I personally hope most library developers will pretend modules don't exist, and that only app developers will use them. I have this selfish desire that most libraries stick to the vanilla Redis features, because for most things you can use app code to manipulate the features you need without modifying Redis and compromising performance. Modules might add to the heap of things to think about, such as special Redis deployments or the things that can go wrong with the module code.
My fear is that instead of years of using the simple Lego blocks that is vanilla Redis, modules will start a pattern where we need to install special modules to use certain libraries.
TL;DR, I'm worried for the potential for abuse with Redis modules. My love for Redis was how simple and powerful it is, and yet now Redis has grown up, ready for modules.
I can totally sympathize with your feelings, but I believe that it depends a lot on how the Redis project will set the tone about this feature. Example of things that will make, in my opinion, the difference:
1. There will never be a way to install modules automatically provided by the Redis OSS project. The use case is to grab a few modules you may need for vertical use cases that are otherwise not cover, so you just don't install a number of modules depending on other modules and so forth.
2. I'll not write modules but will care about exporting a good modules interface. What I believe has to be done, will be implemented inside the Redis core, not outside. SO it's totally a community thing. For the project itself is like: "we are going to give you this possibility, so that the fact the core is conservative about what functions to add, does not block you". But the model for Redis does not change, Redis is what we provide in the core. I work only on core stuff. The site only documents what we have in the core.
3. I'm not going to debug for hours or days crashes happening with modules loaded. On crash the list of modules installed will be reported in the crash trace.
So modules are going to be a good addition and there will be use cases that can be solved very well with modules, but are not going to change what Redis is. It's up to the user if to use Redis "core", or if to leverage modules. Wise users will pick modules when needed, and will inspect modules for quality and fit. Unwise users will install a lot of garbage, then eventually say Redis is a shit. But this is how the world works :-)
The only other C project I know of with a module system in Linux. I've actually found that Linux modules make development _much_ easier because you can isolate your code from the rest of the Linux code base. Of course, the last time I was professionally developing with the Linux kernel, git hadn't been invented yet and you really couldn't keep the entire kernel in version control without going insane. So, maybe things are different now.
Nginx's modularity and tolerance/embrace of plugins is absolutely amazing. I use a combination of openresty[0], lua scripting, and third party modules in our API servers (coincidentally, one that talks directly to Redis) and it's still one of the fastest, best pieces of software in our stack.
I understand the concern, but if it ends up being half as flexible (and I already love me some Redis) I can only see good things.
I was talking more of my experience than C projects in general. However, I'd expect a smaller percentage of C projects to support modules compared to Ruby given the differences in the language.
My hope is that the modules will be used more for common use cases (leaderboards, real-time counters, etc) instead of turning into gem dependency hell. Having a community backed module for a single purpose in that regard sounds pretty good, and the fact that they are written in C should filter out devs that follow a higher level dependency model as raw performance is a key metric.
I agree. I think Sidekiq is a good example of how modules are used. You can add the Pro features, use a community plugin, but the plugins usually don't affect core functionality (such as the Sidekiq "cron" scheduler, error reporting options, etc).
I think C is both a blessing and a curse. The challenge of using C correctly has led to memory leaks, remote code execution, segfaults, etc. I'd use Go if it's possible to write Redis modules.
It is possible to make Go redis modules since go 1.5 thanks to the c-shared buildmode, I would be surprised if there wasn't a go package to make it easy within the month.
You could use C++, Ada, or Rust if you use the right pragmas. I don't use Go but I'd be surprised if it can't export C ABI compatible functions somehow.
agree. Magento, wordpress and npm come to mind as places where quality is difficult to ascertain and can easily turn a great setup into a brick'd prod server.
What projects are people using Redis for? I'd love to know how others are using Lua or plan to use the new module system specifically.
We're using redis as a queue (Sidekiq) and also to store the index.html page for our SPA. We have an "active" key in redis and its value is the key for the current index.html. This let's us push new versions of our app into redis (each generation of our asset files has a unique name so a new index.html gives you a new version of the SPA). We can then add a get param to our URL and pull a specific version of the index.html out of redis. Once we've smoke tested, we update our "current" value to point to the new index.html and, boom, zero downtime deploys.
At my previous job (industry-specific CMS supporting many websites) we used redis simply as a distributed cache to support our this application since it was deployed to a load-balanced webfarm. The CMS platform we were building off of had its own webfarm-cache solution but we found it to result in a lot of inter-server communication which only hurt our load issues even more. Redis came in as a very simple out-of-the-box solution to that problem and performance was improved tremendously. I abstracted the client in case we had to switch to Memcached etc but Redis really just knocked our socks off and the documentation was a godsend. I left before adding Redis Cluster to our stack unfortunately but I did get to experiment with that and it also looked very promising. Having used it I strongly recommend my colleagues look into it anytime caching comes up.
We use Redis as a record level distributed lock on top of pub/sub and a cache and a queue. We also take advantage of expiring keys to detect if users are connected or have activity. We used to use it a session store but switched to encrypted cookies. I know some have used it as a semaphore as well.
What did the semaphore usage look like? Did it use pub/sub or something else?
Also what encryption algorithm are you using on your cookies. I ask because certain algorithms produce malleable ciphertexts, which could potentially be a huge vulnerability.
I use redis for a multiplayer mobile game. Stores user details, game scores, global leaderboard and randomly generated game data that exists for the time of a single game.
The Internet Archive uses Redis for a lot of things. We use it as a read-only cache for all of the metadata about our "items" (books/videos/etc). We use it to store information about "save page now" pages in the Wayback Machine, so that you can immediately replay a saved page. (The main Wayback index is 60 TBytes; not in Redis today, maybe someday.)
There was a nice talk given by someone at Twitter about how they use Redis, including their shortcomings (which, interestingly, they had to solve by forking Redis and fiddling with the internals; maybe modules will help them get back closer to upstream ?)
I started playing with Modules & LuaJIT, mostly a proof of concept of what can be done. Much more to do, but my goal is to replace my existing Lua/Redis scripts with this.
I think it's a very good idea because it will reduce the pressure on the core development team about feature X or Y. "You really want this feature? Write it as a module."
Then it can serve as an incubator before importing functions into the core library.
The only problem with modules is that poorly written modules may cause unstability. One way to mitigate this is to have some sort of fuzzer/stress tester for developers to test their modules against.
Totally agree. It's really easy to break redis if you're not careful. When I wrote my module I made sure to do lots of random input testing, stress testing, and running things in valgrind to make sure everything is kosher.
I really like these types of examples but to fixate on the fact its 15 lines of code, forgetting about the many thousands of lines of code below it that do the actual work. Don't forget to thank your developers for making an API that can show you how to use the library in 15 lines.
You are misreading my title. As someone who actually helped with building it, the point was not to boast about it, but show how little you need to get a module stub up and running.
Usually I hate those titles too because they show some "import gravity" type of thing, but here the point was to just show that the boilerplate is roughly 15 LOC.
What's the point of being so proud it only takes 15 lines of code? If it had taken 20, would people have groaned and walked away? Examples like this aren't "real world" anyway, so...?
It's small enough to get a rough feel for how complicated a module would be. It's small enough that it's like "hello world" - test your toolchain (etc) install. It's simple enough to start understanding the process without having to read a bunch of docs. It's straight-forward, so there's no need to both puzzle out the module interface AND someone's module logic.
There are probably plenty more reasons I'm not thinking of ATM.
Redis is such a well-written C software that I can't help but think modules might become an antipattern to Redis. I like the idea that the option to use modules exists, but I hope it doesn't spiral into dependency hell as it is with the Ruby ecosystem. I hope library developers think very hard before employing and requiring a module.
Redis is fast, easy to use, and extremely flexible, and makes it easy to use any of the hosted options (Redis Labs, RedisToGo). Its allure is that Redis is batteries-included yet small enough like a Swiss knife. I wouldn't mind installing a module if it brought in significant optimization for a specific use case. I personally hope most library developers will pretend modules don't exist, and that only app developers will use them. I have this selfish desire that most libraries stick to the vanilla Redis features, because for most things you can use app code to manipulate the features you need without modifying Redis and compromising performance. Modules might add to the heap of things to think about, such as special Redis deployments or the things that can go wrong with the module code.
My fear is that instead of years of using the simple Lego blocks that is vanilla Redis, modules will start a pattern where we need to install special modules to use certain libraries.
TL;DR, I'm worried for the potential for abuse with Redis modules. My love for Redis was how simple and powerful it is, and yet now Redis has grown up, ready for modules.