"In addition, the names of Hazelcast’s datatypes, and the functions provided on those types, imply a certain fitness-for-purpose, e.g. that users can use these types and functions in a meaningful way. What is the point of an ID Generator which emits duplicate IDs? A lock that doesn’t lock? Who wants an AtomicReference which is not atomic? Of what possible use is a queue which doesn’t, well, queue?"
Wow. I wonder if the documentation and naming schemes are bad because they're aspirational, or because the developers don't know what they're doing?
Not with respect to Hazelcast specifically, but it's often the latter, but not maliciously so. It's fairly difficult to implement a lot of those things correctly, and fairly easy to convince yourself you were thorough.
Probably not. You as a developer could write a Jepsen harness for your system and find the same results seen here. You could also write property-based tests to improve your testing model and input space coverage. You could also use mutation testing to see how good your tests are, etc. etc. etc.
At my last job we held off running a chaos test suite against our platform in the certification environment until _after_ the QA department came back with their report and gave the platform all green checkmarks. This was an exercise in both validating the value of the chaos engineering/testing work as well as partially discrediting the idea that QA is capable of exercising enough of the system in enough ways to expose non-obvious/nuanced edge cases.
I mean I guess there's an argument to be made that QA could be engineering and building their own rich tooling like a chaos testing framework, but that seems extremely unlikely to happen in the current typical formulation of QA.
The documentation/naming schemes comes from the Java interfaces which Hazelcast implements. The issue of course is those interfaces were I assume intended to only run on one JVM.
And of course they knew what they were doing since that was the whole point of Hazelcast. You could use the same Java constructs you knew and loved but on a distributed grid. Problem is of course that it wasn't well implemented and likely not such a good idea to begin with.
"When there is a network partition we favour availability over consistency. Which is what the Jepsen test shows. [...] It is also the PACELC contract of the entire category of IMDG systems. IMDGs are used for maximum speed and low latency. They represent a set of design trade-offs to achieve this primary purpose."
For anyone wondering if there is a set of Hazelcast like data structures and primitives that actually have safe, strong consistency (based on Raft consensus), check out Atomix:
Based on what data? Even hazelcast had that same claim before they got exposed?
I've only seen 2 systems that did rigorous in-house fault-tolerant testing (foundationdb and cockroachdb) and later when "jepsen-verified", they actually backed their claim or had few modifications to uphold their claim.
We use Hazelcast at Verizon Labs for a couple services, and one of the issues we ran into is how much work it takes to replicate split brain to test various and custom merge policies.
When reading that the distributed map implements the ConcurrentHashMap interface we were immediately skeptical, but it took more work to prove our skepticism was founded.
Hazelcast has a fantastic foundation for building clustered applications, but some of the design choices they made both internally and API wise are not choices we would have made.
We are moving to a custom merge policy because of some of these choices.
If a system is so "broken by design", shouldn't it include a flag for running under worst-case mode, so that users can more easily detect faults in their assumptions about the system?
Wait, is this Kyle Kingsbury? If so, where is his name? Who are the authors? I notice this text doesn't mention the authors:
"We wish to thank Jordan Halterman for his discussion of Hazelcast use cases. Luigi Dell’Aquila & Luca Garulli from OrientDB, and Denis Sukhoroslov from BagriDB, were instrumental in understanding those systems’ use of Hazelcast. Thanks also to Julia Evans, Sarah Huffman, Camille Fournier, Moishe Lettvin, Tim Kordas, André Arko, Allison Kaptur, Coda Hale, and Peter Alvaro for reading and offering comments on initial drafts. This research was performed independently by Jepsen, without compensation, and conducted in accordance with the Jepsen ethics policy."
Hahaha I suppose that's an entirely legitimate question, isn't it? Just to reassure you, Lisa, yes, this is me, and I'm the only person who works at Jepsen. I'm still exploring voices for the Jepsen brand--since at some point I might hire more researchers, more recent stuff is written with an organizational "we". Suppose I should start adding bylines. :)
completely off-topic:
have you ever thought about giving your training (https://jepsen.io/training) in an online way (mooc, udemy, whatever)?
I would love to learn about distributed systems from you, but today I think it would be almost impossible since you only seems to give in organization trainings.
FWIW I think you're right to take this approach--most in-depth technical classes aren't good fits, for that matter, from a pricing perspective if nothing else. I charge a few hundred dollars a head to teach somebody in-person. The Fundamentals of Cloud Architecture course that I typically do is one I could, for the most part, extend to an online course...but, given that it includes the ability to contact me, etc., I wouldn't want to charge less than $100 for it.
It's a good course; better, IMO, at what it teaches than equivalent Udemy courses. And I'm a pretty good teacher, I can be pretty engaging while talking about this stuff and it's fun. But at the race-to-the-bottom prices of the MOOC economy, it's a nonstarter. The Udemy "every class is ten bucks" disease discourages really capable, competent people from sharing what they know.
(And Packt et al. finding somebody to read some slides is not a good counterexample. I said "really capable, competent" for a reason. I was approached by one of their competitors--a bigger company than they are--to write a book on Mesos on the back of two blog posts...)
Have you explored high level consulting in inception phase of distributed system design? (and p.s. thank you for your exceptional work and knowledge sharing.)
Oh yeah, I guess this is a startup website isn't it! And Jepsen is... sort of... a startup?
For context, Jepsen started as a series of volunteer nights-and-weekends blog posts and conference talks. About three years in I bootstrapped the business as a consultancy, with one client lined up and ~15K USD in [available] savings. Scraped through the first year by dipping into credit cards, learned a lot about pricing and pipelines, and am doing pretty well now. Having money in the bank lets me do more volunteer work, like this analysis. :)
Jepsen makes money through consulting services (usually paid analysis work), training classes, speaking engagements (I charge at for-profit orgs and speak for free at nonprofit events), and Amazon Marketplace subscriptions.
The convention in serious consulting is that a report printed without author names under the company's name is backed by the whole company and speaks for the whole company, which turns out to be the case here.
Gladly would voice my support for and validation of this research. It might not be glorious but this fundamental synchronization validation keeps us all in check.
Another solution in the same space is Apache Geode. Its commercial version has been around since a long time and has been well proofen on very large loads in systems that require low response times.
Worked on GemFire prior to Pivotal acquisition (before Geode) and currently work for SnappyData.
You can imagine GemFire/Gridgain as an apples-to-apples comparison. Both are "enterprise" in-memory data grids originally intended for managing data in low-latency OLTP applications which later added analytics/OLAP features. Geode/Ignite are the open source options for these two IMDGs and also a good apples-to-apples comparison. (Hazelcast also has enterprise/OSS verisons I would compare accordingly)
I can't speak to the current comparison between these systems, but I can compare them to SnappyData. SnappyData deeply integrates GemFire with Spark to bring high concurrency, high availability and mutability to Spark applications. In the world of combining Spark with a datastore over a connector (cassandra, hive, mysql, mongo etc) to enable "database-like" features in Spark, SnappyData has taken the next step of integration. In Snappy, the database (GemFire) and the Spark executors share the same block manager and VM so the systems no longer communicate over a "connector." This, along with our database optimizations, provides the best performance for Spark applciations in what I like to call the "Spark Database Ecosystem."
As such, comparing SnappyData to GemFire/Hazelcast/Gridgain does not make much sense unless you are trying to use Spark in conjunction with these systems. In that case, the main difference I would point out is that SnappyData will necessarily perform better as any of them would need to use a connector to interact with Spark. The better comparison would be between SnappyData and Ignite, as Ignite contains a direct Spark abstraction called "IgniteRDD." That said, the majority of the comparisons/benchmarks we've run have been against MemSQL+Spark and Cassandra+Spark, so I don't have much to say about Ignite vs SnappyData.
User manigandham mentions SnappyData's Approximate Query Processing features (called Synopses Data Engine) which is unique within this space, but a discussion of which would take this too far afield.
1st time coming across snappydata, looks like it's more of a sql execution engine built into/on top of apache spark with a focus on statistical accurate but much faster queries using a margin of error. Pretty neat in itself.
I haven't explored ignite seriously. Care to explain if you mean they implement these kind of features (locks, counters) properly or that they don't try to provide features that their design safely can't.
GridGain, the commercial company behind Ignite, and the community seem to be implementing the features correctly so that they work as advertised, backed up by the case studies and personal accounts I've heard.
Used Ignite on two projects. Was a good match for one, was a poor match for another (not Ignite's fault - poor choice at project outset - did not match initial team's skills or project requirements). In both cases was very impressed with Ignite's docs and community. It works pretty much as advertised/documented - and the GridGain guys really know their stuff. The commercial version - GridGain - has some nicer tooling/ UI capabilities and support - if you have the budget, otherwise the Apache project is solid to build on and well documented.
Question: why do we still use sequential ID's for identifying objects in data stores? There are many known problems with using numeric ID's (anyone remember Twitter's overflow issue), so when they are sent to a remote system, they should probably be strings anyways. Sorting or doing math on ID's is basically useless. Memory savings are the only thing I can think of, and while yes that's a nice feature, when a cluster of N nodes is generating ID's that are supposed to be globally unique, aren't UUID-type things better?
We have been using Hazelcast for a while now. Currently evaluating K tables if we could get the same performance. Any learnings or advices would be welcome if anyone has done the switch.
What fascinates me about the now-known-to-be-untrue claims in the Hazelcast documentation is that the claims were made despite the developers having know way of knowing if they were true because they’d not conducted these kinds of tests themselves.
Documentation that reflects what the developers wish were true rather than what is actually true is not a new phenomenon, but is potentially fatal for this kind of software.
For someone listed as a partner on their homepage to be actively attempting to remove the system entirely is also pretty bad news. Presumably a number of users of Hazelcast have been bitten by problems, and enough of them are able to attribute them to Hazelcast...
Wow. I wonder if the documentation and naming schemes are bad because they're aspirational, or because the developers don't know what they're doing?