I was following Jepsen results since Kyle's first post and it's amazing that the blog post series became a well respected company
The report revealed the following unknown consistency issues which we had to fix:
- duplicated writes by default
- aborted read with InvalidTxnState
- lost transactional writes
The first issue was caused by a client: starting with recent versions the client has idempotency on by default but when the server-side doesn’t support it (we had idempotency behind a feature flag) the client doesn’t complain. We will enable idempotency by default in 21.1.1 so it shouldn't be an issue. Also it's possible to turn the feature flag on for the older versions.
The other two issues were related to the transactions; we haven’t chaos tested the abort operation since it’s very similar to commit but even the tiny difference in logic was enough to hide the bug. It’s fixed now too.
The report revealed the following unknown consistency issues which we had to fix:
The first issue was caused by a client: starting with recent versions the client has idempotency on by default but when the server-side doesn’t support it (we had idempotency behind a feature flag) the client doesn’t complain. We will enable idempotency by default in 21.1.1 so it shouldn't be an issue. Also it's possible to turn the feature flag on for the older versions.The other two issues were related to the transactions; we haven’t chaos tested the abort operation since it’s very similar to commit but even the tiny difference in logic was enough to hide the bug. It’s fixed now too.