Yes, quite. Unless my function has no side-effects, idempotency will always be my responsibility. This feature is a lie and an outright hazard. It should not be called idempotency and it should have a big bold warning stating that the function may be called more than once.

I'd appreciate it if you could point out how the function can be called more than once. Here's the source code for the orchestrator [1] and a demonstration of it [2].

If there's an error in the source code or the copy, I'll happily retract it.

[1] https://github.com/differentialhq/differential/blob/236ffc53...

[2] https://github.com/differentialhq/differential/blob/236ffc53...

Without looking at the code, I can tell you that this will reproduce one of two things:

1. Invoke function that has two side effects

2. Function does first side effect

3. Pull the computer's plug out of the wall

A framework can only do one of two things:

A. Invoke the function again

B. Not invoke the function again

If you choose A, you have at least once processing

If you choose B, you have at most once processing

Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one.

I believe you mention exactly once processing in another comment. It's this. The massive lie that many of these types of frameworks tell is that they can manage idempotence for you. Aside from functions that are side effect free (that is, pure, that is referentially transparent, that is can be made mathematically idempotent) you cannot possibly make them idempotent. It's the two generals problem.

So, I believe what you have implemented is idempotence at the sender with the reservation pattern. This is not the same as idempotence at the processor which is literally the only thing that matters. Idempotence at the sender is simply a performance (or storage, in the case of event sourcing) optimization.

Please correct me if I'm wrong on any of this, but if I am, then clearly my understanding of the two generals problem and distributed systems are wrong and I'm going to have to do some significant rethinking of our system's architecture.

I don't think you're wrong about any of this. I just feel like we're talking over each other. You clearly demonstrate an understanding of the domain, so I'm happy to engage. After all, if we fail, we want to fail fast.

Maybe we can focus the conversation on the scenario you mentioned above.

> Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one.

So, what I'm telling is, yes - the first side-effect (or any side-effect) that you put through the system can be made idempotent through the same tooling that makes the caller idempotent.

  func foo(n1) {
    return networkCall(n1)
  }

  func bar(n2) {
    return networkCall(n2)
  }

  func foobar(n1) {
    n2 = foo(n1)
    n3 = bar(n2)
  }

  func main() {
    foobar(42)
  }

Given the above, the framework allows you to wrap foo, and bar all independently in higher order functions which will not issue a duplicative function calls for the same idempotency key. Therefore, you can call foobar repeatedly, and get the same result.

Understood, and perhaps the difference here is more about what I said in my other comment. I don't see this as particularly useful, and I see it as deceptive. What matters is that when we intend to do something is that it happens. We actually want retries. We want exactly-once processing with at-least-once handling. That's the only way that I know of to build resilient systems.

Our back end components literally crash when they fail. No other processing is done. They crash and if there is a bug they will keep crashing. Because our back end components have autonomy, this results in a very narrow service disruption. We fix the issue, identify the root cause and eliminate it. As a result, we have extremely resilient systems that, even in the face of 3rd party downtimes, we can know that every command that gets submitted will be effected. Dead letter queues or anything of the sort are anathema. We try until we succeed. Idempotence is considered in every single handler all the way from the inside to the outside as it must be, mathematically.

I see elsewhere you mention that you use at most once processing. So, I was possibly mistaken that the function can be called more than once assuming you did this right (cleared the job before even invoking the job). My primary point still stands: this is not idempotent, at least not in the way that matters. This is single-try processing. If all you are doing is returning a synchronous error to the user so that they can retry (and again, every other operation leading up to the one that failed to process is actually idempotent) then this would not be catastrophic. The issue is when people start to think that this actually is idempotence, i.e., they attempt to reuse an idempotence key across requests when the first one straight up failed (or not, who knows).

In other words, don't use the idempotence helper if you care about the thing happening. If it's optional, sure, go for it. You could rename it to "tryOnce", or "maybeRun" rather than "idempotence" and it'd be more accurate / less misleading.

> You could rename it to "tryOnce", or "maybeRun" rather than "idempotence" and it'd be more accurate / less misleading.

This is valuable feedback. Thank you.