Gitaly is still a centralized system. It shards the git repositories over multiple git servers in one datacenter.
We looked into Git Ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ke... for a distributed system based on hooks. But this requires a sync per repository (very slow with 100k+ repos per server) and it is no longer actively being worked on by Google.
There are plenty of off-the-shelf consensus algorithms/services you could use in the git-hooks that don't have the git-ketch limitation. Zookeeper, etcd, consule, redis, etc. It really just matters your criteria and requirements, they all of trade-offs but I'm sure one fits.
To be frank I think your time is better spent looking into why one of these consensus packages isn't the right call for your needs. Since they could easily plug into the git-hooks & git-shell commands, it could avoid a massive investment of time from your team and you.
Next time you think about distributing git, you should take a look at the combination of jgit and a distributed key value store.
A little while ago I spent a night and I was able to get a fully distributed git http server up and running. I used HBase and was able to get it most things working. That allows the only contention to be around refs. Everything else is well distributed and spread across as many machines as you need. Caching is handled on the key value store side, and the http server side. I didn't get to git gc work though.
> That allows the only contention to be around refs.
This is solved without modifying the git client using a distributed/shared lock wrapping the git-shell & git-hooks (pre-receive) . If you were to modify the git server behavior to say use epaxos, one could provide even better latency guaranties https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf .
I'm not sure about the GC operation though, did you use the DfsGarbageCollector that is part of Jgit? I thought that was provided as part of the DFS interface.
We looked into Git Ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ke... for a distributed system based on hooks. But this requires a sync per repository (very slow with 100k+ repos per server) and it is no longer actively being worked on by Google.