Perhaps I don't understand the problem domain, but I don't understand why CephFS is being considered for this task. You're trying to treat your entire set of files across all repos as a single filesystem, but that's an entirely incorrect assumption. The I/O activity on one repo/user does not affect the I/O activity of an entirely different user. Skip the one filesystem idea, shard based on user/location/whatever.
I'd appreciate any comments explaining why I'm wrong, because this doesn't seem to be a productive design to me.
Treating the whole thing as one FS is the current architecture GitLab uses, so is more of an existing constraint than a proposed architecture. To get distributed storage you either need to rewrite GitLab to deal with distributed storage, or run it on another layer that presents an illusion of one big FS (whether that's CephFS or a storage appliance).
Perhaps I don't understand the problem domain, but I don't understand why CephFS is being considered for this task. You're trying to treat your entire set of files across all repos as a single filesystem, but that's an entirely incorrect assumption. The I/O activity on one repo/user does not affect the I/O activity of an entirely different user. Skip the one filesystem idea, shard based on user/location/whatever.
I'd appreciate any comments explaining why I'm wrong, because this doesn't seem to be a productive design to me.