A bit confused why you are trying to roll your own storage solution. Ceph is great for a lot of applications, but I am not sure it really fits the bill for what you describe. Especially when you indicate you are going to use spinning disk behind it. Have you looked at any of the storage arrays on the market? Your TCO is likely to be much lower and your performance/resiliency much higher if you buy something with $100's of millions of R&D behind it rather than all the hours and costs of rolling your own. Not saying its impossible to make it work, but it just sounds like something that will be a PITA going forward.
I think a storage appliance (NetApp, etc.) makes a lot of sense in the short term. The TCO is lower since we'll spend a lot of time making Ceph work.
In the longer term the storage appliance will lock us in and will get very expensive. I've heard pretty bad stories of companies betting on it. Especially with many small files like us (IOPS heavy).
And one goal of GitLab.com is to gain experience that we can reuse at our customers. Most of our customers use a storage appliance now but are interested in switching to something open source.
