This looks like someone is building their own filebased database-system, in typescript, while still using mature features of database-servers. So instead of trusting the optimized, regularly maintained and battletested solution, they build something by themselves. This smells ugly, like something that will scale poor in performance, and will have security and tooling-problems.
Simplification of installation seems not like a good enough reason to trust your whole backend on this. Installing and maintaining a database-server is not that hard today. This is well established and documented, unlike this. But I also don't know enough about this app, maybe this is just one of several options, meant for a specific usecase? Using this in a standalone desktop-app would make sense, while still offering a mature sql-backend available for server-installations.
Each users' data is naturally partitioned at the atproto repository level, so this is the sweet spot for per-user SQLite databases. It would make total sense for a PDS instance to have just a single user on it, and in fact that is likely for many self-hosters. It's also worth noting that the PDS software already had SQLite support, which made this change somewhat easier.
There are legitimte trade-offs to this kind of a system but it comes out way ahead in this case, and it's not as wild as it may seem to those not familiar with the power of SQLite.
A major consideration is that we're planning to run at least 100+ instances, which would require operating 100+ high availability (primary+replica) Postgres clusters. This would be a huge amount of operational and financial overhead.
We chose this route because it is better for us as a small team, with relatively limited resources. But it also has the property of being much easier for self-hosters, of which we hope there will be many.
> Using SQLite is most certainly not "building their own filebased database-system"
[..] Each user has their own SQLite file [..]
[..] We also introduce 3 separate SQLite databases for managing service state [..]
This doesn't use SQLite for the database-managment, but for the individual "document". The database-managment itself is handled in the application-server. You jiggle around files and poke wherever it matches, this is by a classical filebased database-managment-system.
> It would make total sense for a PDS instance to have a single user, and in fact that is likely for many self-hosters.
Sure, if it's just a low-user-instance, the performance is not much of a deal. But from my impression here, this is also the code Bluesky uses for everything else, from low to massively high user-instances. And then I want to see how RAM holds up, when you have 10k+ user-databases open at the same time on one instance.
> There are trade-offs to this kind of a system but it comes out way ahead in this case.
Which is why I want to see some actual numbers and solid explanations going more into details then the gossip in the comments here.
> A major consideration is that we're planning to run at least 100+ instances, which would require operating 100+ high availability (primary+replica) Postgres clusters.
Are those independent instances, or just 100+ instances servers from the same company on different locations? But I don't see how this can replace a whole postgres-cluster without removing significant functionality. I mean sqlite does not have good replication on it's own AFAIK, so as you seem to still use replication, you just replace it with another solution? Which also means you remove the same options for anyone else, and forces them to use your solutions?. I don't see how this will be beneficial for self-hosters.
I'm not a professional coder, only side-projects. Never formally taught. I looked at the solution and thought it kinda sounds like something I'd come up with. Like when I didn't know how to use data tables and would hold data in an array of arrays to form the rows and columns. Somewhat clever, "works", but would probably make my professional coder friends vomit if I explained it to them.
Simplification of installation seems not like a good enough reason to trust your whole backend on this. Installing and maintaining a database-server is not that hard today. This is well established and documented, unlike this. But I also don't know enough about this app, maybe this is just one of several options, meant for a specific usecase? Using this in a standalone desktop-app would make sense, while still offering a mature sql-backend available for server-installations.