The demo at the top of the homepage (https://reflect.net/) is a lot of fun. While I was watching, every time the puzzle got completed, people would shake their cursor in excitement, like saying, "yay we did it!!"
The first thing I did was start making the word 'FUCK' with the pieces. As soon as others cottoned on and joined in, it was a very heart warming experience.
There's a visual bug where you can hide pieces of the 'e' behind the completed portions of the 'e' - it doesn't work for other letters because they still show the outline.
It's very fun to troll other players with, especially that middle bar piece is easy to hide. I've also seen someone play keep-away with the last piece. Just overall very funny how much you can goof off with a simple puzzle!
I don't think this demo is a good demonstration of the features this library can offer. Once you grab a piece, you seem to lock it for only you to use? This means there are no conflicts to resolve, the only conflict is when two people pick a piece at the same time, from which the only resolution is to give it to the user that picked the piece first.
I'd want a demo that shows a much better example of conflicts occuring.
Are you sure about that - when I landed on that page, folks were moving pieces around but didn't recognize that they had to fill up the ALIVE text - and I was able to easily grab a piece that another user had grabbed before and place it in the text - which triggered a eureka moment among other players and we finished the puzzle...
Seems to run worse on Firefox than on Chromium / Edge / Chrome. Rather choppy, mouse cursor lagging behind and laptop fans spinning up.
Smooth on Chromium and friends though.
People might remember this showing up once or twice before as Replicache. Reflect adds a fully managed, incredibly fast sync server.
The local-first/realtime space is getting busy these days. Replicache/Reflect is well worth checking out for the beautifully simple data model / coding model. I'm very happy with it.
One plus point compared to CRDTs is that you get to decide how you want to deal with conflicts yourself, with simple, sequential code. I'm not up to date with the latest in CRDT-land but I believe it can be complicated to add application specific conflict resolution if the built-in rules don't fit your needs.
> One plus point compared to CRDTs is that you get to decide how you want to deal with conflicts yourself, with simple, sequential code. I'm not up to date with the latest in CRDT-land but I believe it can be complicated to add application specific conflict resolution if the built-in rules don't fit your needs.
I agree wholeheartedly — we took the same approach for PowerSync with a server reconciliation architecture [1] over CRDTs, which is also mentioned elsewhere in the comments here. For applications that have a central server, I think the simplicity of this kind of architecture is very appealing.
Props to Aaron for bringing awareness to and evangelizing server reconciliation.
I worked extensively on this (also implemented it on JS) about 15 years ago, to escape boredom while on my grandma's place. I wish I had open sourced something, but back then I was a young boy with no ulterior motivations :^).
Anyway, I want to comment on this:
>For example, any kind of arithmetic just works:
Of course, it is extremely trivial to set those up as idempotent operations.
>List operations just work:
Nope ... or at least, I'd have to take a closer look at your implementation, consider:
Some sort of array/list that looks like: [1, 2, 3, 4, 5]
* User A deletes the range from 2-4.
* User B deletes the range from 3-5.
* User C inserts (uh-oh) something between 3 and 4.
All updates reach the server at the same time. What is the solution to that? Timestamps? Fallback to LLW? This breaks the transactional model.
Pick any of those updates as "the winner", what do the other users see? How do you communicate this to them?
If your answer is like "we just send them the whole array again", this breaks the transactional model (x2).
Fair point. Will change. What I really meant here was "(Many) list operations just work" (just like above I said "all kinds of things just work".
> All updates reach the server at the same time. What is the solution to that? Timestamps? Fallback to LLW?
There are a number of issues you might be pointing out, and I'm not sure which one you mean.
In general, you can't use list indexes as identifiers for edit/delete in Reflect because they aren't stable. That's why in this example I commented to use the item itself (if atomic) or more likely a stable ID: https://i.imgur.com/IKzmf0q.png
There is also the potential issue of User C's inserts getting deleted. This seems like a problem at first, but in a realtime collaborative context, nothing can fix this. User C's insert can land just before User A or B's delete. In this case, sync would be correct to delete user C's insert, no matter the protocol. But User C might still be sad. From her perspective, she just wrote something and it got deleted. This is the nature of two humans working in the same place at the same time and potentially having different intentions. For these problems, undo and presence indicators go a long way to avoid these problems through human/social mechanisms.
>For these problems, undo and presence indicators go a long way to avoid these problems through human/social mechanisms.
x1,000 to this. This guy builds. That's exactly my takeaway as well from years working on this; provide a solid (by that I mean clearly defined) deterministic algo, and solve the rest w/ UX.
If you actually look at how people use most multiplayer editing tools they tend to avoid conflicts anyway. It’s extremely painful to have two people editing the same sentence in Google Docs let alone more than that even if it sort of works. Collaboration tends to be negotiated so you end up with I-go-you-go, reviews and other patterns forming around the use of the tools. People can also see and resolve conflicts in real-time.
That said this sort of reconciliation can be extremely painful if people are working separately offline for any length of time as arbitrarily resolving many conflicts doesn’t necessarily result in coherent results. But as I understand it that’s also an issue for other ways of solving the issue as well.
Another missing piece is signalling failure. For example two people trying to fill the last slot of a limited resource. Having to infer from the state that comes back whether your operation succeeded or not is not a fun way to write multiplayer code.
This points towards a big challenge that I think gets overlooked way too often. A lot of these sync libraries/frameworks claim to solve for both real-time interaction, and local-first, sync when you reconnect. But the UX needs are wildly different.
For the former you want (as mentioned) good presence info and undo. For the latter you (sometimes) want explicitly flagged conflicts and manual resolution, like we have in git etc.
With the Replicache/Reflect model, you could actually handle both approaches. Might not be easy but the model can support it.
If a client is offline for a while and makes a bunch of edits, would rebasing many thousands of mutations not be a major performance issue? (For example: if complexity is O(n) per edit, as is often the case with string insertions, the process might hang for a long time.)
It seems to me that maybe saying that CRDTs are "good for one particular problem," whereas Reflect can do anything, is a bit misleading? Reflect is eventually consistent, but if you want e.g. string mutations to make sense, you still need to de facto model your text as a CRDT. (Unique IDs per letter, tombstones, etc.) Otherwise, you'll end up with garbled text.
You don’t even need to involve deletes; if all of these happen “simultaneously”:
- client A appends “a” to list L
- client B appends “b” to list L
- client C appends “c” to list L
the server will arbitrarily apply the appends in some order, and then send the result back to the clients. But, each of the clients has applied its append locally and is showing those results until it gets different information from the server, so the next time step might be
- client A thinks L is [“a”]
- client B thinks L is [“b”]
- client C thinks L is [“c”]
- server sends state event to clients saying that L is [“c”,”b”,”a”]
Then when the clients receive the state from the server, I guess they all discard their pending mutations and use the state from the server as their state of the world. But what if each of the three appends results in that client winning the game if their mutation gets applied first? Do all the clients display “you win” for 300ms while waiting for the server’s update?
We have the "game-winning" scenario on the reflect.net homepage right now: https://reflect.net/. It's a puzzle and when the last piece is placed, it shows a celebration and the puzzle resets moments later. But we only want that logic to run once. We can't do this client-side because there are many people and we'd end up having to choose a leader or something. This is the same exact situation as wanting to show in the UI who won (who placed the last piece) and only wanting to do that once.
Reflect has first-class support for this kind of situation by allowing mutators to run special code server-side. I talk about this a bit here:
You modified the operations, I specifically chose a scenario where User C (in my example) introduces an undecidable situation. Appends are trivial to solve.
>If your answer is like "we just send them the whole array again", this breaks the transactional model (x2).
True. I was thinking of a simpler criticism that, if it held, would also make Reflect less useful. aboodman replied in the sibling that (of course) they thought of that, so my criticism doesn't apply.
By your example, 5 was marked as deleted, so it would just be [1, <new element>].
Most likely this would be implemented with a single mutator for each element. But if you implemented a "range delete", then it would depend on what the order of arrival was. Notably, the server has to sequence actions somehow, so there isn't going to be an issue of "all actions arriving at the same time". If C arrived last in that case, the new element would be there. But if C arrived first, you'd just be left with [1].
>Notably, the server has to sequence actions somehow [...]
Yes, that's my whole point, you have to fallback to something like LLW, and then it's over for the transactional model :P.
Edit: Btw, I'm not trying to be the snarky, pessimistic dude. This kind of models are really interesting and I love working with them. I am just trying to illustrate how things could go awry without much complexity involved.
In a collaborative application the server could resolve the three operations in any order. The three users observe the result, realize that they are operating on the same data at the same moment and messed with the state. Then they fix it. Think about editing a document.
If they can't see each other working on the data together they could be surprised but they could eventually fix the data to the desired state.
If they don't check the result or can't check the result, the state won't be OK. However it's not what probably happens in interactive apps like collaborative editing or games.
Congrats! I've been watching this space for a while, having built a couple multiplayer sync systems in the past in private codebases, including a "redux-pubsub" library with rebasing and server canonicity that is (IIUC?) TCR-like. There's a lot to like about this model, and I find the linked article quite clear - thank you for writing and releasing this!
1. You wrote "For example, schema validation and migrations just sort of fall out of the design for free." - very curious to read about what you've found to work well for migrations! I feel like there's a lot of nice stuff you can build here (tracking schema version as part of the doc, pushing migration functions into clients, and then clients can live-update) but I never got the chance to build that.
2. Do you have a recommendation for use-cases that involve substantial shared text editing in a TCR system? I'd usually default to Yjs and Tiptap/Prosemirror here (and am watching Automerge Prosemirror with interest). The best idea I've come up with is running two data stores in parallel: a CRDT doc that is a flat key/value identifying a set of text docs keyed by UUID, and a TCR doc representing the data, which occasionally mentions CRDT text UUIDs.
What does this mean for Replicache development, is the client-side codebase mostly shared and can expect continued updates, or more likely to replace the primary focus for you?
Thank you! Really like Replicache though only used it for hobby projects so far, and enjoyed also the backend side of it (minimal and entirely in supabase stored functions, probably not prod ready). Hope the new approach is a great success for you!
It seems really interesting. The key thing to understand is that there are deep architectural choices made in each of these systems that really influence what you can do. There is "mechanical sympathy" with certain kinds of applications.
ElectricSQL is a distributed database. It's not going to run anywhere near 60fps with tons of users, because it's not running in memory on the server like Reflect/PartyKit/Liveblocks do. OTOH it will allow you to filter/query interact with much more data at the same time than Reflect/PartyKit/Liveblocks do – Reflect has a limit currently of 50MB per room and it's not practical to have dozens of rooms open at one time to get around this.
Systems in the Reflect room/document based model are well suited for applications where you primarily interact with one "document" at a time and you want that to be as realtime as possible. Figma is the canonical example. You want the entire document to move together at 60 FPS, completely fluidly. It's not going to be possible to do this well in the ElectricSQL model IMO. Spreadsheets, presentations and documents are other examples. Systems in the ElectricSQL model are more suited for applications where you are interacting with lots of documents at the same time. Think CRMs, bug trackers, etc.
Of course real applications are messy. Even document editors always have a dashboard where you can at least see all your documents at once. And CRMs of course have a detail view that looks like a document editor. Both types of systems are going to track toward the other to support the needs of real applications.
There's no max enforced. Right now, it will fall apart pretty quick around 25-50, depending on how much work is going on.
For the GA, we plan to implement a scheme that will allow the room to support up to ~100 concurrent active users (actually doing things) and thousands just watching.
This is a pedantic comment, but the terminology is confusing. Games have "players" so their sync systems are called "multiplayer," but regular software has "users" so their sync systems should be called "multiuser." The page reads strangely mixing the terms "user" with "multiplayer."
Maybe. But user is like a consumer. So HN is multiuser, but seems weird to call HN multi-player. Live concurrent interactions feels like something more than multiuser. Just my 2-cents, but multiplayer as in multiple users acting and interacting is a good use of the term multiplayer.
Let me guess: you are not regularly playing multiplayer computer games?
I totally feel hgs3's confusion when reading the discussion here or that website. I always confuse the use cases - "are they now talking about computer games, or are they talking about the broader scope of any kind of software in which multiple users can act concurrently within a shared environment?".
I think this confusion would be much less severe if I hadn't been an avid gamer for decades and thus had a very specific idea of the meaning of the term "multiplayer".
As someone who's casually been eyeing up CRDTs for two years I've wondered continually what the story for authorization is, and this article seems to suggest (in line with my own understanding) that a CRDT library on its own like Y.js can't really handle applications where mutations need to be checked for authorization because it lacks a central authority (i.e., the server) while Reflect assumes that a server will be mediating client interactions. Is this all correct?
"Can't" is a strong word. You can get auth with a CRDT with effort, for example you can put a server in the middle of everything and have the server reverse any changes it sees which are unauthorized.
This ends up being a lot of work to maintain and easy to break as the application gets bigger, and it also defeats some of the benefits of the CRDT in the first place (now the server has to mediate everything and you can't have peer-to-peer sync).
Also if there are side-effects of any actions which require auth which aren't undoable, then that gets more complicated.
If you already have a server in the middle, it's a lot simpler to just use a protocol that allows the server to reject messages in the first place.
CRDTs can still involve a server, just not necessarily in the middle of everything. Consider you could have connections authenticated by the server, for one.
E.g., someone connects peer to peer to you and claims some privilege etc., you could use the server to verify their claim.
Also, I don't think CRDT necessarily implies peer to peer, as you can use a central server for message passing, but keep the CRDT model for resolving the current state on both server and client.
One way to do authorization is to sign each operation/message and then verify the signature to match a public key in an access control list. This also enables CRDTs to work in a peer to peer context.
But as aboodman says in a sibling comment, if there’s a server as an authority, it can simply reject messages from unauthorized clients.
How does one handle an upgrade to mutators? If a client is running old code then the operation will differ to the server. Obvious answer would be to version them independently: `increment_v1`, `increment_v2`, but wondering if there is a better answer?
At the moment there is no better answer than do not remove your old mutators until you know there are no more clients out there that might have pending changes.
Right now that window is pretty small because we haven't turned on persistence yet.
Yes they are in the same space. The key difference is in how opinionated each is.
PartyKit is extremely unopinionated. It's essentially lightweight javascript server, that launches fast and autoscales (I don't say this as as a bad thing, it's a useful primitive). Most people seem to run yjs in PartyKit, but you can also run automerge or even Replicache – my company's other project.
Reflect is entirely focused on providing the best possible multiplayer experience. We make a lot of choices up and down the stack to tightly integrate everything so that multiplayer just works and you can focus on building your app.
Thanks for taking the time to answer! I've been trying to keep up with the rise of more collaborative tooling, so I appreciate the comparison. Will have to try out reflect sometime on a future project :)
For reference, this strategy is called "deterministic lockstep" in the gamedev world, and is used especially often in games which have many entities which need their state synced. The canonical example is rtses, which pretty much all use it.
That is an algorithm for peer-to-peer games, where each peer waits for the input from all other players before advancing the game simulation. The deterministic part is because players share inputs (not simulation results) AND the simulation is deterministic for the same inputs. The lockstep part is because all clients advance at a coordinated pace. The Age of Empires series use this approach, and that’s why units don’t move immediately when you click. Starcraft uses this too, but it has some tricks to smooth the gameplay experience. Both are peer-to-peer with no single “server”.
In the case of Reflect, we have a server-authorative simulation (not peer-to-peer). Clients send their inputs to the server, but they do not wait for the result, they instead predict the result locally without confirmation from the server. The server also rolls back time and then replays inputs to compensate for individual client latency. And the client corrects/reconciles their local simulation once the server sends a simulation result that included one of their inputs.
The keywords for this algorithm are:
- Server-authoritative
- Predicted
- Lag compensated (simulation rollback / server rewind / input replay)
I’m not sure if Reflect has it, but client-side interpolation is also a common feature in FPS games, where upon receiving a world update, the client will tween entities to their new positions/rotations over a fixed interval (such as 0.1s). This allows you to send updates to clients at only 10Hz but have entities move smoothly (without needing the client to locally simulate the physics of every entity).
There is only one authority, the server, so determinism isn’t super important. However, it is nice to have for client predictions to reduce the occurrences of mispredictions. Mispredictions occur when the server state did not advance the way the client predicted. These can happen when:
(1) Another client’s input changed the world state in an important way,
(2) The simulation is not deterministic (e.g. the random number generation is not synced).
In FPS games, (1) is impossible to eliminate. These mispredictions are often smoothed using interpolation. (2) should be minimised, but may actually be desirable. For example, Counter Strike does not sync the RNG for bullet spread randomness, to prevent “nospread” cheat programs from predicting where each bullet will go and instantly adjusting the player’s aim so the bullet lines up perfectly.
Yes, I know there's some differences in what you're doing here from classic deterministic lockstep, but the fundamental concept of syncing a queue of inputs from which the game state can be derived, instead of syncing the game state itself is the same, which is why I mentioned it.
Also, while deterministic lockstep can be used peer to peer, it is not fundamentally a peer to peer model. Factorio for example, uses server authoritative deterministic lockstep, and has lag compensation (but only for certain actions like movement).
All multiplayer games sync a queue of messages to derive game state, that’s inherent in networking.
Deterministic lockstep means that the speed of the simulation is dependent on the slowest member as everyone waits to get all the inputs before advancing a frame. You’ll get literally pauses and jank if someone is on a poor connection. That’s not the case here.
The difference with this model of networking is what is synced in the lockstep case the simulation being deterministic means you only need to send inputs to keep the game in sync so you vastly reduce network traffic. In this case it’s not necessarily deterministic and the server sends back the new state to the clients after determining what it is as it’s the authority on what that state is. Clients can predict ahead by making the change locally but must correct mispredictions. In the deterministic case there can be no mispredictions and it’s a pain in the butt to ensure that your game is actually deterministic and resolving sync issues is a real pain.
Factorio sounds like it uses a deterministic model that doesn’t wait to run in lockstep and instead reconciles errors that causes. The typical way of doing that is to rollback to a prior state where the simulation diverged due to missing inputs and redo the following ticks. That’s probably prohibitively expensive given the scale of Factorio versus something simpler like a fighting game so I assume there is a hybrid model and some extra info is sent by an authoritative source that can be checked locally.
The whole idea of DL as I understand it is to take advantage of determinism to reduce bandwidth. But Reflect doesn't do this.
I think this is a feature for developers. It's nice to be able to have the server just do whatever it wants for whatever reason it wants. It also means you can use normal JS, and don't need to try and enforce determinism somehow.
But in the end I felt like since we're going to continue to evolve this and have made specific choices, it made sense to give our version its own name so that we can refer to it.
Besides trystero which is great but depends on things I cannot get into our company, anything open source? I cannot get this into our company either as it's vital infra and as such needs to be replaceable with open source at will. We have that as a rule. The only things that are exceptions are necessary evil; payments & cloudflare (bot-fight specifically).
While I do love the simplicity of this approach on the surface. Some of the complexity gets brushed under the rug.
Firstly is rebasing these operations in this style of system takes a bigger hit on performance than I expected in my experience. Sure, managing counters and doing integer math can consistently run at 120FPS but more complex operations can really bog down the system when a lot of users are interacting with it once.
The second challenge is that when operations fail (e.g. for authorization) undoing that operation isn't always straightforward. You either pop the operation and replay ops from a earlier snapshot (which necessitates storing all ops in order), writing an inverse operation manually, or having the server send the entire state back to rebase your unconfirmed ops on which can also end being expensive.
In general, the challenges of this approach are true for all operation-based CRDTs too. State-based approaches are a good deal simpler just tend to over-send data over the network.
Hey Matlin, it's true that this approach is harder to implement, but that's a one-time cost, and it's one we take on for our users and they don't have to worry about.
We are targeting 100 concurrent users at 120 FPS. Not completely there, but it is certainly achievable, and we will scale back framerate when we need to.
So are you using fps management just for scaling purposes or also as a way of ensuring fairness? The reason I ask is because games like Forza have had to resort to setting all online players' frame rates to be the same regardless of their machine's capabilities due to the way the in-game physics work. PC players on high end hardware would consistently get better lap times than Xbox users because of it.
Maybe this is a bit out of scope of what the goal is for this project but just thought I would bring it up.
Huh! That's interesting. I don't think this applies to the type of collaborative apps (ie figma) we are targeting, but it's fascinating to know. Thanks!
Can you share how y'all made it fast? Obviously it can be made fast because multiplayer online games are living proof but I'm curious what it takes to get there or what's missing in my mental model.
What's your opinion on using this library as a base for (eg) a simplified "world-of-warcraft" clone?
I've done a lot of thinking (but not coding) around "simply replicating state" being the right answer to interconnected apps.
The degenerate case is to model everything as a git transaction log, but be able to run it "faster" than really using git. It's very intriguing to hear your discussion of "branching, sequentialization, etc..." because that seems like the right way to go. ( edit: and implicit _physics_ vs. state transfer in some cases... it's cheaper to send nuke@[x,y,z] instead of the literal calculations and changes to state that single action may imply ... physics in games maybe being gravity and ballistics, but 'business-physics' might be update/select/modify, etc. )
...eg: Unguaranteed, Guaranteed, Guaranteed Order, Most Recent State, Guaranteed Quickest (and each client has it's own "camera" / perspective against the "scene", where +180 might be "need it now b/c I'm looking at it", but -180 might be "meh, need close-to-the-latest-state before I turn around and look at it").
I'm rooting for you b/c "multiplayer + physics" seems like the right answer (eg: the ghost of couchdb), but no one seems to have (yet) cracked the semantic code as to how to describe "as fast as local, but synchronized with $THESE tradeoffs and $EXCEPTIONS".
Wow this looks incredible. I had the most fun I've had on a computer in a long time this last weekend using Partykit.io to build https://is-sf-back.cconeill.partykit.dev/ and this looks so much smoother.
When will it finally be possible to use UDP protocol within browsers?
TCP got way too big overhead. A multiplayer engine in which you send 30 messages per second for 128 users is what i need. Not possible with websockets withput having super computers.
WebRTC makes this possible, but server implementations are quite difficult to set up. I recommend you check out Pion (Golang) as a good implementation that's also easy to deploy and integrate with Node (presumably your game server is implemented in JS).
This isn’t as true anymore but it’s definitely best practice to build on UDP. Games built on UDP normally have several different policies including the ability to send messages that guarantee delivery even if packets get lost.
Has anyone done the cost + risk assessment of building a for-profit product on top of this? Would love to know, as I am working on a web IDE with collaboration. There’s also the matter of obscuring client data from 3rd and even 1st party.
If what you're building on top of leaks into other parts of your codebase, can't be isolated and can't easily be swapped out, I'd say avoid building a product on top of it.
Don’t call it multiplayer, call it multiuser or concurrent user. Player assumes it’s a game. It’s not a game. It’s very very real. We are the agents. Cool project. Collaboration is hard. Synchronicity even harder. This makes steel become butter.
I think multiplayer is a good name for a different reason. I think it’s a clear and accessible metaphor for how people will interact, i.e. by concurrently working toward a shared goal with a coherent shared state.
I don’t think it trivializes the goal or the work being done, it just distinguishes it from parallel non-coordinated use.
Also, players are not just for games; the Bard had some thoughts on that. ;)
I'm curious what trade-offs you looked at regarding sending state vs mutations from the server back to clients. As the server is authoritative, I imagine it could also send mutations to the clients in the order in which the server has determined parallel operations need to be resolved (and adjust mutations as needed to resolve conflicts or permissions issues). Clients would then need to rebase any pending local mutations, so that would involve more logic vs the data sent for mutations most likely being smaller than the "full" state of certain objects being updated?
Very allagmatic. Instead of storing & transferring structures transfer operations.
Iw as wondering what kind of checkingpoint or snapshotting there might be. Such that one doesn't have to "replay-the-world" if something goes awry (localstack persistence did/l (does?) this, storing state by just recording/replaying API calls). It sort of seems like having a globally accepted state is enough & makes sense, but it still seems like work to rollback & do again, still seems to require snapshotting.
Something like this feels like a 100% fit for having immutable data structures. Where you can snapshot the world very very quickly/at low cost.
We'll post something soon comparing all of the options (in our opinion). But as I understand it:
- Supabase Broadcast: one client sends every other client a one-time message
- Supabase Presence: one client tells all other clients that its own state has changed. This is similar to above, except that server keeps track of all the last-state for each client so that new members of the channel can get up to date.
- Supabase Realtime: server tells all clients when server state changes (so you will not get optimistic client-side changes this way)
Supabase’s real-time stuff lets various parts of your system send messages about changes to each other. But when it comes to realtime editing, that’s the easy part!
Reflect is a much higher level abstraction that handles the hard parts:
- maintaining a cache on the device (this alone is annoying hard to get right)
- queuing updates to send to the server reliably
- handling conflicts between different users while preserving the intent of each user, by rebasing histories locally, and choosing the winner on the server
One consequence of this approach is that your mutators will need to be versioned, and you can pretty much never delete or change them once they're live. (Please correct me if I'm wrong).
This looks great! I'm curious what the recommended use cases are for something like this. Is it worth adding the additional complexity for complex crud-y saas? I know Linear is built on this local-first model but from seeing their presentations on the challenges they've had scaling it, I'm left wondering if it makes sense for something like a full-featured CRM with hundreds of tables. Is Reflect/Replicache suited for that kind of use case?
My 2c as a game dev is that you really want to know the cases where you need real-time collaboration. Otherwise the still collaborative but slower updating techniques people normally use are perfectly fine. Even in something like an MMO parts of the game will use real-time game servers and others might just use a REST API.
The other side of it is to make sure that real-time collab is something users actually want and use. There’s a lot of froth at the moment about real-time but IMO and IME it has a much smaller set of useful cases than most people actually think. But where it is useful, it is a step up for collaboration.
For example with Google Docs I find myself and our broader team mostly solo write but collaboratively review and edit. Where we collaboratively write its carefully structured so people aren’t stomping on each other and often done with someone acting as a mediator.
The main value prop of CRDTs is they can be used without a central server (for example, in p2p networking, or when offline.) If you're choosing between a CRDT or something else like this for your centralized server, you're probably not thinking about things properly.
Edit: I have no idea why this was downvoted, it's true
> If you're choosing between a CRDT or something else like this for your centralized server, you're probably not thinking about things properly.
I think that’s the point of comparing this to CRDTs - to show that there may be a better tool for the job if you’ve been considering CRDT because it’s seen some recent popular exposure
Frameworks for hard problems like this seem like such a good public service. Is there much low-hanging fruit in this area -- common concurrency problems where people are likely to roll their own solutions because it's hard to find a preexisting framework?
This is a hosted service that makes it much easier for you add multiplayer real-time collaboration (as seen in Google Docs and Figma) to your own web application.
Not at all any expert in this space, so maybe a dumb question, but how do you model plain text data for something like a collaborative markdown editor with things like this or CRDTs?
What’s the transport for this? Websockets? I have a hard time imagining 120 messages for 100 clients over websocket in a CF Durable Object, but maybe I’m just completely misimagining.
It's websockets, and yes we do make use of Durable Objects. But we do extensive batching and buffering on top of the raw platform. You can't literally send 120 messages / second to and from 100 concurrent clients. Actually the math tops out at about 8 :).
There are a variety of relevant limitations. Some of them are listed here: developers.cloudflare.com/durable-objects/platform/limits/
For this particular problem, it just comes down to CPU time. At the bottom of the stack ws.send() is a syscall and it takes significant time. You can only do so many of these calls per second. We measured it at about 8k/sec a year ago.
With a conservative limit of 2k calls to ws.send() (so that we can stay about about 50% utilization and only use half of the time for calling send), this implies 6 clients at 60 FPS using a naive approach (6 * 6 * 60 = ~2k).
It's not really a DO problem, it's just that doing n^2 messages can't really work in any platform.
This technique is pretty similar to Operational Transformations, except that it requires an authority for serialization and must re-apply any state which is out of sync.
It's not the same thing. OT works by having a discrete set of operations that are known to the system so that they can be reordered properly server-side.
The critical difference is that Reflect doesn't need to know the operations up front. That is what allows developers to provide their own operations, or as we call them "mutators".
I'm using Replicache in a big project - this simple fact is a big deal. It's basically the same coding model that all the front-end frameworks have gravitated to (central store, unidirectional dataflow, mutators separate from data access) but it all stays in sync across the distributed system.
Depends what you mean by very large. Reflect can store up to about 50MB/room. For a PNG, that's "fairly large" I guess. We plan to increase this limit in the future.
You'd have to split up the image into chunks of about 1KB each, this would be fairly easy to do by pixels. It would be a fun project to try.
TCR is just a small generalization of what the game industry has been doing for awhile. So in that sense, we are not at all the first.
But I'm honestly not sure why it's not been done on the web yet. To me, it's a really elegant approach. It is harder to implement in a general way (like as a library) because you need to run code on the server. And as others have pointed out, it's harder to make fast.
Perhaps multiplayer is just new enough to the web that there hasn't been time for these approaches to evolve.
I mean, React was also inspired by game techniques that at that time were decades old.
I strongly endorse Trystero (https://github.com/dmotz/trystero) for enabling P2P communication in web apps. It’s open source and leverages public infrastructure for matchmaking.
It looks very slick, and I'd probably be interested in this if I were building a centralized local-first app. But I think this serves a fundamentally different use case vs. CRDTs. Reflect says "Your mutation code runs server-side and is authoritative" (emphasis theirs).
- If you can have an authoritative server, CRDTs come with unnecessary restrictions and overhead. Reflect presumably gains a lot of efficiency by loosening those constraints.
- Conversely, if you want clients to collaborate without a central server, you can't use a service like Reflect, since there's no server to run your conflict resolution logic.
Yeah, it's interesting to me how from a user's point of view both technologies do something similar, but from an architecture point of view the use cases are mostly disjoint. I totally agree that something like Reflect makes more sense for most apps people build today.
Anyway, congrats on the launch! Reflect looks great. I'm really excited to see the building blocks emerge for local-first software.
What makes Erlang (and Elixir) great isn't just the actor model, it's the language-level commitment to it. Things that would mess with concurrency (say, race conditions on the reading and modification of a global variable) are simply unexpressible in BEAM languages.
