Hacker News new | past | comments | ask | show | jobs | submit login

The unified log pattern - and event sourcing based on that - is a very powerful pattern.

We are doing some work at my job right now inspired by this approach (in no small part, that LinkedIn article). I hope we can open-source some of this tech, or at least start blogging about the concepts as nobody really talks about it, and the mechanics are a little tricky to get right.

We currently think the best approach is to bring ideas from DDD (specifically the idea of a bounded context and a domain), a unified log which we call the GEL (for Global Event Log), domain-specific event logs, event-sourcing with CQRS and then some quite lightweight means to produce "projections" (different structural representations of data).

We therefore have a few pieces that get names and become easy to reason about: every event goes to the GEL. Domains have systems that listen to the GEL for events they are interested in and map those events into commands on a command service. Clients can issue commands directly, as well. Commands alter projections and produce events back to the GEL. Clients and domains can query a domain's query service.

It's not really rocket science, these ideas are as old as the oldest mainframe you've ever heard of, but it affords incredible flexibility and unrivalled performance.

We're using Kinesis for our GEL that has a read performance issue, but not a big enough issue for us yet to look at alternatives - the zero-admin cost is attractive to us right now.

For me, now, MVC and CRUD style applications look like really odd anti-patterns. I'm amazed we got as far as we have with them.




DDD, CQRS, Event Sourcing has created some of the most overengineered applications I have ever seen. Some that have been horrible to maintain some years later.

I'm sure those patterns are used in great applications but personally I would really think hard if I need the functionality before I went there.


Yes ... it's a very powerful pattern and Redis streams look promising; building event sourced systems is still perceived as difficult, but I think a lot of that is because people are picking overly heavy, complicated plumbing for it when it's not really needed to get started. This has great potential to help with that.


> For me, now, MVC and CRUD style applications look like really odd anti-patterns. I'm amazed we got as far as we have with them.

MVC I can understand but how does the unified log pattern/event sourcing help with (or replace) the datastore behind CRUD style applications?


> help with

Not gp, but maybe because without book-keeping you lose information in a CRUD style app? The append-only (until/unless you truncate it) log gives you a lot of durability and insight into the system via replaying events from a snapshot + log combination. At least, that's my understanding


I've always seen the CRUD aspect of an application more of a reference to the aspects of the public facing API, rather than the architecture or implementation of the service in an application.


Paul are you based in London? I've been thinking about dusting off my Unified Log London meetup (https://www.meetup.com/unified-log-london/) and it sounds like you would have a ton to talk about there. Email address is in my profile!


Ill buy you both a drink if you make this happen.


That sounds great, I'd definitely come to the meetup!


Will fly to London for this


I'm still pretty new to event sourcing but when I looked at Kinesis it seemed less than ideal for ES due to it's finite retention (max 168hrs). How do you get around that?


My company uses Kinesis for stream consumption, but everything is stored in both Redshift and Glacier (we expire redshift data after a year or two but Glacier is kept forever). This is our canonical data store.

Smaller subsets of events are kept in Redshift/Postgres/MongoDB databases, depending on the purpose and how they'll be queried. These keep anywhere from 48 hours to 6 months of data depending on the purpose, sometimes it's just the raw events filtered in a certain way, and sometimes a projection based on those events. Either way, if we catch a bug in our logic we can re-run it for the past year of data or whatever.


Kinesis is designed to be consumed, it's not a long-term data store. It's good for quickly shuffling data with some guarantees, and then you stream/batch process it in semi-realtime after the fact.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: