Hacker News new | past | comments | ask | show | jobs | submit login

Look into the new Graylog (v1.0). It was recently released, and they use Kafka internally now for buffering. We're sending thousands of msgs per second to Graylog without issue.



According to their documentation, the data storage for log events is Elasticsearch and if the Elasticsearch data is lost then the logs are gone. Consideribg that ES is not a database and may lose data this sounds a bit scary to me.


My understanding is that appending only is fine. So if your use case is logging elasticsearch is totatly fine. You probably can't compare it to the robustness of MySQL & PostreSQL. But most NoSQL are not known to be that robust either.


You can solve this in one of two ways:

1. Whatever mechanisim you're using to send data to Graylog, you send that data to S3 as well. You can then reload Graylog at anytime with S3 data.

2. Backup your Elasticsearch nodes to S3

I should've mentioned I run this in AWS. Sorry about that!


read up here... https://www.elastic.co/blog/resiliency-elasticsearch/

they had a major data corruption bug last year but have taken measures to correct it. I asked if elasticsearch could be a source of truth at elasticon and they didn't say yes, but they indicated that you "could do it" and it is a goal




Consider applying for YC's W25 batch! Applications are open till Nov 12.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: