>Limited Retention: Binance was retaining most logs for only a few days. Their goal was to extend this to months, requiring the storage and management of 100 PB of logs, which was prohibitively expensive and complex with their Elasticsearch setup.
Just to give some perspective. The Internet Archive, as of January 2024, attests to have stored ~ 99 petabytes of data.
Can someone from Binance/quickwit comment on their use case that needed log retention for months? I have rarely seen users try to access actionable _operations_ log data beyond 30 days.
I wonder how much $$ can they save more by leveraging tiered storage and engs being mindful of logging.
Government regulators take their time and may not investigate or alert firms to identified theft, vulnerability, criminal or sanctioned country user trails for months. However, that does not protect those companies from liability. There is recent pressure and targeted prosecution from the US on Binance and CZ along this angle. They've been burned on US users getting into their international exchange, so keeping longer forensic logs helps surveil, identify, and restrict Americans better (as well as the bad guys they're not supposed to interact with).
Just to give some perspective. The Internet Archive, as of January 2024, attests to have stored ~ 99 petabytes of data.
Can someone from Binance/quickwit comment on their use case that needed log retention for months? I have rarely seen users try to access actionable _operations_ log data beyond 30 days.
I wonder how much $$ can they save more by leveraging tiered storage and engs being mindful of logging.