The advice I give to people is "think about logging like a feature with customers, just like any other feature." If you think about logging this way, you ideally put yourself in the position of your logs' "customer"- a bleary-eyed teammate who just got paged out of bed at 2am, or a security guy who isn't intimately familiar with your code. Providing extra context for those users is really kind, as they don't have a "secret decoder ring".
Since the article discusses it, I also highly recommend AWS CloudWatch logs. It's super easy to log to CloudWatch from Lambda with the Python logger class. It's a little harder from EC2, but not really that difficult. The advantage is that you don't need to SSH somewhere and navigate to a file and then locate the interesting stuff in that file - You just keep the CloudWatch console open (or keep a link to it handy) and the latest stuff is on top with easy filtering.
Cloudwatch is cheap (even free) for small environments and has very small operational overhead.
The cost, however, can easily get out of hand and AWS does not have per resource billing enabled by default. So it might not be obvious which Log Group is responsible for for which charge.
I also think its console is way inferior than something like Grafana for searching and visualisation.
The console (and its appearance in linked services' 'Logs' tab or similar) is infuriating IMO when your model/what you care about isn't the single instance/invocation that the streams are probably set up as, and you just want all the logs from (all occurrences of, but you're not thinking in those terms) 'thing'.
E.g. a Lambda serving API requests, and you just want 'the logs'.
Other than searching (for something specific) you can only do it (afaik?) with third-party CLI tools, or (found this as I write) v2 (as in, the separate v2 project/binary/distribution methods, not just being up to date :eyeroll:) of the awscli. Short of rearranging your group/stream naming anyway.
Whereas what I really want is just to see all of the logs in the group, with the stream they're from as a filterable property.
(And maybe keep the hyper-specific-ness as a new 'publisher' (what unique thing put the event in the stream) property or something?)
I hear you, and when I did some research on this in the past, going to get logs was the most common reason for engineers to SSH to production hosts.
As a security guy, I hate SSH to production (the whole "cattle, not pets" thing). In my last company we had an internal tool to federate you to the AWS console. We had runbooks in a wiki, and had links literally to the logs for a particular component/service/region - the link would federate you to the right account and take you directly to the target log in cloudwatch logs in the appropriate region. Safer and easier than ssh-to-prod.
> It's super easy to log to CloudWatch from Lambda with the Python logger class
Or even just print(): You've got a unique log group for a lambda, CloudWatch logs autonatically timestamps entries, and stdout & stderr from a lambda both go to the current CloudWatch log stream in the lambda’s CloudWatch logs group.
My experience with Python in Lambda was that my functions ran significantly faster when I used logger rather than print for logging. I agree that Lambda captures stdout to CloudWatch, but my experience was that logger was better (and more intentional).
ed: clarified that this relates to AWS Lambda only
The advice I give to people is "think about logging like a feature with customers, just like any other feature." If you think about logging this way, you ideally put yourself in the position of your logs' "customer"- a bleary-eyed teammate who just got paged out of bed at 2am, or a security guy who isn't intimately familiar with your code. Providing extra context for those users is really kind, as they don't have a "secret decoder ring".
Since the article discusses it, I also highly recommend AWS CloudWatch logs. It's super easy to log to CloudWatch from Lambda with the Python logger class. It's a little harder from EC2, but not really that difficult. The advantage is that you don't need to SSH somewhere and navigate to a file and then locate the interesting stuff in that file - You just keep the CloudWatch console open (or keep a link to it handy) and the latest stuff is on top with easy filtering.