Hacker News new | past | comments | ask | show | jobs | submit login

Can you elaborate more? Is this related to S3 uptime SLA or is there a different reason?



I believe it is different to the S3 uptime SLA.

You can measure the impact and potentially automate recovery of missed events by:

1) Keep a track of events published.

2) Generate an S3 inventory daily.

3) Compare events received to objects listed in inventory.

You rarely end up with fewer events than objects, but it does occur.

I've personally only observed this with SNS target, but due to it being a problem with S3, I believe Lambda can fall afoul of this too.


I wouldn’t trust S3 events to lambda. Sure lambda supports retries and a dead letter queue but you can’t reprocess the data.

A much more resilient approach would be:

S3 event -> SNS Topic -> SQS Queue -> lambda.

and set up a dead letter queue for the SQS queue.

It doesn’t help with the reliability of S3 events (and I’ve never seen that happen), but it does help if their is an error running your lambda.

Move the S3 object after processing it. As long as you move it to a bucket in the same region, there aren’t any charges.

Then if you are really paranoid, you can have a timed lambda that checks the source S3 bucket periodically and manually sends SNS messages to the same topic to force processing.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: