Onboard for what? Using what tools? What is the end goal you’re trying to achieve?
To me, this seems like a trivially simple pipeline for processing data, and in what is likely to be one of the most expensive methods possible, given the almost complete reliance on Lambda to do ETL on every single piece of data going through, and then storing all that data in S3, which you then query with Athena.
I don’t see how you could use this pipeline to process any non-trivial amount of data, and do so for anything remotely resembling a reasonable cost.
But maybe I’m missing something obvious?
Have you actually done the math for what this would cost on a per megabyte or per minute basis?
Have you run real world amounts of data through this system, and again for what purpose?
Onboard for what? Using what tools? What is the end goal you’re trying to achieve?
To me, this seems like a trivially simple pipeline for processing data, and in what is likely to be one of the most expensive methods possible, given the almost complete reliance on Lambda to do ETL on every single piece of data going through, and then storing all that data in S3, which you then query with Athena.
I don’t see how you could use this pipeline to process any non-trivial amount of data, and do so for anything remotely resembling a reasonable cost.
But maybe I’m missing something obvious?
Have you actually done the math for what this would cost on a per megabyte or per minute basis?
Have you run real world amounts of data through this system, and again for what purpose?