Hi, I am searching for benchmarks that compare the performance of various machine learning model serving frameworks. Some of the previous posts such as the following exist but they don't paint the full picture. Is there any reliable benchmark that gives a good snapshot of the state of art?
1. https://news.ycombinator.com/item?id=28760158
2. https://github.com/cortexlabs/cortex/tree/v0.15.1
Not exactly what you’re looking fir, but perhaps you’ll find it useful - llama benchmarked on all M-series chips, and in comments there are comparisons with nvidia.
There is no single answer, mostly a lot of strengths and weaknesses around which formats you can use on your specific hardware / amount of VRAM / if you want to ue multiple GPUs, whether they are exact same GPU or not / etc.
https://mlcommons.org/benchmarks/inference-datacenter/