If a process is busy processing a task, it is busy.
A process could receive N tasks in a batch and process them at the same time.
In this scenario it is possible for all workers to be utilised whilst the queue is not “100% utilised” - which is a failure mode the link you shared explained:
> And by induction, we can show that a system that’s forever at 100% utilization will exceed any finite queue size you care to pick
The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks.
It is specific on the utilisation type though, I’m mainly thinking about ML inference that has a fairly fixed initial cost per inference but scales quite nicely with >1 input.
You can't have 100% utilization of any processing capacity (ignore queues) without reaching this type of failure. 100% CPU utilization is essentially just meaning you filled up the queue on your worker server.
> The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks.
Right, but at some point they will have to have SOME idle time. If they are NEVER idle the queue by definition is growing, not shrinking, as you said.
Unless you have some system that monitors queue length and doesn't add jobs if it is like >n, which is what the article suggests (maximum queue size). But then you have to figure out what you do with those jobs that can't be added, which probably means just creating some kind of queue higher up in the pipe.
If they are processing more than is being added, eventually things will run out of the queue.
A process could receive N tasks in a batch and process them at the same time.
In this scenario it is possible for all workers to be utilised whilst the queue is not “100% utilised” - which is a failure mode the link you shared explained:
> And by induction, we can show that a system that’s forever at 100% utilization will exceed any finite queue size you care to pick
The inflows to the queue are less than the outflows, despite every worker spending 100% of its time processing tasks.
It is specific on the utilisation type though, I’m mainly thinking about ML inference that has a fairly fixed initial cost per inference but scales quite nicely with >1 input.