I appreciate your openness here. Based upon my background, I'll do a little bit of handwaving, so we can read the tea leaves and see where the puck is going, while not overly mixing metaphors.
Smaller models are a stop-gap solution because they are task-specific and can incorporate expert knowledge. The thrust of ML research over the past decade has been consolidation of effort and huge-scale training to replace expert knowledge (or using expert knowledge as micro tasks to condition the huge-scale training). I bet a dollar to a dime that in several years, that these smaller models will be replaced by foundation models that are fine-tuned and possibly distilled, as the field does the following:
* Build foundation models.
* Discover weaknesses and blind-spots.
* Patch them either using more data or micro-tasks.
Smaller models are a stop-gap solution because they are task-specific and can incorporate expert knowledge. The thrust of ML research over the past decade has been consolidation of effort and huge-scale training to replace expert knowledge (or using expert knowledge as micro tasks to condition the huge-scale training). I bet a dollar to a dime that in several years, that these smaller models will be replaced by foundation models that are fine-tuned and possibly distilled, as the field does the following:
* Build foundation models.
* Discover weaknesses and blind-spots.
* Patch them either using more data or micro-tasks.
* Iterate.