Do we even have a good definition of what it means to generalize? I keep seeing this term thrown about and while it makes intuitive sense, and certainly train/test/validation splits are useful, I sort of question the whole premise. I mean, when we consider all possible sets, all optimization problems perform equally well (or poorly), so in the most general sense, generalization is impossible. Which means we have to restrict our sets somehow, but to do that we have to have some measurement as to how they relate, but how do you know in advance? In a sense what the machine is learning is a distance metric between these sets, but the only way to know it's working is to actually run it. To say in advance "well, this machine generalizes this well on this class of problems" seems like an awful stretch.
So much of what we mean when we say "this model generalizes well" seems like baked-in human assumptions about the data distributions that may not really have any basis in reality.
Generalize does need to be more specific in order to make sense. In current ML work, generalizability means something else than what it typically used to mean in AI work.
One way to look at what generalizability is would be like this: problem domains typically come with sets of axioms. Generalizability is the ability for a solution approach to work across different domains that don't have 100% axiom overlap. The wider the difference between axiom sets for a domain, the more difficult / impressive the generalization.
Solving two problems within the same domain, necessarily sharing a 100% overlapping axiom set, is not generalization at all.
The reason the axioms supporting the domain matter is that they (in part) guide which heuristics work, and how they should be applied. And that's the sort of generalizability that is missing from current work: some solutions can make some pre-programmed choices about applying different heuristics depending on the problem set (driverless cars are doing this now). This is the "bag of tricks" approach. But they don't typically morph in how they are applied, or the end that they're trying to accomplish when they're applied.
I have pondered this and have decided what my standards are. I realize I don't have the authority to set those standards for everyone.
GAI is, to me, when machine is able to be given a problem and then, without prompting, decides which data to consume to learn how to solve that problem. It could be told to optimize an automotive design for a 5% efficiency increase without a loss of safety features and while keeping the performance the same - and then go out and figure out what data it needs to learn so that it can solve that task. It would assemble and process that data and then come up with the answer, which might just be that it is impossible with current tech and here is what is needed and this is how to do it.
That's rather verbose and I'm absolutely not the person who gets to define it. But, when someone says AI, that is how I think of it. More so when they say general AI.
So much of what we mean when we say "this model generalizes well" seems like baked-in human assumptions about the data distributions that may not really have any basis in reality.