I rate Iain M Banks as my favourite author of all time, but the AIs in his book have a very different "flavour" compared to the intelligent-but-not-sentient aliens and AIs in Blindsight.
An interesting aspect of the Culture Minds is that they're described as hybrids of human-like / neural-net / organic intelligence and also direct numerical compute and storage capacity. They can both "think" and "compute", something few if any current AI models attempt.
A well-known limitation of current LLMs like ChatGPT is that they are atrocious at even basic arithmetic, whereas a $2 pocket calculator from the 1980s outperforms them in this respect!
I'd love to see an attempt at a hybrid architecture where the neurons are also connected to a numeric compute component of the architecture.
Think of parsing an expression such as:
let foo = 5 + 10
ChatGPT parses this as:
"let", " foo", " =", " 5", " +", " 10"
Essentially, its lexer produces a series of token IDs that don't contain the numeric values of "5" and "10":
[1309, 22944, 796, 642, 1343, 838]
Programming language lexers would parse that as:
[LET, ID("foo"), EQ, NUM(5), PLUS, NUM(10)]
In other words, lexer tokens in programming languages can carry baggage such as numeric values.
What if we lifted this restriction and made an AI embedding where every token had both an ID and a value? For literal text just set the ID to NaN or whatever.
E.g.: assume that "integer constant" is ID 1000, then an LLM might be fed this input instead:
Then the model could be wired up such that the additional tensor dimension can be sent to both the neural net or a direct compute block that does various kinds of maths, such as adding all inputs, multiplying all inputs, etc...
An interesting aspect of the Culture Minds is that they're described as hybrids of human-like / neural-net / organic intelligence and also direct numerical compute and storage capacity. They can both "think" and "compute", something few if any current AI models attempt.
A well-known limitation of current LLMs like ChatGPT is that they are atrocious at even basic arithmetic, whereas a $2 pocket calculator from the 1980s outperforms them in this respect!
I'd love to see an attempt at a hybrid architecture where the neurons are also connected to a numeric compute component of the architecture.
Think of parsing an expression such as:
ChatGPT parses this as: Essentially, its lexer produces a series of token IDs that don't contain the numeric values of "5" and "10": Programming language lexers would parse that as: In other words, lexer tokens in programming languages can carry baggage such as numeric values.What if we lifted this restriction and made an AI embedding where every token had both an ID and a value? For literal text just set the ID to NaN or whatever.
E.g.: assume that "integer constant" is ID 1000, then an LLM might be fed this input instead:
Then the model could be wired up such that the additional tensor dimension can be sent to both the neural net or a direct compute block that does various kinds of maths, such as adding all inputs, multiplying all inputs, etc...Could be interesting...