> If instead of a list of tokens you want to pass in a list of embedding vectors, that is trivial (assuming you're running a LLM with your own code locally), that would be a <5 line function.
That sounds like what I wanted. I haven't got to the point of running open-source LLMs locally yet, so I didn't realize it's as easy as you say - rather, since I haven't seen it being mentioned as something you do with LLM, I assumed it must be nontrivial.
> However, I'm not really sure when/why I would want to do that
High-level dream: I want to explore the latent space of an LLM. Get a feel for how it's organized, and what of it is spurious, and what structures seem to be common among LLMs. Find answers to questions that will come to me when I "go there" and "look around".
It might be a futile pursuit - but I feel it's not the idea that's the problem, but rather that high-dimensional spaces might be too big to explore without mathematical understanding that's way beyond mine.
> as "[the specific ideas I listed]" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts,
In general, or at all?
Am I wrong in suspecting that averaging the embeddings of "orange" and "apple" and "banana" may give us a new embedding that, fed to the LLM, will make it output something meaningful (vs. random)? From what I read today, elsewhere in the thread, it seems that this works as expected for Stable Diffusion family of models.
And sure, if I take two random multi-token prompts, I expect the result will make LLM output some noise in the shape of word salad. But what if it's multiple closely-related prompts? I have my suspicions, but I want to find out for sure.
> and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction.
Shifting prompts in some meaningful direction may not be trivial (or useful), but it's not the only interesting thing to do. What if I push a single token in the prompt a little bit in some direction? What if I push all of them? How things change with subsequent pushes?
> In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level.
I'm definitely not assuming the text can be converted to a single "embedding vector". My questions boil down to, where is debug-level access to the LLM? Why is everyone focused on stringing tokens, instead of looking at what's lurking between the tokens and their sequences, in areas of the latent space you can't easily get to without embeddings that don't map to any possible prompt.
IDK, maybe my mental model of transformer models is completely broken, or maybe all those questions have been asked, and didn't lead to anything of interest, and I just can't seem to find the right papers covering this.
That sounds like what I wanted. I haven't got to the point of running open-source LLMs locally yet, so I didn't realize it's as easy as you say - rather, since I haven't seen it being mentioned as something you do with LLM, I assumed it must be nontrivial.
> However, I'm not really sure when/why I would want to do that
High-level dream: I want to explore the latent space of an LLM. Get a feel for how it's organized, and what of it is spurious, and what structures seem to be common among LLMs. Find answers to questions that will come to me when I "go there" and "look around".
It might be a futile pursuit - but I feel it's not the idea that's the problem, but rather that high-dimensional spaces might be too big to explore without mathematical understanding that's way beyond mine.
> as "[the specific ideas I listed]" are NOT really enabled by that - since that inherently must be a vector of token-embedding vectors combined with positional embeddings, you can't just average different prompts,
In general, or at all?
Am I wrong in suspecting that averaging the embeddings of "orange" and "apple" and "banana" may give us a new embedding that, fed to the LLM, will make it output something meaningful (vs. random)? From what I read today, elsewhere in the thread, it seems that this works as expected for Stable Diffusion family of models.
And sure, if I take two random multi-token prompts, I expect the result will make LLM output some noise in the shape of word salad. But what if it's multiple closely-related prompts? I have my suspicions, but I want to find out for sure.
> and there isn't a clear way how to tweak the individual token embeddings that would map to tweaking the whole prompt in some meaningful direction.
Shifting prompts in some meaningful direction may not be trivial (or useful), but it's not the only interesting thing to do. What if I push a single token in the prompt a little bit in some direction? What if I push all of them? How things change with subsequent pushes?
> In essence, your question seems to make sense if the text was converted to a single "embedding vector", but it's not, at every layer of a transformer model the state is a list of separate embeddings distinctly separated at a token level.
I'm definitely not assuming the text can be converted to a single "embedding vector". My questions boil down to, where is debug-level access to the LLM? Why is everyone focused on stringing tokens, instead of looking at what's lurking between the tokens and their sequences, in areas of the latent space you can't easily get to without embeddings that don't map to any possible prompt.
IDK, maybe my mental model of transformer models is completely broken, or maybe all those questions have been asked, and didn't lead to anything of interest, and I just can't seem to find the right papers covering this.