I think we could have a fascinating discussion about this offline. But in short here’s my understanding: they look at a bunch of queries and try to deduce the vector that represents a particular idea—like let’s say “sphere”. So then without changing the prompt, they inject that concept.
How does this injection take place?
I played with a service a few years ago where we could upload a corpus of text and from it train a “prefix” that would be sent along with every prompt, “steering” the output ostensibly to be more like the corpus. I found the influence to be undetectably subtle on that model, but that sounds a lot like what is going on here. And if that’s not it then I don’t really follow exactly what they are doing.
Anyway my point is, that concept of a sphere is still going into the context mathematically even if it isn’t in the prompt text. And that concept influences the output—which is entirely the point, of course.
None of that part is introspective at all. The introspection claim seems to come from unprompted output such as “round things are really on my mind.” To my way of thinking, that sounds like a model trying to bridge the gap between its answer and the influence. Like showing me a Rorschach blot and asking me about work and suddenly I’m describing things using words like fluttering and petals and honey and I’m like “weird that I’m making work sound like a flower garden.”
And then they do the classic “why did you give that answer” which naturally produces bullshit—which they at least acknowledge awareness of—and I’m just not sure the output of that is ever useful.
Anyway, I could go on at length, but this is more speculation than fact and a dialog would be a better format. This sounds a lot like researchers anthropomorphizing math by conflating it with thinking, and I don’t find it all that compelling.
That said, I see analogs in human thought and I expect some of our own mechanisms may be reflected in LLM models more than we’d like to think. We also make decisions on words and actions based on instinct (a sort of concept injection) and we can also be “prefixed” for example by showing a phrase over top of an image to prime how we think about those words. I think there are fascinating things that can be learned about our own thought processes here, but ultimately I don’t see any signs of introspection—at least not in the way I think the word is commonly understood. You can’t really have meta-thoughts when you can’t actually think.
Shit, this still turned out to be about 5x as long as I intended. This wasn’t “in short” at all. Is that inspection or just explaining the discrepancy between my initial words and where I’ve arrived?
I think we could have a fascinating discussion about this offline. But in short here’s my understanding: they look at a bunch of queries and try to deduce the vector that represents a particular idea—like let’s say “sphere”. So then without changing the prompt, they inject that concept.
How does this injection take place?
I played with a service a few years ago where we could upload a corpus of text and from it train a “prefix” that would be sent along with every prompt, “steering” the output ostensibly to be more like the corpus. I found the influence to be undetectably subtle on that model, but that sounds a lot like what is going on here. And if that’s not it then I don’t really follow exactly what they are doing.
Anyway my point is, that concept of a sphere is still going into the context mathematically even if it isn’t in the prompt text. And that concept influences the output—which is entirely the point, of course.
None of that part is introspective at all. The introspection claim seems to come from unprompted output such as “round things are really on my mind.” To my way of thinking, that sounds like a model trying to bridge the gap between its answer and the influence. Like showing me a Rorschach blot and asking me about work and suddenly I’m describing things using words like fluttering and petals and honey and I’m like “weird that I’m making work sound like a flower garden.”
And then they do the classic “why did you give that answer” which naturally produces bullshit—which they at least acknowledge awareness of—and I’m just not sure the output of that is ever useful.
Anyway, I could go on at length, but this is more speculation than fact and a dialog would be a better format. This sounds a lot like researchers anthropomorphizing math by conflating it with thinking, and I don’t find it all that compelling.
That said, I see analogs in human thought and I expect some of our own mechanisms may be reflected in LLM models more than we’d like to think. We also make decisions on words and actions based on instinct (a sort of concept injection) and we can also be “prefixed” for example by showing a phrase over top of an image to prime how we think about those words. I think there are fascinating things that can be learned about our own thought processes here, but ultimately I don’t see any signs of introspection—at least not in the way I think the word is commonly understood. You can’t really have meta-thoughts when you can’t actually think.
Shit, this still turned out to be about 5x as long as I intended. This wasn’t “in short” at all. Is that inspection or just explaining the discrepancy between my initial words and where I’ve arrived?