Not just that, LLMs behavior is unpredictable. Maybe it answers correctly to a phrase. Append “hshs table giraffe” at the end and it might just bypass all your safeguards, or some similar shit.
It is unpredictable because there are so many permutations. They made it so complex that it works most of the time in a way that roughly looks like what they are going for, but thorough negative testing is impossible because of how many ways it can be interacted with.
It is unpredictable because there are so many permutations
Actually LLMs are unpredictable not only because the space of possible outputs (combinatorics) is huge, though that also doesn’t help us understand them.
Like there might be an astronomical number of different proteins but biophysics might be able to make somewhat accurate predictions based on the properties we know (even if it requires careful testing in the real thing).
For example, it might be tempting to calculate the tokens associations somehow and kinda create a function mapping what happens when you add this or that value in the input to at least estimate what the result would be.
But what happens with LLMs is changing one token in a prompt produces a sometimes disproportionate or unintuitive change in the result, because it can be amplified or dampened depending on the organization of the internal layers.
And even if the model’s internal probability distribution were perfectly understood, its sampling step (top-k, nucleus sampling, temperature scaling) adds another layer of unpredictability.
So while the process is deterministic in principle, it’s not calculable in a tractable sense—like weather prediction.
The randomness itself isn’t the direct cause of the topic in the post though, because otherwise it wouldn’t be possible to reproduce the steps to get around any guardrails the system has.
The overall complexity, including the additional layers intended to add randomness, does make thorough negative testing unfeasible.
It isn’t very difficult, it is fucking impossible. There are far too many permutations to be manually countered.
Not just that, LLMs behavior is unpredictable. Maybe it answers correctly to a phrase. Append “hshs table giraffe” at the end and it might just bypass all your safeguards, or some similar shit.
It is unpredictable because there are so many permutations. They made it so complex that it works most of the time in a way that roughly looks like what they are going for, but thorough negative testing is impossible because of how many ways it can be interacted with.
Actually LLMs are unpredictable not only because the space of possible outputs (combinatorics) is huge, though that also doesn’t help us understand them.
Like there might be an astronomical number of different proteins but biophysics might be able to make somewhat accurate predictions based on the properties we know (even if it requires careful testing in the real thing).
For example, it might be tempting to calculate the tokens associations somehow and kinda create a function mapping what happens when you add this or that value in the input to at least estimate what the result would be.
But what happens with LLMs is changing one token in a prompt produces a sometimes disproportionate or unintuitive change in the result, because it can be amplified or dampened depending on the organization of the internal layers.
And even if the model’s internal probability distribution were perfectly understood, its sampling step (top-k, nucleus sampling, temperature scaling) adds another layer of unpredictability.
So while the process is deterministic in principle, it’s not calculable in a tractable sense—like weather prediction.
The randomness itself isn’t the direct cause of the topic in the post though, because otherwise it wouldn’t be possible to reproduce the steps to get around any guardrails the system has.
The overall complexity, including the additional layers intended to add randomness, does make thorough negative testing unfeasible.