Post
48
Why can a language model do better on a longer text?
Imagine asking an AI to process a document while following 32 instructions. It gets only a third of the required actions right. Then you give it a document several times longer, with the same 32 instructions, and it makes almost no mistakes.
A paradox? That's exactly what happened in an experiment with GPT-4o mini.
In the IEC instrument, a model rewrites text using simple word-replacement rules like
Turns out, 32 instructions don't always mean the same load.
Ordinary prompts work the same way. Opening the answer with a greeting is one instruction; banning commas from the whole answer is one too. But the first is satisfied once, while the second must be respected at every step of generation. Higher density, higher load.
So what actually determines the load?
My new article answers that. I explain why some seemingly obvious candidates had to be rejected and how the data revealed five load axes: input text length, word position in the response, replacement density, number of rules, and the rule's position in the list.
Why does it matter? Benchmarks usually just add instructions, but as their number grows, so do their positions in the list and the length of the prompt and the response. A single curve mixes all these effects: the model errs more often, but you can't tell why. The axes separate the causes, and each follows its own law of degradation.
Code and data are open, so you can check everything yourself.
Article: https://huggingface.co/blog/kutomanovhennadii/what-affects-a-models-behavior-under-load
Data: kutomanovhennadii/iec-gpt-4o-mini-2024-07-18
Originally written in Russian. English translation assisted by AI.
Imagine asking an AI to process a document while following 32 instructions. It gets only a third of the required actions right. Then you give it a document several times longer, with the same 32 instructions, and it makes almost no mistakes.
A paradox? That's exactly what happened in an experiment with GPT-4o mini.
In the IEC instrument, a model rewrites text using simple word-replacement rules like
lake → stone. In one task, the rules covered all 32 distinct words of the text; in the other, only 32 out of 256. The number of instructions stayed the same, but the replacement density dropped eightfold.Turns out, 32 instructions don't always mean the same load.
Ordinary prompts work the same way. Opening the answer with a greeting is one instruction; banning commas from the whole answer is one too. But the first is satisfied once, while the second must be respected at every step of generation. Higher density, higher load.
So what actually determines the load?
My new article answers that. I explain why some seemingly obvious candidates had to be rejected and how the data revealed five load axes: input text length, word position in the response, replacement density, number of rules, and the rule's position in the list.
Why does it matter? Benchmarks usually just add instructions, but as their number grows, so do their positions in the list and the length of the prompt and the response. A single curve mixes all these effects: the model errs more often, but you can't tell why. The axes separate the causes, and each follows its own law of degradation.
Code and data are open, so you can check everything yourself.
Article: https://huggingface.co/blog/kutomanovhennadii/what-affects-a-models-behavior-under-load
Data: kutomanovhennadii/iec-gpt-4o-mini-2024-07-18
Originally written in Russian. English translation assisted by AI.
