When you train a Large Language Model, there are various tricks you use to increase its usable context length. Back in the primitive dark ages of 2022, models maxed out at 8k tokens or so. Modern models typically operate at 1 million context length or higher!
The main technique is to simply train the model on examples that have longer context length. These are more expensive to train on (and more expensive to collect), but improve performance very reliably. There are other tricks, but I won’t bore you with those.
So: you have a model, you train it on long context examples, and it’s good at long context. Train it on a bunch of short context examples, and it’s bad at long context (and perhaps even degrades on previous long context performance).
Perhaps unsurprisingly, so too with the little meat language model in your head! Sadly, social media and the modern internet is positively drowning with short context examples. Scrolling X? 140 characters or less, baby! You receive an unlimited slot machine of short context examples to “train on”, needing very little processing power to consume. Short form video is even worse, as you don’t even need to actively consume the words on the screen, they are simply jammed into your ears regardless of your consent.
Compare this to one of the best activities in the world, reading a book1! The ultimate long context example, hundreds of pages of a universe, painstakingly built up by another human to be delivered into your brain. Try scrolling a bit and then try to read a hundred pages of dense philosophy — your mental GPU cluster will protest and overheat immensely2! When I read a book with my phone sitting nearby, I can practically feeling it calling to me, as if it’s the One Ring. My phone is so tempting, it just wants me to pick it up and scroll for a minute… the book will still be there!
Like most things in life, the hard thing is the most valuable: reading books builds your mental acuity and ability to reason about hard problems, reading many high frequency short examples destroys it.
Seek long context training examples, avoid short ones.
Footnotes
-
My AirPods are engraved with the phrase “Read and Lift”, as I really do believe those are two of the most beneficial and important activities for wellbeing. It’s a subtle variation of Will Smith’s famous “Running and Reading” speech, where he argues similarly — I just hate running. Sorry Will, cardio sucks. ↩
-
Hahaha, aren’t “humans are computers” analogies fun? I could do this all day! ↩