Tag: information-theory
All the articles with the tag "information-theory".
Secondhand Scars
Published: at 03:11 PMA corpus is a record of other people's scars, and that is why pretraining works so well and stalls at the frontier, where nobody has been hurt yet. A hypothesis about why the world only writes into a system through damage.
The Past Is Only Mixed
Published: at 04:22 AMPhysics has no delete key. Stir a drop of ink into water and the ink is not gone, only spread past reading. What the equations actually say about loss, and why the past is entropied rather than destroyed.
The Mirror Problem
Published: at 03:44 PMA model trained on its own output loses range generation after generation, the tails of its distribution slowly disappearing. I hit this kind of loop in my own systems, and it changed how I think about the difference between a model learning and a mirror reflecting.