Tag: machine-learning
All the articles with the tag "machine-learning".
Secondhand Scars
Published: at 03:11 PMA corpus is a record of other people's scars, and that is why pretraining works so well and stalls at the frontier, where nobody has been hurt yet. A hypothesis about why the world only writes into a system through damage.
The Mirror Problem
Published: at 03:44 PMA model trained on its own output loses range generation after generation, the tails of its distribution slowly disappearing. I hit this kind of loop in my own systems, and it changed how I think about the difference between a model learning and a mirror reflecting.
The Forgiveness That Dies First
Published: at 01:42 PMmuP makes learning-rate transfer work by keeping the set of good rates wide. I measured what low precision does to that forgiveness, and the surprising part: the transfer survives. The forgiveness is what dies first.