Tag: data-quality
All the articles with the tag "data-quality".
Secondhand Scars
Published: at 03:11 PMA corpus is a record of other people's scars, and that is why pretraining works so well and stalls at the frontier, where nobody has been hurt yet. A hypothesis about why the world only writes into a system through damage.
The Mirror Problem
Published: at 03:44 PMA model trained on its own output loses range generation after generation, the tails of its distribution slowly disappearing. I hit this kind of loop in my own systems, and it changed how I think about the difference between a model learning and a mirror reflecting.