How much do we assume human written code for training the original models is free of human made slop? We all know we all take shortcuts and have code we are not proud of. If they are I deed averaging machines, is it possible their output is the average of human output?
I'd argue that the way attention works plays into the slop too
Yes, that's why good datasets are important. But usually the difference between model generations is architectural improvements
late training is where a lot of capability gains come from too
How much do we assume human written code for training the original models is free of human made slop? We all know we all take shortcuts and have code we are not proud of. If they are I deed averaging machines, is it possible their output is the average of human output?
I'd argue that the way attention works plays into the slop too