I think we should revisit outrageously small neural nets.
I needed a cheap model that runs at over 10k token/sec on a single CPU core for some data processing. So I gave Anthropic claude code a pile of tokens to build one.
It made three discoveries that I thought were interesting:
1) One Intel AMX core can train a 3M active parameter MoE foundation model at 6,616 tok/s on 4.91B NVIDIA Nemotron tokens in a few days.
2) That model shows emergent in-context copying, positional analogies, and basic arithmetic after about 250M tokens.
3) The foundation model gives large gains in downstream SFT, and the training & eval loss keep going down all the way through 4.91B (and likely beyond).
Claude is not as good as a great MLE at debugging MoE. It made a bunch of bone headed mistakes, but it got there in the end.
I asked it to write a paper about it's work, and it produced this.
I think we should revisit outrageously small neural nets.
I needed a cheap model that runs at over 10k token/sec on a single CPU core for some data processing. So I gave Anthropic claude code a pile of tokens to build one.
It made three discoveries that I thought were interesting:
1) One Intel AMX core can train a 3M active parameter MoE foundation model at 6,616 tok/s on 4.91B NVIDIA Nemotron tokens in a few days.
2) That model shows emergent in-context copying, positional analogies, and basic arithmetic after about 250M tokens.
3) The foundation model gives large gains in downstream SFT, and the training & eval loss keep going down all the way through 4.91B (and likely beyond).
Claude is not as good as a great MLE at debugging MoE. It made a bunch of bone headed mistakes, but it got there in the end.
I asked it to write a paper about it's work, and it produced this.
Claude Co-Authored Paper: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct/blo...
I read through it and it sounds a bit LLMy, but the main points and experiment results are correct.
Some of the models are published on HF: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct