Pretty impressive how the Mac ends up less expensive than the Strix Halo boxes, at least here in Ireland. A 128GB Mac Studio with an M5 Max (the Ultra can only have 96 or 256GB) still costs less than the "GMKtec EVO-X2" or the Nvidia DGX Spark with similar performance.
I'm encountering the same behavior. I've tried 4-8bit quants and get 14-17 tok/s with one run that achieved 19. I'm eagerly awaiting dflash2 support in Unsloth or LM Studio, as allegedly that should increase throughput to around 30tok/s, which is the baseline for what I consider at least somewhat interactive.
Jealous of the folks with 5090s running ninfer and getting >100tok/s. At those speeds it's a true frontier replacement IMO.
That isn’t right. Use oMLX or something similar to serve the model with internal MTP enabled. You should get at least 40 tok/sec, but I’m not sure what hardware you are using. Dflash will mess up batching, not really worth it. You need to sift through hugging face for the right model though, and some of the MTP models are meant for rapid MLX and not oMLX.
Apart from learning, I cant see the point of spending that kind of money and energy to get such awfull token generation speed. Assuming memory bandwith is the bottleneck, is it just a matter of time until we start to see hbm4 based chip able to run qwen3.8 for normal people running at more than 500 token / seconds? Is memory speed the only technological bottlenecks that prevent us from having fast local model?
For me, a HUGE benefit to running local models is that my data stays mine, on my computers only.
Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law means nothing to them.
With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap.
For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant.
I am really surprised you’re seeing half the generation performance of 3.6 with 3.8 at the same parameter size and quantization (and same prefill performance to boot) - is there just an optimization in the stack somewhere for 3.6 that hasn’t landed yet for 3.8?
Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.
I suspect this might be due to MTP mispedictions. Either because the 3.8 quantized model weights do not include MTP heads, or as what happened with Ornith-1.5 recently, corrupted MTP heads, or due to a software issue in Ollama.
These numbers are a lot lower than I expected from such pricey hardware.
I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.
The local LLM scene needs a Draw Things equivalent for Mac. Too much fiddle for things that doesn't make sense (Qwen 3.8 27B should be exactly the same speed as Qwen 3.6 27B). It feels like that I am teasing (I am the author of Draw Things) something, because it is.
What you get with Draw Things: 1. Download the app from Mac AppStore; 2. Download the model; 3. Tap "Try recommended settings", have the guarantee that for whatever model it supports, it is the fastest in Mac ecosystem, no need to fiddle.
What you get with local LLM options: 1. Download the inference engine app from the web; 2. Download the model; 3. Configure MTP / DFlash / DSpark whatever; 4. Configure your Pi / OpenCode harness to point to this local LLM inference engine. 5. Configure tools for these harness to be effective. 6. Switching between Ollama, LM Studio, llama.cpp (DwarfStar4), oMLX, MTPLX, to see which one is fastest for your workload. 7. Again switching between different quants of the same model to see which one is less dumb.
To be honest, llama.cpp probably the closest to deliver on "just use it, don't worry about speed" if your focus is about a pure LLM inference engine.
This low-effort slop post is misleading. It suggest that 3.8 is half as fast as 3.6, but this is almost definitely because MTP isn't enabled by default. The two models should perform the same. When you get AI to think for you, you lose.
> Qwen3.8 27B (Q4_K_M, 17GB) generates at ~14 tokens/s on my Mac Studio M3 Ultra
>~14 tokens/s
For anyone reading that has never ran local llms, please understand that anything under 100 tok/sec is worthless. You are faster typing stuff into Gemini free version that you get with a google account and copy/pasting it in (and you can easily build browser automation with playwright or any other js runtime to have this available in a chat window)
I get 80 tok/s on the same model, and it's pretty usable. I'm not chatting with it; it's either given a bag of tokens to generate an answer or it's doing some agentic coding.
I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright fast. Don't forget there is a limit entering content into meatspace.
Also, Gemini may be free but what if I don't want to give all my data to Google? This is precisely why I have a lot of stuff locally.
And will it remain free? How are they going to make back all those trillions of investment?
But yeah I would kinda balk at 14tk/s too that's why I use old datacenter/workstation-class GPUs.
I wouldn't say worthless but once you get used to 100+ tokens / s, it's really a visible slowdown, especially when running multiple agents acting on something more than basic prompt processing.
Still, as you said - for day to day, 50-60 tokens / s and my interactions is generally fine.
Pretty impressive how the Mac ends up less expensive than the Strix Halo boxes, at least here in Ireland. A 128GB Mac Studio with an M5 Max (the Ultra can only have 96 or 256GB) still costs less than the "GMKtec EVO-X2" or the Nvidia DGX Spark with similar performance.
I'm encountering the same behavior. I've tried 4-8bit quants and get 14-17 tok/s with one run that achieved 19. I'm eagerly awaiting dflash2 support in Unsloth or LM Studio, as allegedly that should increase throughput to around 30tok/s, which is the baseline for what I consider at least somewhat interactive.
Jealous of the folks with 5090s running ninfer and getting >100tok/s. At those speeds it's a true frontier replacement IMO.
That isn’t right. Use oMLX or something similar to serve the model with internal MTP enabled. You should get at least 40 tok/sec, but I’m not sure what hardware you are using. Dflash will mess up batching, not really worth it. You need to sift through hugging face for the right model though, and some of the MTP models are meant for rapid MLX and not oMLX.
Apart from learning, I cant see the point of spending that kind of money and energy to get such awfull token generation speed. Assuming memory bandwith is the bottleneck, is it just a matter of time until we start to see hbm4 based chip able to run qwen3.8 for normal people running at more than 500 token / seconds? Is memory speed the only technological bottlenecks that prevent us from having fast local model?
For me, a HUGE benefit to running local models is that my data stays mine, on my computers only.
Even if Google, OpenAI, Anthropic whatever promise not to use it, how sure can I be of that? Data is gold anyway. And they've already been caught scraping sites they had no business to, and pirating books. Clearly their promises and the law means nothing to them.
With local models besides the speed you also lose a lot of inference quality but it helps to mitigate that. For example making sure your RAG inputs are properly prepared and categorised so the model can find them easily without having to wade through a bunch of misdirected crap.
For example what I do with my bookmarks and chats (the latter are recorded per day), is before I enter them into a RAG corpus I run a small LLM over it to summarise what's being discussed or what the webpage is about. That really helped retrieval quality, and doing this is a batch task that can run asynchronously so speed is not very relevant.
I am really surprised you’re seeing half the generation performance of 3.6 with 3.8 at the same parameter size and quantization (and same prefill performance to boot) - is there just an optimization in the stack somewhere for 3.6 that hasn’t landed yet for 3.8?
Curious if other folks are also surprised by this apparent discrepancy or have a ready explanation.
I suspect this might be due to MTP mispedictions. Either because the 3.8 quantized model weights do not include MTP heads, or as what happened with Ornith-1.5 recently, corrupted MTP heads, or due to a software issue in Ollama.
These numbers are a lot lower than I expected from such pricey hardware.
I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.
any numbers to share? also, what inference engine do you use and with which api?
The local LLM scene needs a Draw Things equivalent for Mac. Too much fiddle for things that doesn't make sense (Qwen 3.8 27B should be exactly the same speed as Qwen 3.6 27B). It feels like that I am teasing (I am the author of Draw Things) something, because it is.
I’ve never personally used it, but isn’t that what LM Studio is positioned as? From what I can gather, it seems extremely beginner-friendly.
What you get with Draw Things: 1. Download the app from Mac AppStore; 2. Download the model; 3. Tap "Try recommended settings", have the guarantee that for whatever model it supports, it is the fastest in Mac ecosystem, no need to fiddle.
What you get with local LLM options: 1. Download the inference engine app from the web; 2. Download the model; 3. Configure MTP / DFlash / DSpark whatever; 4. Configure your Pi / OpenCode harness to point to this local LLM inference engine. 5. Configure tools for these harness to be effective. 6. Switching between Ollama, LM Studio, llama.cpp (DwarfStar4), oMLX, MTPLX, to see which one is fastest for your workload. 7. Again switching between different quants of the same model to see which one is less dumb.
To be honest, llama.cpp probably the closest to deliver on "just use it, don't worry about speed" if your focus is about a pure LLM inference engine.
Or you can just ask Sol to do all that for you, and do split testing, to find best performance.
I would like to see "guarantee" for that app. You get full refound and it is free, right?
This low-effort slop post is misleading. It suggest that 3.8 is half as fast as 3.6, but this is almost definitely because MTP isn't enabled by default. The two models should perform the same. When you get AI to think for you, you lose.
> Qwen3.8 27B (Q4_K_M, 17GB) generates at ~14 tokens/s on my Mac Studio M3 Ultra
>~14 tokens/s
For anyone reading that has never ran local llms, please understand that anything under 100 tok/sec is worthless. You are faster typing stuff into Gemini free version that you get with a google account and copy/pasting it in (and you can easily build browser automation with playwright or any other js runtime to have this available in a chat window)
I get 80 tok/s on the same model, and it's pretty usable. I'm not chatting with it; it's either given a bag of tokens to generate an answer or it's doing some agentic coding.
https://openrouter.ai/anthropic/claude-opus-5 is it worthless because its 65 tps?
re: gemini
https://openrouter.ai/google/gemini-3.7-flash worthless as well?
That being said, 14 tok/s is pretty slow.
What? Below 100tk/s is worthless?
I agree that 14t/s is pretty tedious for interactive use, yes. But 50-60tk/s is faster than I can read. 100tk/s is outright fast. Don't forget there is a limit entering content into meatspace.
Also, Gemini may be free but what if I don't want to give all my data to Google? This is precisely why I have a lot of stuff locally.
And will it remain free? How are they going to make back all those trillions of investment?
But yeah I would kinda balk at 14tk/s too that's why I use old datacenter/workstation-class GPUs.
I wouldn't say worthless but once you get used to 100+ tokens / s, it's really a visible slowdown, especially when running multiple agents acting on something more than basic prompt processing.
Still, as you said - for day to day, 50-60 tokens / s and my interactions is generally fine.