Two issues with this. One, it's profitable assuming you just keep serving the same model forever, which is not realistic in this market. A given model has a shelf-life, which these days is measured in months, not years. Which means that trying to separate the cost of training the model from the cost of serving it doesn't make much business sense. And two, for providers that provide inference only via open weight models, the margins quickly move to commoditization. The "someday" when frontier model providers can enjoy their current high inference margins without the burden of significant training costs is never going to arrive.
Commoditization means there's price competition. From a consumer perspective, that's good. You want it to be a low-margin, high volume, competitive business.
Although from a business perspective, it can end up being ruinous competition like solar panels or airlines. A stable equilibrium with prices neither too low or too high isn't guaranteed; it depends on market structure.
It's anyone's guess whether this reaches an equilibrium or not, but I still expect that there will be companies like OpenRouter and Fireworks that offer inference at reasonable prices.
Profitable on a per token basis is meaningless. Ed Zitron doesn’t argue that it is impossible to offer profitable inference, he argues that the business as it stands today is deeply unprofitable and only getting worse because it isn’t a high-margin inference business.
Play out the most likely pessimist’s scenario: LLMs are useful but frontier models are overkill so businesses just use dirt cheap open weight models on their own hardware and/or they rent hardware instead of paying per token. Then what for OpenAI and Anthropic?
OpenAI’s business collapses if customers are happy with an LLM that costs $0.10 per million tokens even if it only costs OpenAI $0.05 in inference per million tokens. The insane bonkers claim from Garry Tan that in 2 years we will be using 90,000x as many tokens as today is… well, obviously not true.
The fixed costs that OpenAI and Anthropic have created need inference demand far beyond what is plausible.
edit: and hand waving away the vast losses of companies like OpenAI because of “training” is ridiculous. Anthropic are spending a billion dollars per month to rent additional capacity from xAI for inference, not training. The models don’t need to get better: if there is a case for LLMs to change business forever, GPT 5.4 is just as capable of achieving it as GPT 5.5.
Two issues with this. One, it's profitable assuming you just keep serving the same model forever, which is not realistic in this market. A given model has a shelf-life, which these days is measured in months, not years. Which means that trying to separate the cost of training the model from the cost of serving it doesn't make much business sense. And two, for providers that provide inference only via open weight models, the margins quickly move to commoditization. The "someday" when frontier model providers can enjoy their current high inference margins without the burden of significant training costs is never going to arrive.
Commoditization means there's price competition. From a consumer perspective, that's good. You want it to be a low-margin, high volume, competitive business.
Although from a business perspective, it can end up being ruinous competition like solar panels or airlines. A stable equilibrium with prices neither too low or too high isn't guaranteed; it depends on market structure.
It's anyone's guess whether this reaches an equilibrium or not, but I still expect that there will be companies like OpenRouter and Fireworks that offer inference at reasonable prices.
Agree fully.
The smaller models will only get better which push out the usefulness of older gpus.
What is the actual Return on Equity of Anthropic and OpenAI?
Profitable on a per token basis is meaningless. Ed Zitron doesn’t argue that it is impossible to offer profitable inference, he argues that the business as it stands today is deeply unprofitable and only getting worse because it isn’t a high-margin inference business.
Play out the most likely pessimist’s scenario: LLMs are useful but frontier models are overkill so businesses just use dirt cheap open weight models on their own hardware and/or they rent hardware instead of paying per token. Then what for OpenAI and Anthropic?
OpenAI’s business collapses if customers are happy with an LLM that costs $0.10 per million tokens even if it only costs OpenAI $0.05 in inference per million tokens. The insane bonkers claim from Garry Tan that in 2 years we will be using 90,000x as many tokens as today is… well, obviously not true.
The fixed costs that OpenAI and Anthropic have created need inference demand far beyond what is plausible.
edit: and hand waving away the vast losses of companies like OpenAI because of “training” is ridiculous. Anthropic are spending a billion dollars per month to rent additional capacity from xAI for inference, not training. The models don’t need to get better: if there is a case for LLMs to change business forever, GPT 5.4 is just as capable of achieving it as GPT 5.5.