are you asking what moat you can establish around a product that uses customer-facing LLM inference?
IMO, it's incredible the ability to build things rapidly, solo, and extremely cheaply. So as long as that's the case, the only moat left for you is customer access, customer loyalty, and pricing (though I think that last one is arguable)
The challenge as I see it is that for new products in this space to be successful, the solo dev either needs to BE or be partner with someone who has deep intimate domain knowledge and large network for the pain point their product solves (e.g. a 15 year supply chain manager for large manufacturing company rolls into solo dev a meaningful solution to a problem in their domain) or else the solo dev needs to already have a large social reach they can leverage to access customers.
Those are the kinds of moats I see being most effective.
By moat I just mean something a solo dev can build/pursue that doesn't get absorbed by vLLM/SGLang in few months. I'm fairly new to this space so not trying to get too deep into kernels or hardware.
Something around inference tooling: benchmarking or eval/regression checks for quantized models or something else. Something that's useful as a standalone tool/project or to learn the space properly. Just curious where there's still some opportunities.
are you asking what moat you can establish around a product that uses customer-facing LLM inference?
IMO, it's incredible the ability to build things rapidly, solo, and extremely cheaply. So as long as that's the case, the only moat left for you is customer access, customer loyalty, and pricing (though I think that last one is arguable)
The challenge as I see it is that for new products in this space to be successful, the solo dev either needs to BE or be partner with someone who has deep intimate domain knowledge and large network for the pain point their product solves (e.g. a 15 year supply chain manager for large manufacturing company rolls into solo dev a meaningful solution to a problem in their domain) or else the solo dev needs to already have a large social reach they can leverage to access customers.
Those are the kinds of moats I see being most effective.
Define a "Moat in LLM Inference".
By moat I just mean something a solo dev can build/pursue that doesn't get absorbed by vLLM/SGLang in few months. I'm fairly new to this space so not trying to get too deep into kernels or hardware.
Something around inference tooling: benchmarking or eval/regression checks for quantized models or something else. Something that's useful as a standalone tool/project or to learn the space properly. Just curious where there's still some opportunities.