I recommend reading some foundational materials on federated learning and distributed inference. The fundamental issues are (1) compute (2) latency and (3) security.
On the compute side, you can't load most individual experts into even high-end consumer-grade cards (e.g. RTX 5090) because the weights are simply too big.
On the latency side, you need to ship the activations and wait for the somewhat underpowered cards to actually do the matrix multiplications, then send the results back. This means each token takes hundreds of milliseconds or even more, which is borderline unusable.
Finally if you're running a P2P network, the only way to verify that peers aren't lying to you is to run duplicate calculations - I don't see how SHA-256 or signed model state help here unless I'm missing something (What are you calculating the hash of? What exactly is signing attesting to?).
This is one of the few places where borrowing ideas from cryptocurrencies actually makes sense - you could imagine a system where peers place bonds and forfeit them if they lie about calculations. You can look at projects like Bittensor and others for how these kind of things are currently implemented.
There needs to be some sort of safeguard to prevent people from abusing this and running queries to do something an owner objects to. I'd feel horrible if I discovered that someone used my AI to figure out how to kill someone. Another problem would be privacy. My highly personal queries would be available to whomever ran that query for me thru logging.
This is not to say it's a bad idea, but it needs some thought toward protecting users or at least acknowledgment and warnings that their information might be available to unintended eyes.
I recommend reading some foundational materials on federated learning and distributed inference. The fundamental issues are (1) compute (2) latency and (3) security.
On the compute side, you can't load most individual experts into even high-end consumer-grade cards (e.g. RTX 5090) because the weights are simply too big.
On the latency side, you need to ship the activations and wait for the somewhat underpowered cards to actually do the matrix multiplications, then send the results back. This means each token takes hundreds of milliseconds or even more, which is borderline unusable.
Finally if you're running a P2P network, the only way to verify that peers aren't lying to you is to run duplicate calculations - I don't see how SHA-256 or signed model state help here unless I'm missing something (What are you calculating the hash of? What exactly is signing attesting to?).
This is one of the few places where borrowing ideas from cryptocurrencies actually makes sense - you could imagine a system where peers place bonds and forfeit them if they lie about calculations. You can look at projects like Bittensor and others for how these kind of things are currently implemented.
LLMs are already slow on one machine. Now you are going to add network calls to it?
There needs to be some sort of safeguard to prevent people from abusing this and running queries to do something an owner objects to. I'd feel horrible if I discovered that someone used my AI to figure out how to kill someone. Another problem would be privacy. My highly personal queries would be available to whomever ran that query for me thru logging.
This is not to say it's a bad idea, but it needs some thought toward protecting users or at least acknowledgment and warnings that their information might be available to unintended eyes.
Here is some feedback: don't put slop in your readme. It's unreadable.