I’m Alexandre, I have worked on Docker Swarm and the integration with distributed key value stores (etcd/consul/zookeeper), but also on Docker Swarmkit and the Raft distributed datastore, with the integration of etcd's Raft library.
Last July, I open sourced Mantissa (https://mantissa.io): a fully distributed scheduler and container orchestrator. I did this quietly to keep working on patches and stability. It is still at an experimental stage and will probably stay that way for a while, but could be tested to get a "look and feel" of the project.
Mantissa is using CRDTs[1] and Merkle Search Trees[2] for its data replication strategy (via anti-entropy/gossip). It is a bit of a departure from existing systems that are relying on strong consistency/linearizability (like Kubernetes, Swarm and Nomad): Mantissa focuses on fast convergence and self-stabilization instead. This comes with plenty of benefits (no quorum and Primary/Follower setup, simplified deployment and upgrades, etc.) but also trade-offs (increased storage size, no resource overcommit, temporary divergences in or outside of partition events, etc.) that would be too long to list and detail here.
Mantissa is a particular fit for small clusters, home labs, agent orchestration but could also be used in larger settings (I’d be comfortable to say up to a hundred nodes per cluster view/shard, but I’m making no claims for higher than this, since I don’t yet have access to hardware that would allow me to test and make such a claim). At the core was the idea that it should be trivial to interconnect machines together and create a transparent overlay on top of the operating system, regardless of the scale.
It is sort of a spiritual successor to Docker Swarmkit: going even further with simplification and reducing the deployment and maintenance burden. You'll see that it retains some of its design elements (with the node join flow) while taking inspiration from other systems like Google Omega[3].
Why building Mantissa while Kubernetes and other systems exist? I think it is worthwhile to experiment with new approaches and see how it may impact existing systems. Improving on the UX/DX of schedulers and orchestrators has been a long obsession of mine and Mantissa is a long story, with its initial codebase and architecture being 10 years old. The scope is incredibly large however and working on this alone is quite difficult even with the help of LLMs. AI is a double-edged sword, and I could write a full article around my frustrations with building complex distributed systems with LLMs (they have a hard time “reasoning” correctly about systems relying on convergence for example).
I’m having a lot of fun building this however and I hope this could be useful to some of you, despite its current limitations and experimental status. Feedback is also welcome to help steer the project in a direction where it could be the most useful/impactful to you.
I would think eventual consistency would be a big problem for orchestration. I am not seeing how it makes sense, too many edge cases that would make it even harder to reason about fleets, versions, load balancing, and the like - and then make sure what you want is actually what the system has.
I recall a google paper with a tl;dr like "distributed data; central control"
Hello HN,
I’m Alexandre, I have worked on Docker Swarm and the integration with distributed key value stores (etcd/consul/zookeeper), but also on Docker Swarmkit and the Raft distributed datastore, with the integration of etcd's Raft library.
Last July, I open sourced Mantissa (https://mantissa.io): a fully distributed scheduler and container orchestrator. I did this quietly to keep working on patches and stability. It is still at an experimental stage and will probably stay that way for a while, but could be tested to get a "look and feel" of the project.
Mantissa is using CRDTs[1] and Merkle Search Trees[2] for its data replication strategy (via anti-entropy/gossip). It is a bit of a departure from existing systems that are relying on strong consistency/linearizability (like Kubernetes, Swarm and Nomad): Mantissa focuses on fast convergence and self-stabilization instead. This comes with plenty of benefits (no quorum and Primary/Follower setup, simplified deployment and upgrades, etc.) but also trade-offs (increased storage size, no resource overcommit, temporary divergences in or outside of partition events, etc.) that would be too long to list and detail here.
Mantissa is a particular fit for small clusters, home labs, agent orchestration but could also be used in larger settings (I’d be comfortable to say up to a hundred nodes per cluster view/shard, but I’m making no claims for higher than this, since I don’t yet have access to hardware that would allow me to test and make such a claim). At the core was the idea that it should be trivial to interconnect machines together and create a transparent overlay on top of the operating system, regardless of the scale.
It is sort of a spiritual successor to Docker Swarmkit: going even further with simplification and reducing the deployment and maintenance burden. You'll see that it retains some of its design elements (with the node join flow) while taking inspiration from other systems like Google Omega[3].
Why building Mantissa while Kubernetes and other systems exist? I think it is worthwhile to experiment with new approaches and see how it may impact existing systems. Improving on the UX/DX of schedulers and orchestrators has been a long obsession of mine and Mantissa is a long story, with its initial codebase and architecture being 10 years old. The scope is incredibly large however and working on this alone is quite difficult even with the help of LLMs. AI is a double-edged sword, and I could write a full article around my frustrations with building complex distributed systems with LLMs (they have a hard time “reasoning” correctly about systems relying on convergence for example).
I’m having a lot of fun building this however and I hope this could be useful to some of you, despite its current limitations and experimental status. Feedback is also welcome to help steer the project in a direction where it could be the most useful/impactful to you.
Cheers.
Github: https://github.com/MantissaLabs/mantissa
[1] https://www.lip6.fr/Marc.Shapiro/papers/RR-7687.pdf [2] https://inria.hal.science/hal-02303490/file/paper%20%281%29.... [3] https://research.google/pubs/omega-flexible-scalable-schedul...
I would think eventual consistency would be a big problem for orchestration. I am not seeing how it makes sense, too many edge cases that would make it even harder to reason about fleets, versions, load balancing, and the like - and then make sure what you want is actually what the system has.
I recall a google paper with a tl;dr like "distributed data; central control"