I originally built this to help with self-hosting Hugging Face models, but it evolved into an authenticated pull-through cache.
A big part of this project was an experiment to see how far I could get Cursor + (Opus/Fable) to optimize the hot path. I initially used fasthttp and Go's net/http, but after some deep optimizations, the only way to squeeze out more performance was to eliminate syscalls.
I ended up building two optimization paths that use a custom-built HTTP/1.1 parser: one for macOS using sendfile + sf_hdtr, and one for Linux using io_uring. For anyone interested in the threat model or how the custom parser is tested against Go's standard library, I wrote a detailed security breakdown here: https://pulsys.io/docs/security/
The result is that it can sustain 1.36M req/s at 4 KiB and 90 GB/s at 16 MiB on an EC2 instance (see benchmarks https://pulsys.io/docs/benchmarks/)
It drops right in front of existing clients (just set HF_ENDPOINT).
I’d love to hear your thoughts, feedback, or any questions on the AI-assisted optimization process!
I’m the creator of Pulsys.
I originally built this to help with self-hosting Hugging Face models, but it evolved into an authenticated pull-through cache.
A big part of this project was an experiment to see how far I could get Cursor + (Opus/Fable) to optimize the hot path. I initially used fasthttp and Go's net/http, but after some deep optimizations, the only way to squeeze out more performance was to eliminate syscalls.
I ended up building two optimization paths that use a custom-built HTTP/1.1 parser: one for macOS using sendfile + sf_hdtr, and one for Linux using io_uring. For anyone interested in the threat model or how the custom parser is tested against Go's standard library, I wrote a detailed security breakdown here: https://pulsys.io/docs/security/
The result is that it can sustain 1.36M req/s at 4 KiB and 90 GB/s at 16 MiB on an EC2 instance (see benchmarks https://pulsys.io/docs/benchmarks/)
It drops right in front of existing clients (just set HF_ENDPOINT).
I’d love to hear your thoughts, feedback, or any questions on the AI-assisted optimization process!