Ask HN: Why aren't LLM token CDN cache's a thing?

1 comments

• minimaxir an hour ago

  LLM caches only work for 100% input matches for all previous text, which will never happen outside of a single session.

  Things such as system prompts are already cached.