Audience
AI engineers and infrastructure teams looking for a tool to lower latency, reduce compute cost, and scale throughput
About LMCache
LMCache is an open source Knowledge Delivery Network (KDN) designed as a caching layer for large language model serving that accelerates inference by reusing KV (key-value) caches across repeated or overlapping computations. It enables fast prompt caching, allowing LLMs to “prefill” recurring text only once and then reuse those stored KV caches, even in non-prefix positions, across multiple serving instances. This approach reduces time to first token, saves GPU cycles, and increases throughput in scenarios such as multi-round question answering or retrieval augmented generation. LMCache supports KV cache offloading (moving cache from GPU to CPU or disk), cache sharing across instances, and disaggregated prefill, which separates the prefill and decoding phases for resource efficiency. It is compatible with inference engines like vLLM and TGI and supports compressed storage, blending techniques to merge caches, and multiple backend storage options.
Other Popular Alternatives & Related Software
Amazon ElastiCache
Amazon ElastiCache allows you to seamlessly set up, run, and scale popular open-Source compatible in-memory data stores in the cloud. Build data-intensive apps or boost the performance of your existing databases by retrieving data from high throughput and low latency in-memory data stores. Amazon ElastiCache is a popular choice for real-time use cases like Caching, Session Stores, Gaming, Geospatial Services, Real-Time Analytics, and Queuing. Amazon ElastiCache offers fully managed Redis and Memcached for your most demanding applications that require sub-millisecond response times. Amazon ElastiCache works as an in-memory data store and cache to support the most demanding applications requiring sub-millisecond response times. By utilizing an end-to-end optimized stack running on customer-dedicated nodes, Amazon ElastiCache provides secure, blazing-fast performance.
Learn more about Amazon ElastiCache
BaseRT
BaseRT is a high-performance LLM inference runtime for Apple Silicon that lets developers pull models from Hugging Face, chat with them locally, or serve an OpenAI-compatible API from one CLI. Accelerated by hand-written Metal kernels, it is designed to deliver fast prefill and decode performance on M-series Macs, with published benchmarks showing up to 6.4× faster prefill than llama.cpp, 3.9× faster than MLX, and up to 1.33× faster decode. The basert CLI handles model downloading, conversion, interactive chat, serving, completion, benchmarking, inspection, and bundle signing. Its server supports chat, completions, embeddings, transcription, tool calls, continuous batching, paged KV caching, and prefix caching, while supported models can process text, vision, and audio. BaseRT uses its own .base model format with Q2–Q8 affine quantization, optional AWQ calibration, and signed bundles, and can convert GGUF, Hugging Face, and MLX checkpoints.
Learn more about BaseRT
oMLX
oMLX is a macOS-native MLX server designed to make local AI faster and more practical on Apple Silicon. Built for the way coding agents actually work, it uses paged SSD KV caching to persist cache blocks to disk, allowing previously seen prefixes to be restored across requests and server restarts instead of being recomputed from scratch. This can reduce time to first token on long contexts from 30–90 seconds to under five seconds after the first turn. Continuous batching handles concurrent requests through mlx-lm’s BatchGenerator, improving generation throughput without forcing requests to wait behind a single job. oMLX can serve LLMs, vision-language models, embedding models, and rerankers simultaneously, using LRU eviction when memory runs low. It supports any MLX-format model from Hugging Face, including Qwen, LLaMA, Mistral, Gemma, DeepSeek, MiniMax, and GLM, and can reuse models already stored in the standard Hugging Face cache, LM Studio folders, or custom directories.
Learn more about oMLX
DeepInfer
DeepInfer is an open-source inference framework for running DeepSeek language models on Linux systems with supported NVIDIA or Huawei Ascend accelerators. It provides offline inference and runtime components for model execution, optimized kernels, distributed parallelism, asynchronous batch scheduling, speculative decoding, deterministic generation, and structured outputs such as JSON Schema, regular expressions, and EBNF formats. The project documents DeepSeek-V4.1-Flash support and includes build instructions, configuration guidance, examples, and evaluation tools. It is intended for developers and infrastructure teams building or evaluating self-managed inference deployments. Its repository notes that production services may require additional KV-cache, prefill/decode disaggregation, and load-balancing components not included in this release.
Learn more about DeepInfer
Pricing
Starting Price:
Free
Free Version:
Free Version available.
Integrations
No integrations listed.
Company Information
LMCache
United States
lmcache.ai/
Videos and Screen Captures
Other Useful Business Software
Ship Agents Faster
Gemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
Product Details
Platforms Supported
Cloud
Training
Documentation
Live Online
Support
Online
LMCache Frequently Asked Questions
Q: What kinds of users and organization types does LMCache work with?
LMCache works with these users and organization types: Mid Size Business, Small Business, Enterprise, Freelance, Nonprofit, Government, and Startup.
Q: What languages does LMCache support in their product?
LMCache supports these languages: English.
Q: What kind of support options does LMCache offer?
LMCache offers support via online.
Q: What type of training does LMCache provide?
LMCache provides training in the form of documentation and live online.
Q: How much does LMCache cost?
Pricing for LMCache starts at Free.
