oMLX Makes Local AI Coding Agents Faster on Apple Silicon Macs

Running an AI model locally on a Mac is no longer particularly difficult. Apple Silicon Macs have become capable local AI machines, and tools such as MLX make it relatively straightforward to run models without sending your prompts and files to the cloud.

The bigger challenge starts when you want to use those local models with AI coding agents.

Coding agents don’t simply send a short prompt and wait for an answer. They can build up enormous amounts of context containing system instructions, project files, previous commands, tool results and earlier conversation history. When a session resumes, the model may need to process tens of thousands of tokens again before generating its first response.

That’s where KV cache persistence becomes particularly interesting.

What Is the KV Cache and Why Does It Matter?

When a large language model processes a prompt, it creates internal data known as the key-value cache, or KV cache.

The cache stores information about tokens that the model has already processed. Instead of repeatedly calculating the same context from scratch, the inference engine can reuse this cached information.

For a normal chatbot conversation, this can already make a noticeable difference. For coding agents, the impact can be much larger.

An agent may have already processed:

• The system prompt
• Your project structure
• Source files
• Configuration files
• Tool instructions
• Previous commands
• Previous tool outputs
• A large conversation history

If that context disappears every time the server unloads the model or restarts, the model has to process it again.

And when you’re dealing with tens of thousands of tokens, that initial processing can become a significant bottleneck.

oMLX Keeps the KV Cache on the SSD

This is where oMLX takes a different approach.

Instead of keeping the KV cache exclusively in memory, oMLX can persist the cache to disk using the SafeTensors format.

That means previously processed context can be loaded from the SSD instead of being calculated again.

The important part is persistence.

The cache can survive the model being unloaded and can remain available even after restarting the inference server.

For coding agents, this can be particularly useful because the same project context is often processed repeatedly across multiple sessions.

In other words, rather than asking the model to reread and recalculate the same information every time, oMLX can reuse what it has already processed.

LM Studio Takes a Similar Approach, But With an Important Difference

oMLX isn’t the only local AI server to write MLX cache data to storage.

LM Studio also stores its MLX cache on disk, which provides a similar mechanism for avoiding unnecessary recomputation.

However, there is an important distinction between the two implementations.

LM Studio stores the cache in a temporary file that is removed when the model is unloaded.

Both applications therefore use the SSD to store cache data, but oMLX is designed to retain the cache rather than treating it as temporary data.

That difference becomes particularly relevant for long-running coding-agent workflows where the same context may need to be restored repeatedly.

Connect Claude Code Directly to oMLX

Another advantage of oMLX is its compatibility with common AI development tools.

The oMLX server exposes both OpenAI-compatible and Anthropic-compatible APIs, allowing applications to communicate with locally hosted models through localhost.

That means tools such as Claude Code can be configured to communicate directly with an oMLX server instead of sending inference requests to a remote service.

oMLX also provides a dashboard that displays the commands needed to connect Claude Code and other compatible clients to the local server.

This makes the setup considerably easier for users who don’t want to manually figure out API endpoints and environment variables.

The Disk Cache Can Consume a Significant Amount of Storage

Persistent caching is useful, but it isn’t free.

oMLX enables its disk cache by default and sets a default maximum size based on 10% of the capacity of the disk hosting the cache.

For example, on a 1TB SSD, the cache could potentially use up to approximately 100GB of storage.

That’s a substantial amount of space, particularly on a Mac with a relatively small internal SSD.

The good news is that you remain in control.

The cache can be cleared from the oMLX dashboard, and its maximum size can also be adjusted according to your available storage and workload.

If you’re running multiple large models or working with enormous coding projects, keeping an eye on available disk space is therefore a good idea.

Be Careful With pip Installations and Metal Kernels

There’s another issue that Mac users should be aware of when installing oMLX.

A standard installation through pip does not necessarily provide the compiled Metal kernels required for optimal performance.

Without the appropriate kernels, certain models can fall back to a more generic execution path.

This can have a significant impact on performance.

Models such as GLM-5.2, MiniMax M3 and Qwen3.5 are particularly relevant here, as users can end up using a generic path without immediately realizing that the optimized Metal kernels aren’t being used.

For that reason, using the official DMG packages is the safer approach when available. These packages include the required compiled Metal kernels, reducing the chance of accidentally running an optimized model through a slower fallback path.

For Apple Silicon users, this is an important distinction: installing the software successfully does not necessarily mean you’re getting the best possible inference performance.

Does Persistent KV Cache Actually Make the Model Faster?

There’s an important nuance here.

A persistent KV cache primarily improves time to first token, or TTFT.

In simple terms, it reduces the amount of work the model has to perform before it can start generating its response.

It does not automatically increase the model’s token generation speed once generation has started.

So there are really two different performance characteristics to consider:

Prefill performance: How quickly the model processes the existing context before responding.

Generation performance: How quickly the model produces new tokens after processing that context.

oMLX’s persistent cache primarily targets the first part.

If your coding agent regularly resumes sessions containing tens of thousands of tokens, reducing that repeated prefill work can make the interaction feel substantially more responsive.

Once the model starts generating, however, its output speed still depends on the model, quantization, memory bandwidth, GPU/Neural Engine capabilities and the particular Apple Silicon machine you’re using.

Why This Matters More for Coding Agents

For ordinary local AI conversations, persistent context caching is useful but may not always be essential.

Coding agents are a different story.

An agent can repeatedly work with the same repository while performing tasks such as:

• Reading source files
• Searching the codebase
• Running shell commands
• Inspecting build errors
• Modifying files
• Running tests
• Reviewing previous changes
• Calling external tools
• Continuing a previous task

All of those interactions can contribute to a large context window.

Without persistent caching, the server may have to process much of that context again after a model reload.

With persistent KV caching, the previously processed context can potentially be reused.

That’s why persistent KV caching can be particularly valuable for agentic AI workloads, even if the improvement isn’t obvious during a short, one-off conversation.

oMLX vs. Ollama and LM Studio for Agentic Workloads

Ollama and LM Studio remain excellent options for running local models on Macs. They are easy to install, widely supported and suitable for many everyday local AI workloads.

But the requirements change when the primary workload is an AI agent that maintains a large context.

For that scenario, oMLX’s persistent KV cache gives it an interesting advantage.

The comparison is essentially this:

The choice ultimately depends on what you’re doing.

If you mainly want a convenient local chatbot, all three can make sense.

If you’re building a workflow around large-context coding agents on Apple Silicon, oMLX’s approach to persistent KV caching is considerably more interesting.

What You Should Know Before Trying oMLX

There are a few things worth keeping in mind before switching your local AI setup.

First, the persistent cache consumes disk space. On a large SSD, that may not matter much. On a Mac with limited storage, it can become significant.

Second, cache persistence doesn’t magically make every model faster. Its biggest benefit is reducing the amount of context that has to be processed again.

Third, installation matters. Using an installation that lacks the appropriate Metal kernels can prevent some models from taking advantage of Apple’s hardware acceleration.

Finally, the biggest gains are likely to appear in long-context and agentic workflows, rather than short conversations where only a few thousand tokens are processed.

Conclusion

Local AI on Apple Silicon is becoming increasingly sophisticated, but the next performance bottleneck isn’t always raw inference speed.

For coding agents, repeatedly processing the same enormous context can waste considerable time before the model even generates its first token.

oMLX tackles that problem by making the KV cache persistent on disk, allowing previously processed context to be reused instead of recalculated after a model unload or server restart.

That doesn’t make the model generate tokens faster once it starts responding, but it can significantly reduce the waiting time before the first token appears.

For developers running large local models and AI coding agents on Apple Silicon Macs, that’s a meaningful advantage. Add OpenAI and Anthropic-compatible APIs, Claude Code integration and optimized Metal kernels through the official packages, and oMLX becomes an especially compelling option for serious local agentic AI workloads.

If your goal is simply to chat with a local model, you may not need it. But if you’re asking a local AI agent to repeatedly understand a large codebase, persistent KV caching could be one of the more useful optimizations you can make.