LM Studio – To run LLMs locally and use them directly in your code

With everything that’s currently happening on the OpenAI side, I’m exploring free alternatives to seamlessly integrate into my existing scripts without requiring extensive rewriting. It’s simply a precautionary measure to ensure that I’m not caught off guard in case the quality of service on the ChatGPT side drops.

To achieve this, I’m looking for a language model and a tool that allows me to convert that model into an API, which I can then call in my code.

At the moment, all of this is in the research and development phase, but I thought some feedback would be appreciated. Therefore, I opted for an OpenChat model, which is supposed to be as effective as ChatGPT 3.5. So far, it hasn’t been too complicated

I successfully ran this template in Llamacpp without any issues in discussion mode. Subsequently, I sought a bridge to access APIs, and that’s when I discovered Llama-cpp-python with its Server option. Unfortunately, it never functioned properly at home due to dark x64/ARM64 incompatibilities, even with pyenv. In brief…

Due to time constraints, I recalled that I could achieve this with Ollama, a detail I had completely overlooked.

Subsequently, other readers of Easy Tech Tutorials recommended a tool called LM Studio, which I will discuss in this article.

LM Studio is a tool compatible with macOS, Windows, and Linux that facilitates the downloading and local execution of LLMs (Large Language Models). This allows you to engage in conversations with these models, similar to interacting with ChatGPT.

However, the tool offers more than just basic functionality. It includes various settings, including support for Silicon Macs, to optimize the model. The feature that intrigued me the most was the ability to run a local server that provides an API identical to ChatGPT’s.

This makes it possible, without a lot of changes in your code, to switch OpenAI services to local AI in a seamless or almost transparent way for you to use OpenAI lib 0.28.1

pip install openai==0.28.1

Here’s an example code that shows how to call it in Python:

import os
import openai

openai.api_base = "http://localhost:1234/v1" 
openai.api_key = "" 

completion = openai.ChatCompletion.create(
  model="local-model",
  messages=[
    {"role": "system", "content": "Always answer in rhymes."},
    {"role": "user", "content": "Introduce yourself."}
  ]
)

print(completion.choices[0].message)

No API key is required, and there’s no need to overhaul your code—simply migrate to LM Studio. Plus, it’s free 🙂

In my testing, I found that while the tool responds correctly in the ‘chat’ version, the configuration in the server version is a bit different. So, I still need to tweak the settings to find the optimal training for my AI. But I’m almost there.

If you’re interested in trying LM Studio, this is where it happens.