chapter 05

LM Studio, Ground Up

A real OpenAI-compatible backend running on your own machine.

Contents

1 / 10

Why start here instead of jumping straight to OpenAI or Anthropic? Three reasons:

Free - no API key, no credit card, no risk of an expensive mistake while learning.
Private - nothing leaves your machine.
It IS a real backend server, using the exact same request/response shapes as the big cloud providers - so everything you learn here transfers directly to Chapter 7.

There's a fourth reason, less obvious but arguably the most valuable one: running a model locally forces you to see the machinery that cloud APIs normally hide from you. You will watch RAM fill up when a model loads, watch generation slow down on a big model, and see the literal server logs of every request that hits it. That visibility is worth more than any diagram - this chapter is where Chapter 4's concepts stop being theoretical.

5.1 Install and get a model running

Download LM Studio from lmstudio.ai for your OS and install it normally.
Open it. Use the search tab to download a small model to start - something like a 3B–8B parameter model (search "llama 3.2 3b" or similar) so it runs comfortably even on modest hardware. Bigger models = better answers but slower and more RAM/VRAM needed.
Go to the chat tab, load the model, and send it a message to confirm it replies. This confirms the model itself works before we add networking into the mix.

   LM Studio app
   ┌───────────────────────────────┐
   │  [Search] → download a model   │
   │  [Chat]   → load model, test it│
   │  [Developer] → turn on server  │  ← we need this next
   └───────────────────────────────┘

A brief word on what "3B" and "8B" mean: they refer to the number of parameters (the tunable numeric weights) in the model, in billions. Roughly speaking, more parameters means the model can encode more nuance and knowledge - but it also means more math per token generated (slower) and more memory required just to hold the weights (each parameter typically needs 2 bytes of RAM/VRAM at the common "half precision" quantization LM Studio uses by default). An 8B model at that precision needs roughly 16 GB just to load, before you even start generating.

   parameter count   ≈ how much the model "knows" / how nuanced its answers can be
   quantization      ≈ how much each parameter is compressed to save memory
                        (a heavily quantized model is smaller/faster but slightly less accurate)

If a model refuses to load, or your machine grinds to a halt, that's almost always a memory problem - go back to the search tab and pick a smaller model or a more compressed ("quantized") version of the same model. LM Studio typically shows a compatibility indicator (green/yellow/red) per download based on your detected hardware - trust it.

Common mistake: downloading the biggest, most capable-sounding model first, then concluding "local LLMs are useless, they're way too slow." Start small (3B–8B), confirm the whole pipeline works end to end, and only scale up once you understand your hardware's limits.

5.2 Turn on the local server

Go to the Developer tab inside LM Studio and toggle the server on. By default it listens on:

http://localhost:1234

localhost means "this same computer" - nothing is exposed to the internet. 1234 is just the port number LM Studio picked; think of a port as a specific numbered door into your machine that a program is listening at.

   your machine
   ┌───────────────────────────────────────────────┐
   │   LM Studio process, listening at port 1234    │
   │   ┌─────────────────────────────────────────┐ │
   │   │ any program on THIS machine can knock    │ │
   │   │ on localhost:1234 and get an answer      │ │
   │   └─────────────────────────────────────────┘ │
   │   nothing outside this machine can reach it     │
   │   (by default - no firewall rule opens it up)   │
   └───────────────────────────────────────────────┘

Confirm it's alive from a terminal (not Python yet - just to prove the server itself works, same idea as the toy FastAPI server in Chapter 3):

curl http://localhost:1234/v1/models

You should get back JSON listing your loaded model(s), something shaped like this:

{
  "data": [{ "id": "llama-3.2-3b-instruct", "object": "model" }],
  "object": "list"
}

Copy the exact string in "id" - you will need it verbatim in every request in this chapter as the model field. It must match exactly, including capitalization and punctuation.

If this fails, the server toggle in LM Studio isn't on, or a firewall is blocking localhost traffic (rare). A quick sanity check: is LM Studio's window even showing a model loaded in the Developer tab's status area? The server can be "on" with zero models loaded, in which case /v1/models will return an empty list rather than erroring.

5.3 The trick: LM Studio speaks OpenAI's API dialect

LM Studio deliberately mimics OpenAI's request/response shape. This means the official openai Python package works against LM Studio, unmodified - you just point it at your own machine instead of OpenAI's servers.

pip install openai

python · sandbox

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:1234/v1",   # <- the only real change from talking to real OpenAI
    api_key="lm-studio",                    # LM Studio ignores this value, but the SDK requires *something* here
)

completion = client.chat.completions.create(
    model="use-the-model-identifier-shown-in-lm-studio",  # copy this from the Developer or Chat tab
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is 2 + 2?"},
    ],
    temperature=0.7,
)

print(completion.choices[0].message.content)

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Visual trace of exactly what just happened:

   YOUR PYTHON DICT (the messages list)
   [{"role":"system","content":"You are a concise assistant."},
    {"role":"user","content":"What is 2 + 2?"}]
            │
            │  openai SDK converts this to JSON and does an HTTP POST
            ▼
   POST http://localhost:1234/v1/chat/completions
   {"model": "...", "messages": [...], "temperature": 0.7}
            │
            │  travels to LM Studio's server, which runs the model
            ▼
   RESPONSE JSON (LM Studio sends this back)
   {
     "choices": [
       {"message": {"role": "assistant", "content": "2 + 2 = 4"}}
     ],
     "usage": {"prompt_tokens": 21, "completion_tokens": 6, "total_tokens": 27}
   }
            │
            │  openai SDK parses this JSON back into a Python object for you
            ▼
   completion.choices[0].message.content  →  "2 + 2 = 4"

This is the Chapter 3 "dict → JSON → HTTP → JSON → dict" pipe, now doing real work.

Notice the usage field in the response - this is Chapter 4.2's token counting made concrete. Even against a free local server, LM Studio still reports exactly how many tokens your prompt used and how many the reply used, because that's part of the standard response shape every chat.completions.create caller expects. Get comfortable reading it:

python · sandbox

print(f"prompt tokens:     {completion.usage.prompt_tokens}")
print(f"completion tokens: {completion.usage.completion_tokens}")
print(f"total tokens:      {completion.usage.total_tokens}")

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Against a paid provider in Chapter 7, this exact same usage object is what you'd multiply by a price-per-token to estimate cost - the shape doesn't change, only whether it costs you money.

Common mistake: copying the model string from documentation or from memory instead of from your own running LM Studio instance. Model identifiers are arbitrary strings chosen by whoever packaged the model file - always confirm with curl http://localhost:1234/v1/models (5.2) rather than guessing.

5.4 Streaming - getting the reply word-by-word

Recall from Chapter 3 that a slow response benefits from arriving in pieces. Set stream=True:

python · sandbox

stream = client.chat.completions.create(
    model="use-the-model-identifier-shown-in-lm-studio",
    messages=[{"role": "user", "content": "Count from 1 to 5."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

   WITHOUT streaming                       WITH streaming
   [.......... wait 3 seconds ..........]  "1" "," " 2" "," " 3" "," " 4" "," " 5"
   "1, 2, 3, 4, 5"  (all at once)          (each piece printed the instant it arrives)

Each chunk is one small JSON object; the loop above is exactly the yield-style generator pattern from Chapter 3.6, except this time the values are coming over the network instead of from your own code.

It's worth being precise about why streaming exists at all: from 4.1, you know the model generates one token at a time internally, regardless of whether you asked for streaming or not. Without stream=True, the server simply waits until the entire reply is finished internally, then sends you one big JSON blob at the end. With stream=True, the server forwards each token to you the instant it's produced, as a sequence of small JSON chunks instead of one big one. The model's own work is identical either way - streaming only changes when the network hands you the results.

   stream=False:  [model generates all N tokens] ────────────▶ [one JSON response sent]
   stream=True:   [token] → [send] [token] → [send] [token] → [send] ... (N small sends)

A more complete streaming example that also accumulates the full text (useful when you need both the live typing effect and the final string for further processing):

python · sandbox

def ask_streaming(prompt, model):
    full_reply = ""
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
            full_reply += delta
    print()  # newline once streaming finishes
    return full_reply

# result = ask_streaming("Name three planets.", model="your-model-id-here")

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Common mistake: forgetting that chunk.choices[0].delta.content can be None for some chunks (e.g. the very first or last chunk, which may only carry role/metadata information with no text). Always guard with if delta: before concatenating or printing, or you'll get a TypeError trying to add None to a string.

5.5 Embeddings from LM Studio (optional, but sets up Chapter 8's "next steps")

Load an embedding-capable model in LM Studio (search for one tagged "embedding," e.g. a nomic-embed-text style model), then:

python · sandbox

def get_embedding(text, model="the-embedding-model-identifier"):
    text = text.replace("\n", " ")
    return client.embeddings.create(input=[text], model=model).data[0].embedding

vec = get_embedding("Once upon a time, there was a cat.")
print(len(vec))          # e.g. 768 - a list of that many floats
print(vec[:5])           # a peek at the first 5 numbers

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

This is Chapter 4.6's "meaning becomes numbers" made real. Note that an embedding model and a chat model are two entirely different model files - LM Studio can have both loaded simultaneously (memory permitting), and you pick which one a given request uses purely via the model string, exactly as in 5.3.

You can combine this with Chapter 4.6's cosine_similarity function to build a tiny local semantic search, entirely offline:

python · sandbox

import math

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(y * y for y in b))
    return dot / (mag_a * mag_b)

documents = [
    "The cat sat on the mat.",
    "Stock markets fell sharply today.",
    "A feline rested on a rug.",
]

# in real code: doc_vectors = [get_embedding(d, model="...") for d in documents]
# then, for a query:
# query_vector = get_embedding("Tell me about cats", model="...")
# scores = [cosine_similarity(query_vector, dv) for dv in doc_vectors]
# best_match = documents[scores.index(max(scores))]

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

This is the entire core loop behind "chat with your PDFs" style tools: embed everything once, embed the question, find the closest match, paste it into the prompt.

5.6 A reusable, defensive version (real code doesn't assume success)

Chapter 1.6 promised you'd wrap network calls in try/except. Here it is, for real:

python · sandbox

from openai import OpenAI, APIConnectionError

client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

def ask(prompt: str, model: str, temperature: float = 0.7) -> str:
    try:
        completion = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=temperature,
        )
        return completion.choices[0].message.content
    except APIConnectionError:
        return "Could not reach LM Studio - is the local server toggled on?"

print(ask("Explain recursion in one sentence.", model="your-model-id-here"))

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Why bother wrapping this at all, when the happy path already works? Because in any real application, "LM Studio's toggle got switched off," "you loaded a different model and the ID changed," and "you closed LM Studio entirely" are not edge cases - they're things that will happen the first week you build anything on top of this. A bare, unguarded API call turns any of those into a stack trace that crashes your whole program; a guarded one degrades gracefully.

You can extend the same pattern to catch more specific failure shapes as your program grows:

python · sandbox

from openai import OpenAI, APIConnectionError, NotFoundError

client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

def ask_robust(prompt: str, model: str, temperature: float = 0.7) -> str:
    try:
        completion = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=temperature,
        )
        return completion.choices[0].message.content
    except APIConnectionError:
        return "Could not reach LM Studio - is the local server toggled on?"
    except NotFoundError:
        return f"Model '{model}' not found - check the exact id with curl /v1/models"
    except Exception as e:
        return f"Unexpected error: {e}"

print(ask_robust("Give me one fun fact.", model="your-model-id-here"))

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Common mistake: catching bare Exception as your only error handling, everywhere. It's fine as a last-resort fallback (as above), but catching the specific, well-known failure modes first (APIConnectionError, NotFoundError) lets you give the user - or yourself, debugging at 1am - a message that actually explains what went wrong.

5.7 Troubleshooting checklist

Symptom	Likely cause
`Connection refused` / `APIConnectionError`	Server toggle is off in the Developer tab
`model not found`	The `model` string doesn't match what LM Studio shows loaded - copy it exactly
Very slow replies	Model too large for your hardware; try a smaller one
Empty/garbled output	Wrong prompt template selected for that model in LM Studio's settings (usually auto-detected correctly)
Reply cuts off mid-sentence	Hit `max_tokens` - raise the limit or expect a continuation
First request is very slow, later ones are fast	Model is loading into memory on first use ("cold start"); this is normal
Machine becomes unresponsive / swaps to disk	Model too big for available RAM/VRAM; pick a smaller or more quantized model

A good debugging habit: whenever something goes wrong, re-run the plain curl command from 5.2 before touching any Python. If curl also fails or returns something unexpected, the problem is in LM Studio itself, not in your code - which saves you from chasing bugs in the wrong place.

5.8 What you now have

A working, local, free, private way to send a messages list and get a real model's reply back, in plain Python, using the industry-standard openai SDK shape. Chapter 6 turns this single call into a loop that can use tools and remember context - an agent.

Check yourself

Before moving to Chapter 6, make sure you can answer these without looking back:

What does base_url="http://localhost:1234/v1" actually change compared to talking to real OpenAI servers - and what does it not change?
Why does the api_key value not matter when talking to LM Studio, and why does the SDK still require you to pass something?
What's the practical difference between the model's own token-generation loop (4.1) and whether you set stream=True or stream=False?
If curl http://localhost:1234/v1/models fails, what should you check first, and why check it before touching Python at all?
Why can an embedding model and a chat model both be loaded in LM Studio at the same time, and how does your code choose which one a given request uses?
Why is try/except around a local, free, private API call still worth writing, if there's no billing risk and no network latency to a remote server?