Goal
By the end you should: (1) install a runner from the official site, (2) download a small model, (3) get one good answer, (4) record exactly what you ran.
Pick a runner
Prefer LM Studio if you want buttons and a chat window. Prefer Ollama if you are fine with a terminal or need a local API soon. See Ollama vs LM Studio for the tradeoffs.
Download only from the official sites linked on our tools pages, not random rebuilds.
Be honest about hardware
Check RAM in system settings. Rough starting range: models in the ~3B–8B class with a 4-bit-style quant unless you know you have headroom. This is a rule of thumb, not a guarantee.
Close heavy browser sessions. Local models eat memory. If the machine swaps to disk, the model is too big, not “broken AI.”
Pull one small model
In LM Studio: search a known family (Llama, Mistral, Qwen), pick a file the UI says should fit, download, load, chat.
In Ollama: `ollama pull` a small tagged model from their library, then `ollama run <tag>`. Copy the exact tag into your notes.
First prompt: explain a topic you already understand in five bullets. If it invents nonsense you can catch it.
Make it reproducible
Write: runner name and version if you know it, model family, size, quant/tag, OS, and whether GPU offload was on.
That paragraph is what turns “I played with AI” into evidence you can show someone else.
What to try next
Run the same prompts on a cloud chat (ChatGPT, Claude, or Gemini) and note quality versus privacy.
If you code, point a tiny script at the local OpenAI-compatible API. Keep secrets out of prompts either way.
If Ollama works and you want a browser chat layout, try Open WebUI pointed at your local Ollama service. Confirm one model still answers before you add the UI layer.
FAQ
Why is it slow?
CPU-only inference on a big context is slow. Shrink the model, shorten context, enable GPU layers if available, or use cloud when speed matters more than privacy.
Is local always private?
Local inference keeps weights and prompts on your machine if you do not enable cloud features. Still encrypt disks and follow policy for regulated data.