Running LLMs Locally: Gemma, Ollama, and Practical Choices

Local LLMs should be treated as their own lane, not as a weaker copy of hosted frontier tools.

If your main goal is the best free hosted coding experience, start with the free stack. If your goal is privacy, offline access, or on-device control, start here.

Quick decision: local vs cloud

If you need…	Choose	Why
Sensitive data stays on your machine	Local	No external API calls
Offline access	Local	No internet dependency
Best available frontier reasoning	Cloud	Hosted frontier models still lead
Predictable marginal cost after setup	Local	Hardware replaces per-token billing
Fastest way to start coding with AI	Cloud	Lower setup friction

Where Gemma fits

Gemma belongs in the local/on-device bucket.

If you already know you want Gemma specifically, start with Gemma 4: Private Local AI From Phone to PC. The short version is simple: E4B is the sane default, E2B/E4B is the phone-class lane, and 26B A4B is the larger stretch target rather than the default on ordinary hardware.

That matters because people often ask one of two very different questions:

“What is the best free coding AI right now?”
“What can I run locally or on-device?”

Those are not the same decision.

Use Gemma-family models when:

privacy matters more than absolute frontier quality
you want offline access
you want to avoid turning every coding question into a hosted API bill
you are experimenting with smaller, more controllable local setups

Do not use local because it sounds ideologically cleaner. Use it because your constraints actually reward it.

Phone and on-device reality

If your target is phone-class or tablet-class hardware, keep your expectations tight:

smaller models matter more than benchmark bragging
runtime support matters more than model hype
latency and memory pressure become the real product

The practical way to think about this:

Laptop/desktop local is the default starting point
phone-class local is an advanced path that needs tighter model and runtime choices

If you specifically care about an iPhone-style local path, optimize for the smallest useful model and the simplest runtime first. Do not assume that a model people praise in the cloud will feel good on-device.

If you want the current concrete answer instead of general guidance, see the Gemma 4 guide.

Start here: pick one runtime

Choose one runtime and get it working before you optimize. The biggest mistake is mixing tools too early.

Ollama

Best default if you want the fastest local on-ramp and a simple local API server.

1
2
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3

Docs: Ollama documentation

LM Studio

Best if you want a desktop UI and quick model switching.

Docs: LM Studio docs

llama.cpp

Best if you want direct control and the broadest low-level local model ecosystem.

Project home: llama.cpp on GitHub

vLLM

Best if “local” really means shared GPUs, multi-user serving, or internal infrastructure.

Docs: vLLM documentation

Hardware planning

These are rough starting points for 4-bit quantized models.

Model size	Typical fit	What to expect
3B to 8B	6 to 10 GB VRAM or 16+ GB RAM	Works on laptops, good for simple tasks
10B to 20B	12 to 24 GB VRAM or 32+ GB RAM	Solid general use if prompts are short
30B to 70B	48+ GB VRAM or multi-GPU	Strong quality, heavy hardware

Notes:

Bigger context windows increase memory pressure quickly.
Apple Silicon uses unified memory, so your RAM budget is your real budget.
CPU-only local is fine for small models and patience.

How to choose a local model without wasting time

Start with your constraint: privacy, offline use, or cost control.
Choose the smallest model that fits your hardware comfortably.
Test on your real prompts, not leaderboard fantasy prompts.
Move up only when the smaller model fails in ways that actually matter.

Useful families to explore:

Gemma for the local/on-device lane
Qwen when you want a broad open model ecosystem
Llama for common runtime support
Mistral when you want another solid general local family

When local wins

working with sensitive code or documents
offline travel workflows
predictable ongoing cost after setup
simple helper tasks that do not need frontier reasoning

When cloud still wins

repository-scale reasoning with strong frontier quality
multimodal coding workflows where hosted models are much better
team workflows where setup and ops overhead matter more than privacy
“I need the answer fast and good right now” situations