Run Local LLMs with Ollama + Open WebUI in About 20 Minutes

The first time you set up a local LLM, it is easy to make the stack more complicated than it needs to be.

You find a model on Hugging Face. Then you need a runtime. Then CUDA. Then Python dependencies. Then a web interface. Then something is inexplicably using 28 GB of RAM while your GPU is sitting at 40% utilization.

For most developers, I would not start there.

A much cleaner setup is:

• Ollama → local model runtime
• Open WebUI → browser-based interface

Ollama handles downloading, loading, and serving the model. Open WebUI talks to Ollama through its API and gives you a familiar chat interface.

This separation matters: if Open WebUI breaks, your models still work through Ollama. If you later replace the interface with a coding agent or your own application, Ollama can remain the backend.

When you use only local models and local features, inference can remain on your machine. That lets you experiment with private code, documents, prompts, and development workflows without sending requests to a third-party model provider. External providers, web search, and connected tools can still transmit data outside your machine.

If Docker is already installed, the hands-on setup should take about 20 minutes. Installing Docker and downloading model images can take considerably longer depending on your machine and connection.

What We Are Building

By the end of this tutorial, the stack will look like this:

In this configuration, Ollama runs natively on the host and can use its GPU support directly, while Open WebUI runs as replaceable Docker infrastructure. Keeping inference separate from the interface also makes each layer easier to diagnose.

Open WebUI does provide an all-in-one Docker image containing Ollama, but for a development workstation I generally prefer keeping them separate. Open WebUI's own documentation supports both configurations. External link: Open WebUI — Quick Start (opens in a new tab)

1. Check Your Hardware Before Installing Anything

You do not need a high-end GPU to run a local LLM.

You do, however, need to choose models appropriate for the machine you have.

The three resources that matter most are:

• VRAM — memory physically available on your GPU
• RAM — normal system memory
• Disk space — where model weights are stored

For NVIDIA users, open PowerShell, Command Prompt, or a terminal on your computer and run:

nvidia-smi --query-gpu=name,memory.total

NVIDIA GeForce RTX 4080, 16376 MiB

On Windows, you can also check Task Manager → Performance → GPU.

For system RAM, open PowerShell, Command Prompt, or a terminal on your computer and run:

systeminfo | findstr /C:"Total Physical Memory"

Total Physical Memory:         64,735 MB

Or just check Task Manager → Performance → Memory.

A practical starting point

These are rough starting points rather than hard limits. They assume an approximately 4-bit quantized model, a modest context length, and enough headroom for runtime allocations:

Do not treat parameter count as a direct measurement of memory usage.

A 12-billion-parameter model does not automatically require 12 GB of VRAM. Quantization can dramatically reduce the memory required for the model weights.

But the model weights are only part of the equation.

Context length, KV cache, runtime buffers, architecture, concurrent requests, and other allocations all consume additional memory.

That distinction becomes important later.

2. Install Ollama

Ollama supports Windows, macOS, and Linux. External link: Ollama Quickstart documentation (opens in a new tab)

Windows

Current Ollama documentation requires Windows 10 22H2 or later. NVIDIA and AMD Radeon GPUs are supported. External link: Ollama Windows documentation (opens in a new tab)

To install Ollama:

1. Go to the official External link: Ollama download page (opens in a new tab) and download the Windows installer.
2. Open the downloaded installer and follow the setup prompts.

Ollama installs as a native Windows application and makes the ollama command available from PowerShell and Command Prompt.

The Ollama API listens on:

http://localhost:11434

Once installation is complete, open PowerShell or Command Prompt and run:

ollama -v

# If Ollama is currently running:
ollama version is <installed version>

# If Ollama is not running:
Warning: could not connect to a running Ollama instance
Warning: client version is <installed version>

# The exact version number will vary.

If Ollama is not already running as a background service, you can start the Ollama server manually with:

ollama serve

Keep that terminal window open while Ollama is running. Then open a second PowerShell or Command Prompt window and verify the installation with:

ollama -v

# If Ollama is installed correctly and the `ollama serve` command worked:
ollama version is <installed version>

If a version number appears, the Ollama CLI is installed.

macOS

Ollama supports macOS Sonoma 14 or newer. Apple Silicon Macs can use both CPU and GPU acceleration, while Intel Macs run Ollama on the CPU only. External link: Ollama macOS documentation (opens in a new tab)

To install Ollama:

1. Go to the official External link: Ollama download page (opens in a new tab) and download the macOS installer.
2. Open the downloaded ollama.dmg file.
3. In the window that appears, drag Ollama into the Applications folder.
4. Open Applications in Finder and launch Ollama.
5. The first time Ollama starts, macOS may ask you to confirm that you want to open the downloaded application. Approve the prompt.
6. Ollama will check whether its command-line tool is available in your system PATH. If it is not, Ollama may ask for permission to create a command-line link in /usr/local/bin. Allow this so you can use the ollama command from Terminal. External link: Ollama macOS documentation (opens in a new tab)

Once Ollama is running, open the Terminal app on your Mac. You can find it under Applications → Utilities → Terminal, or press Command + Space and search for Terminal.

Verify the installation by running:

ollama -v

# If Ollama is installed correctly:
ollama version is <installed version>

# The exact version number will vary.

If Terminal displays an Ollama version number, the installation is complete.

Linux

On Linux, Ollama is installed from the command line using the official installation script.

Open your distribution's Terminal application, then run:

curl -fsSL https://ollama.com/install.sh | sh

This downloads and installs Ollama using the official installer. Ollama also recommends configuring it as a startup service on Linux so it can run automatically in the background. External link: Ollama Linux documentation (opens in a new tab)

Once the installation finishes, verify that the Ollama command is available by running:

ollama -v

# If Ollama is installed correctly and is currently running:
ollama version is <installed version>

# If Ollama is installed correctly but not currently running:
Warning: could not connect to a running Ollama instance
Warning: client version is <installed version>

# The exact version number will vary.

If Ollama is not already running as a background service, you can start the Ollama server manually with:

ollama serve

Keep that terminal window open while Ollama is running. Then open a second Terminal window or tab and verify the installation with:

ollama -v

# If Ollama is installed correctly and the `ollama serve` command worked:
ollama version is <installed version>

If a version number appears, the Ollama CLI is installed.

The official External link: Ollama Linux documentation (opens in a new tab) also provides separate installation instructions for ARM64 systems and an additional ROCm package for supported AMD GPUs.

For the latest Linux installation instructions, see the External link: official documentation (opens in a new tab).

3. Make Sure the Ollama API Is Alive

Before downloading a model, verify that the backend itself works.

In your OS’s terminal, run:

curl.exe http://localhost:11434/api/tags

{"models":[...]}

If Ollama is healthy, you should receive JSON containing the locally installed models.

On a fresh install that list may simply be empty.

Ollama exposes its local API at:

http://localhost:11434/api

The /api/tags endpoint returns the locally available models. External link: Ollama API — List models (/api/tags) (opens in a new tab)

This is a useful distinction when debugging:

Debug one layer at a time.

4. Download Your First Model

For the initial test, resist the urge to immediately download the largest model your machine can theoretically run.

We are verifying the stack first.

A good example is Gemma 3:

ollama pull gemma3:4b

Ollama's CLI uses ollama pull to download models and ollama run to execute them. External link: Ollama CLI Reference (opens in a new tab)

Once the download finishes, run it:

ollama run gemma3:4b

You should get an interactive prompt.

Try something simple:

Explain dependency injection in two paragraphs.

If the model answers, the important part of the system is already working.

Exit the interactive session by entering /bye when finished.

5. Inspect What Ollama Actually Loaded

This is a diagnostic step I would not skip.

Run:

ollama ps

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma3:4b    a2af6cc3eb7f    2.9 GB    100% GPU     4096       4 minutes from now

This shows the models currently loaded by Ollama.

More importantly, it helps you see where the model is executing.

Ollama specifically recommends checking the PROCESSOR information reported by ollama ps when diagnosing model offloading and context allocation.

External link: Ollama FAQ — How can I tell if my model was loaded onto the GPU? (opens in a new tab)

Ideally, a GPU-equipped machine will show the model running primarily or entirely on the GPU.

If a model cannot fit into VRAM, Ollama may offload some computation into system memory.

That can make the difference between:

Wow, local AI is actually fast.

Why is each token taking forever?

CPU fallback is not necessarily a failure.

It is often a capacity problem.

And that distinction saves a lot of pointless troubleshooting.

6. Install Docker Desktop

The easiest supported way to run Open WebUI is Docker.

Open WebUI currently recommends Docker for most installations because it isolates Open WebUI and its dependencies from the host environment. External link: Open WebUI — Quick Start (opens in a new tab)

Install Docker Desktop if it is not already installed:

External link: Docker Desktop (opens in a new tab)

After Docker starts, verify it:

docker --version

# If Docker is installed correctly:
Docker version 29.5.3, build d1c06ef

# The exact version and build numbers will vary.

Then:

docker ps

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

If both commands work, Docker is ready.

7. Start Open WebUI

First, generate a persistent application secret. Use the command for your shell, save the resulting value in a password manager, and reuse the same value whenever you recreate the container.

export WEBUI_SECRET_KEY="$(openssl rand -hex 32)"

$env:WEBUI_SECRET_KEY = [Convert]::ToHexString([Security.Cryptography.RandomNumberGenerator]::GetBytes(32)).ToLower()

Run:

docker run -d \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  -e WEBUI_SECRET_KEY="$WEBUI_SECRET_KEY" \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

On PowerShell, you can put the command on one line:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY="$env:WEBUI_SECRET_KEY" --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Open WebUI publishes an official Docker image and currently recommends its Docker deployment for most users. External link: Open WebUI — Docker Deployment Documentation (opens in a new tab)

The important pieces from the above command are:

-p 3000:8080

Maps port 3000 on your computer to Open WebUI.

-v open-webui:/app/backend/data

Creates persistent storage for Open WebUI.

The named volume preserves chats and configuration when you replace the container. The persistent WEBUI_SECRET_KEY preserves sessions and access to encrypted credentials; losing or changing it can invalidate sessions and make stored OAuth tokens or API keys unreadable.

And:

--add-host=host.docker.internal:host-gateway

Allows the container to reach services running on the host machine.

That last part matters because Ollama is not inside the Open WebUI container.

8. Open the Interface

In a web browser, open:

http://localhost:3000

The first account created becomes the administrator for the local Open WebUI installation.

Open WebUI states that account information and application data are stored locally by default rather than being sent to Open WebUI. External link: Open WebUI — FAQ (opens in a new tab)

Once you log in, Open WebUI will usually detect Ollama automatically.

Your installed model should appear in the model selector.

Select it.

Send a message.

If you receive a response, the stack is working.

9. If Open WebUI Cannot Find Ollama

This is probably the most common problem with this architecture.

You try:

http://localhost:11434

inside Open WebUI.

And nothing happens.

The reason is Docker networking.

From inside the Open WebUI container:

localhost

means:

the Open WebUI container

It does not mean your Windows, macOS, or Linux host.

Instead, configure the Ollama connection as:

http://host.docker.internal:11434

In Open WebUI, go to:

Settings → Admin → Connections

and check the Ollama API connection.

Open WebUI explicitly recommends host.docker.internal:11434 when Ollama runs on the Docker host. External link: Open WebUI — Connect to Ollama (opens in a new tab)

This one networking distinction resolves a surprising number of broken Open WebUI installations.

Native Linux Docker note

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

sudo systemctl daemon-reload
sudo systemctl restart ollama

10. Verify the Entire Stack

At this point I like to verify each layer independently.

Test 1 — Is Ollama installed?

ollama -v

Test 2 — Does its API respond?

curl.exe http://localhost:11434/api/tags

Test 3 — Is the model installed?

ollama ls

Test 4 — Can Ollama run it?

ollama run gemma3:4b

Test 5 — Where is it running?

ollama ps

Test 6 — Is Open WebUI running?

docker ps

You should see:

open-webui

Test 7 — Can Open WebUI reach Ollama?

Open:

http://localhost:3000

Select the model and send a prompt.

This gives us a much more useful definition of model health than simply saying, "the page loaded."

Performance and Capacity Tuning

Once the system works, model choice and memory capacity become the main performance variables. An 8B model may run smoothly while a larger model technically starts but generates much more slowly.

That slowdown often means the model and its runtime state crossed the GPU memory boundary, forcing some work into system RAM and CPU execution. Check the processor split before assuming the runtime is broken.

Run:

ollama ps

Ollama's model scheduler measures memory requirements and attempts to maximize GPU utilization while avoiding out-of-memory failures. Its newer scheduling system also reports memory utilization more accurately. External link: Ollama — New model scheduling (opens in a new tab)

So if performance suddenly falls apart after switching models, ask:

Did I change software, or did I simply ask the hardware to hold more than it can fit?

Those are very different problems.

Context Length Can Quietly Consume VRAM

Model size gets most of the attention.

Context length can quietly wreck your memory budget too.

The context window determines how much information the model can keep available during a conversation or request.

Larger context means more memory consumption.

Ollama currently chooses default context sizes based partly on available VRAM:

Ollama notes that workloads such as coding agents, web search, and other agentic workflows can benefit from contexts of 64K or more—but increasing context also increases memory requirements. External link: Ollama — Context length (opens in a new tab)

A model may support a 128K context window without that setting being practical on your GPU. Supported by the model and sensible for the available memory are different constraints.

If you do not need 128K tokens, you may spend several gigabytes of memory for no practical benefit. Start with a smaller context and increase it only when the workload requires it.

Why System RAM Still Matters

GPU users sometimes ignore system RAM entirely.

Do not make this mistake.

System RAM matters when:

• model layers are offloaded from the GPU
• multiple models are loaded
• Docker and other development applications are running
• the operating system starts paging
• large context windows are used

Once Windows, Docker, your IDE, browser tabs, Ollama, and your local model are fighting over the last couple gigabytes of system RAM, the machine can become dramatically slower.

Watch both VRAM and system RAM during testing.

For development machines doing serious local AI experimentation, 32 GB of RAM is far more comfortable than 16 GB.

64 GB becomes useful if you regularly test larger models or intentionally rely on partial CPU offloading.

Choose Responsiveness Over Maximum Model Size

There is a temptation with local AI to treat the largest model that starts successfully as the correct model.

That is backwards.

A smaller model that runs entirely on the GPU at high throughput can be much more useful than a theoretically smarter model that crawls because half of it is being shuffled through system memory.

For development work, I would optimize for:

—not model size.

Try several models against the actual tasks you care about:

• Code explanation
• Refactoring
• Documentation
• Structured JSON
• Brainstorming
• Classification
• RAG

A smaller model may already solve the problem.

Make the Setup Reproducible

There is one final problem with our quick setup.

We used:

ghcr.io/open-webui/open-webui:main

That tag moves.

Open WebUI's documentation explicitly notes that :main and :latest are rolling images. Version tags such as :vX.Y.Z are immutable and are recommended for reproducible or production deployments. External link: Open WebUI — Quick Start / Docker deployment (opens in a new tab)

That means once you have a known-good environment, replace:

:main

with a specific Open WebUI release:

:vX.Y.Z

Do the same conceptually with your models.

Record:

Model
Model tag
Quantization
Context length
Ollama version
Open WebUI version

For example:

Runtime: Ollama
Model: gemma3:4b
Quantization: Q4_K_M
Context: 8192
Open WebUI: pinned release

Record these details because “we are using Gemma” or “we are running the latest Open WebUI” does not identify a reproducible environment.

Because:

If this stack eventually becomes part of a development workflow, automation pipeline, benchmark, or application, versioning matters just as much here as it does with any other dependency.

Diagnose Models Through the API

Ollama can expose detailed information about an installed model through its API.

For example:

Invoke-RestMethod -Method Post `
  -Uri "http://localhost:11434/api/show" `
  -ContentType "application/json" `
  -Body '{"model":"gemma3:4b"}'

The response can include information such as:

parameter size
quantization
model architecture
context capacity
capabilities

Ollama documents these fields through its /api/show endpoint. External link: Show model details — /api/show (opens in a new tab)

That makes the runtime itself a useful diagnostic tool rather than just a black box.

You can also inspect running models through:

curl http://localhost:11434/api/ps

The response includes fields such as model size, VRAM allocation, context length, and quantization information. External link: List running models — /api/ps (opens in a new tab)

Privacy and Security Boundaries

Running a model locally gives you a significant privacy advantage:

There does not need to be a cloud inference provider anywhere in that path.

But local does not automatically mean secure.

Ollama's local API does not require authentication when accessed at localhost:11434. External link: Ollama API Authentication (opens in a new tab)

That is perfectly reasonable for a service bound to your own machine.

It is not something I would casually expose to the public internet.

The same applies to Open WebUI.

Treat these as local development services unless you intentionally configure authentication, networking, firewalls, TLS, and access control for remote use.

Privacy comes from architecture.

Not from the word "local."

Where We Ended Up

We now have a surprisingly capable local AI environment:

More importantly, each piece has one job:

• Ollama manages inference.
• Open WebUI manages interaction.
• Docker isolates the interface.
• Your hardware determines the practical model and context limits.

That makes this a good foundation for much more than chatting with a local model.

From here, Ollama can become the backend for:

• local coding assistants
• IDE integrations
• Python applications
• REST API experiments
• RAG pipelines
• autonomous agents
• MCP tools
• internal developer utilities
• offline AI workflows

And when something gets slow, you now have somewhere concrete to look:

ollama ps

1. Check the model.
2. Check the processor split.
3. Check the VRAM.
4. Check the context.

Because with local LLMs, the most common performance problem is not that the model is broken.

It is that the model you asked the machine to run no longer fits comfortably inside the machine you asked to run it.