LocalAI setup guide for a VPS
LocalAI is a drop-in OpenAI API replacement that runs local models on CPU or GPU. It speaks /v1/chat/completions, /v1/embeddings, /v1/audio/transcriptions, and /v1/images/generations — so any tool built for OpenAI works against your server without code changes.
When to pick LocalAI over Ollama
Both are great. The honest comparison:
- LocalAI: broader API surface (audio, image, embeddings, function calling — not just chat). Bigger Docker image (~6 GB). Lower-level configurability.
- Ollama: simpler. Easier model management (
ollama pull qwen2.5:14b). Smaller install. Chat + embeddings API.
Pick LocalAI if you need audio transcription (Whisper), image generation, or you want a single API surface across multiple model types. Pick Ollama if you just need chat / completion / embeddings and want the simpler tool.
Prerequisites
- Ubuntu 22.04 or 24.04 VPS.
- At least 8 GB RAM for a useful chat model. 16 GB recommended.
- Docker + Compose installed — see Docker install.
- Patience for the first model pull — multi-gigabyte downloads.
Tested on Ubuntu 22.04, Docker CE 27.3, LocalAI 2.21, on a 16 GB Inferno VPS CPU-only (AMD EPYC).
Step 1 — Compose file
mkdir -p ~/localai && cd ~/localai
nano docker-compose.yml
name: localai
services:
localai:
image: localai/localai:v2.21-aio-cpu
restart: unless-stopped
ports:
- "127.0.0.1:8080:8080"
volumes:
- localai_models:/build/models
environment:
DEBUG: "false"
THREADS: "4"
CONTEXT_SIZE: "4096"
MODELS_PATH: "/build/models"
deploy:
resources:
limits:
memory: 14G
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
volumes:
localai_models:
The aio-cpu tag bundles a curated set of models that work well on CPU — chat, embeddings, Whisper, a small image generator. For GPU swap to aio-gpu-nvidia-cuda-12 and add runtime: nvidia + deploy.resources.reservations.devices.
Step 2 — Bring it up
docker compose up -d
docker compose logs -f localai
The first start downloads several gigabytes of models. Patience — this is a one-time cost. Once you see "Starting LocalAI using 4 threads, with models path: /build/models," it's ready.
Step 3 — Test the OpenAI-compatible API
# List models
curl http://127.0.0.1:8080/v1/models
# Chat completion
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "What is the OSI model?"}],
"stream": false
}'
Yes — you can pass "model": "gpt-4" and LocalAI maps it to its configured backend. That's the whole point: tools sending gpt-4 in the model field "just work."
Step 4 — Embeddings and audio
# Embeddings
curl http://127.0.0.1:8080/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"model": "text-embedding-ada-002", "input": "Hello world"}'
# Whisper transcription
curl http://127.0.0.1:8080/v1/audio/transcriptions \
-H "Content-Type: multipart/form-data" \
-F file="@/path/to/audio.mp3" \
-F model="whisper-1"
Step 5 — Nginx + HTTPS + token auth
Same pattern as our Ollama HTTPS setup:
sudo nano /etc/nginx/sites-available/localai
map $http_authorization $auth_ok {
default 0;
"Bearer YOUR_LONG_RANDOM_TOKEN" 1;
}
server {
listen 80;
server_name localai.example.com;
location /.well-known/acme-challenge/ { root /var/www/html; }
location / {
if ($auth_ok = 0) { return 401 "unauthorized\n"; }
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
proxy_read_timeout 600s;
client_max_body_size 100M;
}
}
sudo ln -s /etc/nginx/sites-available/localai /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d localai.example.com --redirect
Step 6 — Use it from any OpenAI SDK
# Python
from openai import OpenAI
client = OpenAI(
base_url="https://localai.example.com/v1",
api_key="YOUR_TOKEN",
)
resp = client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)
# JS
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "https://localai.example.com/v1",
apiKey: "YOUR_TOKEN",
});
const r = await openai.chat.completions.create({
model: "gpt-4",
messages: [{ role: "user", content: "Hello" }],
});
console.log(r.choices[0].message.content);
Works with LangChain, LlamaIndex, Continue.dev, Aider, and most other LLM tooling.
Loading custom models
Drop GGUF files into the models volume:
docker compose cp ./my-model.gguf localai:/build/models/
Then create a YAML config under /build/models/my-model.yaml describing the model. LocalAI's docs cover the YAML schema in detail. For most users the AIO bundle is enough.
Resource and concurrency tuning
THREADS— number of CPU threads per inference. Set to about (vCPU count - 1). Higher isn't always faster — contention kicks in.CONTEXT_SIZE— max context window. Bigger = more RAM per request. 4096 is a sane default for chat; raise for RAG.F16/ quantization — control with model YAML; AIO defaults are sensible.
Common LocalAI mistakes
"OOM on first prompt"
The configured model + context size doesn't fit in RAM. Drop to a smaller model (or smaller context), or add swap.
"Model not found"
You requested a model name LocalAI doesn't have configured. Check /v1/models for the actual list. The AIO bundle aliases common OpenAI names; non-AIO setups need explicit aliases.
"Slow but everything else looks healthy"
Wrong THREADS value (often left at 1 by default). Set explicitly.
"Image generation hangs"
The CPU image-gen backend is genuinely slow — minutes per image. Use a GPU image for production image generation, or ComfyUI directly.
VPS for a LocalAI stack?
The aio-cpu image needs 16 GB to be comfortable; 8 GB works for chat-only with smaller models. Inferno's 16 GB and 32 GB tiers fit well.
What to do next
- Try Ollama if you only need chat.
- Open WebUI as a chat front-end — also speaks OpenAI API.
- Wire LocalAI into n8n for self-hosted agents.
- Best VPS for local AI.
Author: Riku Lehtinen — Tampere. Last tested: 2026-05-22 on LocalAI v2.21 aio-cpu, Docker CE 27.3, Inferno 16 GB AMD EPYC plan.