Skip to main content
ZICQ

Wiki Infrastructure

Ollama

Infrastructure
Aliases: Ollama ·2026-09-14

Ollama

Ollama is the simplest tool for running open-source LLMs locally or at the edge. One command pulls a model, runs it, and exposes an OpenAI-compatible API.

One-line startup

ollama run qwen2.5:7b

Auto-downloads a quantized model (default Q4_K_M), runs locally, no network dependency.

Features

  • Model library: Qwen, Llama, Mistral, Gemma, DeepSeek, Phi work out of the box.
  • Modelfile: customize system prompt / temperature / template, inheritable.
  • OpenAI-compatible API: http://localhost:11434/v1/chat/completions, drops into existing tools.
  • Cross-platform: macOS / Linux / Windows native; Apple Silicon auto Metal acceleration.
  • GGUF quantization: defaults to llama.cpp + GGUF, CPU/GPU mixed inference.

vs other approaches

Tool Strength Weakness
Ollama Easy, cross-platform, rich model library Less customizable, not for extreme perf
llama.cpp Maximum perf, full control Command-line, manual management
LM Studio GUI friendly Desktop only, no server mode
vLLM Production throughput Needs NVIDIA GPU + Linux

Good for

  • Personal dev / learning / experimentation.
  • Offline / intranet / data-compliance scenarios.
  • Edge deployment (NVIDIA Jetson, Mac mini, etc).