Ollama
Ollama is the simplest tool for running open-source LLMs locally or at the edge. One command pulls a model, runs it, and exposes an OpenAI-compatible API.
One-line startup
ollama run qwen2.5:7b
Auto-downloads a quantized model (default Q4_K_M), runs locally, no network dependency.
Features
- Model library: Qwen, Llama, Mistral, Gemma, DeepSeek, Phi work out of the box.
- Modelfile: customize system prompt / temperature / template, inheritable.
- OpenAI-compatible API:
http://localhost:11434/v1/chat/completions, drops into existing tools. - Cross-platform: macOS / Linux / Windows native; Apple Silicon auto Metal acceleration.
- GGUF quantization: defaults to llama.cpp + GGUF, CPU/GPU mixed inference.
vs other approaches
| Tool | Strength | Weakness |
|---|---|---|
| Ollama | Easy, cross-platform, rich model library | Less customizable, not for extreme perf |
| llama.cpp | Maximum perf, full control | Command-line, manual management |
| LM Studio | GUI friendly | Desktop only, no server mode |
| vLLM | Production throughput | Needs NVIDIA GPU + Linux |
Good for
- Personal dev / learning / experimentation.
- Offline / intranet / data-compliance scenarios.
- Edge deployment (NVIDIA Jetson, Mac mini, etc).