Build from source by cloning this repository - check out our build guide
Once installed:
## Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
## Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli
Built-in web UI against llama serve
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
Plain C/C++ implementation without any dependencies
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
AVX, AVX2, AVX512 and AMX support for x86 architectures
RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
Vulkan and SYCL backend support
CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain
At a glance
Category
Coding
Pricing model
Free
Best for
ai-agents, tool, local-llms
What is llama.cpp? llama.cpp is a coding tool listed on Scriptiii AI Tools. C/C++ library and tools for LLM inference on local hardware. Pricing: Free. Free to use. Best for: ai-agents, tool, local-llms.
Questions & answers
Everything you need to know about llama.cpp
What is llama.cpp?
llama.cpp is a coding tool. C/C++ library and tools for LLM inference on local hardware.
What does llama.cpp cost?
Free. Free to use.
What is llama.cpp best for?
ai-agents, tool, local-llms. It is filed under Coding.
