GGUF (GPT-Generated Unified Format) is a file format for storing quantized LLMs. It bundles model weights, tokenizer, and metadata into a single file. Quantization levels (Q4_0, Q4_K_M, Q5_K_M, Q8_0, etc.) trade quality for file size and speed. Q4_K_M is recommended for most use cases.

llama.cpp Cheatsheet

Q: How do I install llama.cpp?

Option 1: Download pre-built binaries from GitHub Releases. Option 2: Build from source — `git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build --config Release`. Option 3: Install via pip: `pip install llama-cpp-python`.

Q: How do I enable GPU acceleration?

Use `-ngl N` where N is the number of layers to offload to GPU. `-ngl 99` or `-ngl 999` offloads all layers. For NVIDIA GPUs, llama.cpp uses CUDA. For Apple Silicon, it uses Metal automatically. For AMD, use Vulkan or HIP.

Q: How do I run llama.cpp as an API server?

Run `llama-server -m model.gguf --port 8080`. Then use curl like OpenAI API: `curl http://localhost:8080/v1/chat/completions -d '{"model":"model","messages":[{"role":"user","content":"Hi"}]}'`. The server supports OpenAI-compatible endpoints.

Q: How do I quantize a model?

Run `llama-quantize input.gguf output.gguf Q4_K_M`. The input must already be in GGUF format (unquantized or higher precision). Common quantization types: Q4_0 (fastest), Q4_K_M (recommended balance), Q5_K_M (higher quality), Q8_0 (almost lossless, larger).

Q: How do I get embeddings for RAG with llama.cpp?

Start the server with `--embeddings`: `llama-server -m model.gguf --embeddings --port 8080`. Then POST to `/v1/embeddings` with input text. Alternatively, use `llama-embedding -m model.gguf -p 'Your text'` for CLI embedding generation.

llama.cpp is a high-performance C/C++ local LLM inference engine supporting GGUF format models. It offers `llama-cli` (command-line inference) and `llama-server` (HTTP server with OpenAI-compatible API), supporting CPU/GPU/Metal inference and model quantization.

Updated: 2026-07-20·25 commands

Quick Start

``bash # Download a GGUF model huggingface-cli download bartowski/Llama-3.2-3B-Instruct-GGUF --include 'Q4_K_M/*' --local-dir ./models

# Chat inference llama-cli -m ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf -p "Hello, how are you?"

# HTTP Server (OpenAI-compatible API) llama-server -m ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf --port 8080

# Query the server (from another terminal) curl -s http://localhost:8080/v1/chat/completions \ -d '{"model":"model","messages":[{"role":"user","content":"Hello"}]}'`

`CLI Inference (llama-cli)`

Use llama-cli for direct terminal inference. Key flags: -m (model file), -p (prompt), -n (max tokens), --temp (temperature), -ngl (GPU layers), -t (threads). Add --no-display-prompt for clean output.

`HTTP Server (llama-server)`

llama-server exposes an OpenAI-compatible API. Supports chat completions (/v1/chat/completions), text completions (/v1/completions), embeddings (/v1/embeddings), tokenization, and model management endpoints.

`Model Quantization`

Use llama-quantize to convert models between GGUF quantization levels. Typical command: llama-quantize input.gguf output.gguf Q4_K_M. Common types: Q4_0 (fastest), Q4_K_M (best balance), Q5_K_M (higher quality), Q8_0 (near lossless).

`Embeddings`

llama.cpp supports text embeddings for RAG pipelines. Enable the embeddings endpoint in server mode with --embeddings, or use the standalone llama-embedding CLI tool.

`GPU Acceleration`

Control GPU offload with -ngl N. -ngl 99` offloads all layers. Apple Silicon uses Metal automatically. NVIDIA GPUs use CUDA. AMD GPUs use Vulkan or HIP.

Startup & Modes(11)

Command	Level	Example
`llama-cli --help` Show llama.cpp CLI help	Basic	`llama-cli --help`
`llama-cli -m` Specify model file path (GGUF format)	Basic	`llama-cli -m models/llama-3.2-3b-q4_k_m.gguf -p 'Hello'`
`llama-cli -p` Specify input prompt	Basic	`llama-cli -m model.gguf -p 'What is the capital of France?'`
`llama-cli -f` Read prompt from file	Basic	`llama-cli -m model.gguf -f prompt.txt`
`llama-cli -n` Set max generated tokens	Basic	`llama-cli -m model.gguf -p 'Hello' -n 512`
`llama-cli -t` Set thread count (CPU inference)	Intermediate	`llama-cli -m model.gguf -p 'Hello' -t 8`
`llama-cli --temp` Set temperature (0-2)	Intermediate	`llama-cli -m model.gguf -p 'Poem' --temp 0.7`
`llama-cli --top-p` Set top-p sampling	Intermediate	`llama-cli -m model.gguf -p 'Story' --top-p 0.9`
`llama-cli --ctx-size` Set context window size	Intermediate	`llama-cli -m model.gguf --ctx-size 8192 -p 'Long context'`
`llama-cli -ngl` Specify GPU offload layers (GPU acceleration)	Intermediate	`llama-cli -m model.gguf -p 'Hello' -ngl 35`
`llama-cli --no-display-prompt` Output only generated text (no prompt echo)	Basic	`llama-cli -m model.gguf -p 'Hello' --no-display-prompt`

ai-server(8)

Command	Level	Example
`llama-server --help` Show HTTP server help	Basic	`llama-server --help`
`llama-server -m` Start model in HTTP server mode	Intermediate	`llama-server -m models/model.gguf --port 8080`
`llama-server --port` Specify server listening port	Basic	`llama-server -m model.gguf --port 8080`
`llama-server --host` Specify bind host address	Intermediate	`llama-server -m model.gguf --host 0.0.0.0`
`llama-server -ngl` GPU offload in server mode	Intermediate	`llama-server -m model.gguf -ngl 99 --port 8080`
`llama-server --embeddings` Enable embeddings endpoint	Expert	`llama-server -m model.gguf --embeddings --port 8080`
`llama-server --chat-template` Specify chat template file	Expert	`llama-server -m model.gguf --chat-template chatml.json`
`curl localhost:8080/v1/chat/completions` Send OpenAI-compatible request to llama-server	Intermediate	`curl -s http://localhost:8080/v1/chat/completions -d '{"model":"model","messages":[{"role":"user","content":"Hi"}]}'`

ai-model(3)

Command	Level	Example
`llama-cli --hf-token` Use HuggingFace token to download models	Intermediate	`llama-cli --hf-token hf_xxx -m hf:bartowski/Llama-3.2-3B-Instruct-GGUF`
`llama-cli --hf` Load model directly from HuggingFace	Intermediate	`llama-cli --hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF -p 'Hello'`
`llama-quantize` Quantize model to specified precision	Expert	`llama-quantize model.gguf model-q4_k_m.gguf Q4_K_M`

ai-eval(1)

Command	Level	Example
`llama-perplexity -m` Evaluate model perplexity	Expert	`llama-perplexity -m model.gguf -f test.txt`

ai-embedding(2)

Command	Level	Example
`llama-embedding` Generate text embeddings	Intermediate	`llama-embedding -m model.gguf -p 'Hello world'`
`curl -X POST http://localhost:8080/v1/embeddings` Get embeddings via API	Intermediate	`curl -s http://localhost:8080/v1/embeddings -d '{"model":"model","input":"Hello"}'`

FAQ

How do I install llama.cpp?

What is GGUF format?

How do I enable GPU acceleration?

How do I run llama.cpp as an API server?

How do I quantize a model?

How do I get embeddings for RAG with llama.cpp?

This cheatsheet is compiled from official tool documentation. Last updated: 2026-07-20.