What Are Quantized LLM Categories on LLM Explorer (GGUF, GPTQ, EXL2)?

Updated

LLM Explorer indexes 14,683 quantized models and groups them into three format categories: GGUF, GPTQ, and EXL2. These are the compressed versions of open-source language models, reduced in precision so they can run on consumer hardware instead of data-center GPUs. If you're trying to run a model locally, the format you pick depends on your runtime (llama.cpp, vLLM, ExLlama) and your GPU or RAM budget — not on which model is "best" in the abstract.

Why quantization categories exist

A full-precision model stores each weight in 16 or 32 bits. Quantization drops that to 8, 4, or fewer bits, cutting file size and memory use by roughly 2–4x at some cost to output quality. LLM Explorer treats the format as a first-class filter because the format determines where the model can run, while the bit-width determines whether it fits.

The site's category list also includes hardware-fit buckets — Fit 4GB, Fit 8GB, Fit 12GB, Fit 24GB, Fit 32GB RAM/VRAM — which cut across all three formats. That's the practical link: format tells you the runtime, fit tells you the hardware.

The three quantized categories

GGUF

GGUF is the file format used by llama.cpp and most CPU-first or hybrid CPU/GPU runtimes (Ollama, LM Studio, kobold.cpp). It supports partial GPU offload, so a model can spill layers to system RAM when VRAM runs out. On LLM Explorer, GGUF is the largest quantized bucket by count, and it pairs naturally with the "Fit X GB" categories because GGUF quants are usually labeled by bit-width (Q4_K_M, Q5_K_S, Q8_0, and so on).

Choose GGUF if you're on a Mac, a CPU-only machine, or a GPU with limited VRAM and you want flexible offload.

GPTQ

GPTQ is a GPU-oriented post-training quantization format, typically 4-bit, loaded through libraries like AutoGPTQ or served by vLLM and text-generation-inference. It generally expects the whole model to sit in VRAM, so it's a better fit when you have a dedicated GPU with enough memory and want fast batched inference rather than CPU fallback.

Choose GPTQ if you're serving on a single NVIDIA GPU and don't need CPU offload.

EXL2

EXL2 is the ExLlamaV2 format, also GPU-only, and its distinguishing feature is variable bit-width — you can pick 2.5, 3, 4, 5, 6, or 8 bits per weight rather than being locked to a fixed quant. That granularity lets you squeeze a model into an exact VRAM ceiling. It tends to be favored for single-user, low-latency generation on modern NVIDIA cards.

Choose EXL2 if you want fine control over the size/quality tradeoff and you're running ExLlamaV2 or a compatible frontend.

How the categories compare

Category Runtime Hardware target Bit-width flexibility Typical use
GGUF llama.cpp, Ollama, LM Studio CPU, Mac, partial GPU offload Fixed quant presets (Q4_K_M, Q8_0, etc.) Local desktop, mixed CPU/GPU
GPTQ AutoGPTQ, vLLM, TGI NVIDIA GPU, full model in VRAM Mostly 4-bit GPU serving, batching
EXL2 ExLlamaV2 NVIDIA GPU, full model in VRAM Variable (2.5–8 bit) Single-user, low-latency

All three appear alongside the fit categories, so a typical search combines a format filter with a VRAM/RAM ceiling.

How to pick a format on LLM Explorer

  1. Start from your hardware. Decide your hard ceiling — e.g., 8 GB VRAM or 16 GB system RAM.
  2. Apply the fit category. Use "Fit 8 GB VRAM" or "Fit 4GB RAM" to narrow the list to models that will actually load.
  3. Apply the format filter. Pick GGUF for CPU/offload, GPTQ or EXL2 for full-GPU.
  4. Cross-check the model page. Confirm the quant's bit-width and context size before downloading — a model that fits at 4-bit may not fit at 8-bit with a long context.
  5. Verify the runtime. Match the format to the tool you'll actually run (Ollama reads GGUF; vLLM reads GPTQ; ExLlamaV2 reads EXL2).

Common sticking points

  • Fit categories are estimates. "Fit 8 GB" usually assumes a specific quant and context length; a larger context or higher-bit quant can push you over.
  • Format ≠ quality. GGUF, GPTQ, and EXL2 are containers and quantization methods, not quality tiers. A well-made 4-bit GGUF can beat a poorly calibrated 4-bit GPTQ.
  • No CPU path for GPTQ/EXL2. If you don't have a suitable GPU, these categories won't help you — go GGUF.
  • MoE models behave differently. Mixture-of-experts models (the site lists an MoE category) may fit in memory that a dense model of the same parameter count would not, since only some experts activate per token.

If you tell me your GPU or RAM and the runtime you use, the format choice usually collapses to one option immediately.