How to Run Large Language Models Locally with 8-Bit and 4-Bit Quantization

large_language_model

How to Run Large Language Models Locally with 8-Bit and 4-Bit Quantization

Learn how to use model quantization to run massive LLMs on consumer hardware with minimal memory and performance trade-offs.

Jan 8, 2026

How to Run Large Language Models Locally with 8-Bit and 4-Bit Quantization

I’ve been tinkering with large language models for months now, and there’s a constant hurdle I face: these models are enormous. Trying to run something like a 70-billion-parameter model feels like fitting an elephant into a garage. It just doesn’t work with standard hardware. This struggle is what pushed me to explore model quantization. If you’ve ever wanted to run a powerful AI model on your own computer without needing a data center’s worth of graphics cards, you’re in the right place. Let’s walk through how to make these giants fit into smaller spaces.

Why should you care about this? Well, a model stored in full precision might need 140 gigabytes of memory. That’s impractical for most of us. Quantization changes the data format of the model’s weights, using fewer bits to represent each number. Think of it like compressing a high-resolution photo into a smaller file. You keep most of the details, but the size drops dramatically. This means you can run models on consumer GPUs, speed up response times, and cut costs. Isn’t it fascinating how a simple change in number format can open so many doors?

Getting started is straightforward. First, set up a clean Python environment. I use Conda, but any virtual environment works. Here are the commands I run.

conda create -n llm-quant python=3.10
conda activate llm-quant
pip install torch transformers accelerate bitsandbytes auto-gptq

Once that’s done, let’s verify everything is in order. I always run a quick check to ensure my GPU is recognized.

import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")

Now, what exactly is happening during quantization? At its core, it’s about mapping a range of values to a smaller set. Full precision uses 32 or 16 bits per number. Quantization might use 8 or even 4 bits. This introduces a small error, but for many tasks, it’s barely noticeable. Have you ever wondered how much accuracy we might lose? Let’s look at a simple example.

import numpy as np
# Simulate converting numbers to 8-bit
original_values = np.array([1.2, -0.5, 3.7], dtype=np.float32)
scale = 127 / np.abs(original_values).max()
quantized = np.round(original_values * scale).astype(np.int8)
restored = quantized.astype(np.float32) / scale
print(f"Original: {original_values}")
print(f"Restored: {restored}")

The key is finding a balance. Too much compression, and the model’s performance drops. Too little, and you don’t save much memory. That’s where tools like bitsandbytes come in. They make 8-bit quantization almost effortless.

Let me show you how I load a model using 8-bit quantization. I often use Mistral 7B as a test case because it’s a good balance of size and capability.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=quant_config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

After loading, I check the memory usage. It’s always surprising to see the difference.

memory_gb = model.get_memory_footprint() / (1024**3)
print(f"Model memory: {memory_gb:.2f} GB")

With this setup, a model that needed 14 GB might now use only 7 GB. That’s half the memory! But what if we need to go further? What about running even larger models or on devices with very limited RAM? This is where 4-bit quantization, specifically GPTQ, enters the picture.

GPTQ is a more advanced technique. It carefully selects which weights to compress more, aiming to preserve accuracy. It’s like smart compression for models. Setting it up requires an extra step, but it’s worth it.

First, you need to install the auto-gptq library. Then, you can load a pre-quantized model or quantize one yourself. Here’s how I do it with a model that already has GPTQ weights available.

from transformers import AutoTokenizer, pipeline
from auto_gptq import AutoGPTQForCausalLM

model_name = "TheBloke/Llama-2-7B-GPTQ"
model = AutoGPTQForCausalLM.from_quantized(model_name, device="cuda:0")
tokenizer = AutoTokenizer.from_pretrained(model_name)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
result = pipe("Explain gravity in one sentence:")[0]['generated_text']
print(result)

Sometimes, you might want to quantize a model yourself. This takes more time but gives you control. The process involves calibrating the model on a small dataset. Why is calibration important? It helps the quantization understand which parts of the model are most sensitive to changes.

from datasets import load_dataset
from auto_gptq import BaseQuantizeConfig

# Example calibration setup
dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
calib_data = dataset["text"][:1000]  # Use first 1000 texts

quantize_config = BaseQuantizeConfig(bits=4, group_size=128)
# In practice, you'd use AutoGPTQForCausalLM.from_pretrained with quantize_config
# and pass the calibration data. This is a simplified view.

Choosing between 8-bit and 4-bit depends on your needs. Eight-bit is faster to apply and often has minimal accuracy loss. Four-bit saves more memory but might require careful tuning. I typically start with 8-bit for simplicity. If I hit memory limits, I switch to GPTQ. How do you decide which to use? Consider your hardware and how critical top accuracy is for your task.

Let’s talk about putting this into practice. I built a simple API to serve quantized models. It uses FastAPI and can switch between different quantization levels based on available resources.

from fastapi import FastAPI
from pydantic import BaseModel
import torch

app = FastAPI()

class Request(BaseModel):
    prompt: str
    quant_type: str = "8bit"  # or "gptq"

# In a real app, you'd load models here and handle requests
@app.post("/generate")
def generate_text(request: Request):
    # This is a placeholder for model logic
    return {"response": f"Processing prompt with {request.quant_type} quantization"}

For optimization, monitoring GPU memory is crucial. I use this snippet to keep an eye on usage during inference.

import psutil
import torch

def check_memory():
    gpu_memory = torch.cuda.memory_allocated() / (1024**3)
    print(f"GPU memory used: {gpu_memory:.2f} GB")

Throughout my experiments, I’ve found that quantization doesn’t just make models accessible; it changes how we think about deployment. We can now run powerful AI on laptops or small servers. But it’s not magic. There’s always a trade-off. The art is in minimizing the impact on model quality.

I often test quantized models on standard benchmarks. For example, I might compare responses from a full-precision model and a quantized one on the same prompts. The differences are usually small for general questions. Have you tried comparing outputs yourself? It’s a great way to build intuition.

In conclusion, quantization is a powerful tool for anyone working with large language models. It reduces costs, increases accessibility, and speeds up inference. I’ve shared the methods that work for me, from basic 8-bit to advanced GPTQ. Remember, the best approach depends on your specific situation. Start simple, experiment, and see what fits your needs.

If you found this guide helpful, please like, share, and comment with your own experiences. I’d love to hear how quantization is helping your projects. Let’s make big AI models work for everyone, not just those with massive hardware.

As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!

Our Creations

Be sure to check out our creations:

We are on Medium

Share: Facebook Twitter Reddit LinkedIn WhatsApp Telegram Pinterest Email Instagram

large_language_model