What Is CUDA, and How to Run LLMs Locally: A Beginner's Guide

If you want to run an open-weight model like Llama 3, Mistral, Gemma, or Phi-3 on your own machine at a usable speed, you will meet CUDA within the first ten minutes. Most local-inference tutorials assume you already have it working. Most people who follow those tutorials do not, and the symptom is always the same: the model loads, generates at a crawl, and nobody explains why.

This guide covers what CUDA is, how to install and verify it, and three ways to actually run a model once it works: Hugging Face with 4-bit quantization, Ollama, and llama.cpp. It assumes you have an NVIDIA GPU and are comfortable at a terminal. Nothing else.

1. What CUDA is, and why LLMs need it

CUDA stands for Compute Unified Device Architecture. It is a parallel computing platform and programming interface built by NVIDIA [1]. In practice it is a translation layer: you write ordinary Python or C++ against PyTorch, and CUDA arranges for the heavy arithmetic to run on your GPU instead of your CPU.

The reason that matters comes down to how the two processors are built.

A transformer [2] generates one token at a time, and each token requires passing the whole activation through every layer: a long chain of matrix multiplications across the attention and feed-forward blocks. That workload is almost pure parallel arithmetic, which is exactly the shape a GPU is built for and exactly the shape a CPU is not.

The practical difference is large. A 7B or 8B model running on CPU alone typically generates a few tokens per second, slow enough that you watch each word appear. The same model with its layers offloaded to a mid-range consumer GPU generates fast enough to read comfortably. That gap is the entire reason this setup is worth doing.

2. Setting CUDA up

Step 1: check the hardware

• You need a supported NVIDIA GPU: GeForce GTX 10-series or newer, RTX 20/30/40 series, or a workstation or datacenter card such as Quadro, A100, or H100.
• Check your VRAM. 8 GB is a sensible floor for quantized models in the 3B to 8B range. Below that you will be offloading most layers back to the CPU, which undoes the benefit.

Step 2: install or update the driver

Get the current Game Ready or Studio driver from NVIDIA's driver downloads, then confirm it:

nvidia-smi

The output names your GPU, reports total VRAM, and reports the highest CUDA version the driver supports. Write that version down; the next two steps have to agree with it.

Step 3: install the CUDA Toolkit

Pick a version from the CUDA Toolkit archive that your framework supports. The 12.x line is the safe default. Then verify the compiler:

nvcc --version

Step 4: install PyTorch built against CUDA

This is the step people get wrong. The default pip install torch may give you a CPU-only build, and it will fail silently: everything imports, nothing is accelerated. Use the selector on the PyTorch install page and take the command it gives you for your CUDA version [3]. For CUDA 12.1 that is:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

3. Verify it before you debug anything else

Run this before you touch a model. If it prints False, no amount of tuning inference code will help you, and every later error is downstream of this one.

import torch

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")

if torch.cuda.is_available():
    print(f"Device: {torch.cuda.get_device_name(0)}")
    print(f"Capability: {torch.cuda.get_device_capability(0)}")
    vram_gb = torch.cuda.get_device_properties(0).total_memory / (1024 ** 3)
    print(f"Total VRAM: {vram_gb:.2f} GB")
else:
    print("CUDA is not available. Everything below will fall back to CPU.")

4. Route A: Hugging Face with 4-bit quantization

This is the route to take if you want to inspect or modify what happens inside the model, which is usually the case for research work. Combining transformers [4] with bitsandbytes lets you load a model in 4-bit precision and fit something useful into consumer VRAM [5].

pip install torch transformers accelerate bitsandbytes

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

def run_local_llm(prompt: str):
    model_id = "microsoft/Phi-3-mini-4k-instruct"

    # 4-bit NF4 quantization, which is what makes this fit in 8 GB
    quantization_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_quant_type="nf4",
    )

    tokenizer = AutoTokenizer.from_pretrained(model_id)

    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        device_map="auto",          # assigns layers to the GPU automatically
        quantization_config=quantization_config,
        trust_remote_code=True,
    )

    messages = [{"role": "user", "content": prompt}]
    input_ids = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to("cuda")                    # inputs must live on the same device as the weights

    with torch.no_grad():
        outputs = model.generate(
            input_ids,
            max_new_tokens=256,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )

    return tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)

if __name__ == "__main__":
    if not torch.cuda.is_available():
        print("CUDA is not available. Check the driver and the PyTorch build.")
    else:
        print(run_local_llm("Explain in three points why GPUs suit neural networks."))

Two details in there matter more than they look. device_map="auto" is what places layers across your GPU and, if it runs out of room, your CPU. And moving input_ids to "cuda" is not optional: tensors and weights on different devices is one of the most common errors in a first local-inference script.

5. Route B: Ollama

If you only want the model to answer, this is the shortest path. Ollama wraps llama.cpp [6] in a command-line tool and a local API server, detects your CUDA installation, and offloads layers without being asked.

ollama run llama3.2

It pulls the quantized weights, loads what fits into VRAM, and gives you a prompt. To reach it from Python, talk to the local endpoint:

import requests

def generate_with_ollama(prompt: str):
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": "llama3.2", "prompt": prompt, "stream": False},
    )
    response.raise_for_status()
    return response.json()["response"]

print(generate_with_ollama("Summarize the benefits of running models locally."))

6. Route C: llama.cpp, for control

llama.cpp [6] is a lightweight C/C++ inference engine using the GGUF quantized format. It gives you the most direct control over how much of the model sits on the GPU. The catch is that llama-cpp-python must be compiled with CUDA enabled, and the default install is not:

# Linux or macOS
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

# Windows PowerShell
$env:CMAKE_ARGS="-DGGML_CUDA=on"
pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir

from llama_cpp import Llama

llm = Llama(
    model_path="./models/llama-3-8b-instruct.Q4_K_M.gguf",
    n_gpu_layers=-1,   # -1 puts every layer on the GPU
    n_ctx=2048,
    verbose=True,
)

output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a careful technical assistant."},
        {"role": "user", "content": "What is the difference between FP16 and INT4 quantization?"},
    ]
)

print(output["choices"][0]["message"]["content"])

n_gpu_layers is the knob worth knowing. Set it to -1 and everything goes to the GPU. Set it to 20 and twenty layers go to the GPU while the rest run on the CPU, which is how you run a model slightly too large for your card instead of not running it at all.

7. Fitting the model in the VRAM you have

On an 8 to 12 GB card, memory is the constraint that decides everything. A rough budget per billion parameters:

Two things to remember when you read that table. Weights are not the whole cost: the KV cache grows with context length, so a long prompt can push a model that loaded fine into an out-of-memory error mid-generation. And if the total still does not fit, partial offloading through n_gpu_layers is a real answer rather than a failure. FlashAttention [7] and the paged KV caches used by serving engines such as vLLM cut the context-length half of that cost specifically.

8. The four errors you will actually hit

• CUDA out of memory. Weights plus context exceed VRAM. Quantize harder (4-bit instead of 8-bit), lower max_new_tokens or n_ctx, or drop to a smaller model. Restart the process too: a crashed run can leave memory allocated.
• CUDA error: no kernel image is available for execution. Your PyTorch build was compiled for a different compute capability than your card [3]. Reinstall PyTorch for the CUDA version nvidia-smi reported.
• DLL load failed on Windows. CUDA binaries are not on PATH. Add C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.x\bin to your environment variables.
• Everything works but it is slow. Almost always a CPU-only PyTorch build, or layers quietly offloaded to CPU because they did not fit. Re-run the check in section 3, then watch nvidia-smi during generation to confirm the GPU is doing the work.

Closing

CUDA is not glamorous and it is not the interesting part of the work, but it is the part that determines whether local inference is a tool you use or a demo you abandon. Get the driver, the toolkit, and the PyTorch build to agree on one version, verify with the six-line script before anything else, then pick the route that matches what you are doing: Hugging Face when you need to see inside the model, Ollama when you need an answer, llama.cpp when you need to squeeze a model onto a card that is slightly too small for it.

References

1. Nickolls, J., Buck, I., Garland, M., & Skadron, K. (2008). Scalable Parallel Programming with CUDA. ACM Queue, 6(2), 40–53.
2. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS). arXiv:1706.03762
3. Paszke, A., Gross, S., Massa, F., et al. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems (NeurIPS).
4. Wolf, T., Debut, L., Sanh, V., et al. (2020). Transformers: State-of-the-Art Natural Language Processing. Proceedings of EMNLP 2020: System Demonstrations, 38–45.
5. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. Advances in Neural Information Processing Systems (NeurIPS). See also Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS.
6. Gerganov, G., et al. (2023). llama.cpp: LLM inference in C/C++. github.com/ggerganov/llama.cpp
7. Dao, T., Fu, D., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS).

---

Want to share your own experience? Every member can write here: reach out and we'll help you publish your first post.