Skip to content
Underread
Guides

How to Fix “CUDA out of memory” When Running a Local LLM

The error names your free memory and the amount it wanted. Those 2 numbers decide which of the 5 fixes you need.

By Vemuri Abhiram6 min read
Graphics card with cooling fans mounted inside a workstation
Graphics card with cooling fans mounted inside a workstation
On this page

The model loaded fine yesterday. Today it dies part-way through a generation:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 24.00 GiB of which 1.873 GiB is free.

Read the numbers before you touch anything, because they tell you which problem you have. If the gap is small, you need to trim. If the model is twice the size of the card, no amount of trimming will help and you need a different approach entirely.

#Where your VRAM actually goes

Consumer Scales with Can you change it?
Model weights Parameters × quantisation Yes, by quantising
KV cache Context length × concurrency Yes, easily
Activations Batch size Yes
Other processes Whatever else is running Yes, kill them

#1. Find out what's already holding memory

Before blaming the model, check what else is on the card. A stale process from a crashed run holds its allocation until it exits, and a desktop session can quietly consume a surprising amount:

nvidia-smi

Look at the per-process list at the bottom rather than the total. I've lost an hour to this twice now, both times because a Python process from a notebook I'd closed was still resident and still holding 8 GB. The fix was killing a PID, not changing a single parameter.

If free memory is far lower than you expect and no process explains it, restart the machine. Fragmentation is real. Fragmentation accumulates, and a card that can't find 2 contiguous GB will fail even when the total free looks sufficient.

#2. Cut the context window first

This is the highest-value change and the one people reach for last. The KV cache scales with how much context you allow. It's allocated for the maximum you configured, not for what you're actually using. So a 128k window costs you the memory even on a 500-token prompt.

Most work does not need the maximum. Setting a window that matches your actual prompts frees memory immediately with no change in output quality for anything that fits:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

Work out what you need rather than guessing. Guessing high is what caused this. Take your longest realistic prompt, add your longest expected response, add a margin, and set that. If you're not sure what your prompts actually run to, measure before you choose — the context length guide covers counting tokens properly.

#3. Reduce concurrency

Every request in flight needs its own cache. Two parallel requests at a long context can cost more than doubling the model size, which is why a server that was stable for 1 user falls over the moment a second one arrives.

Cap it explicitly rather than letting the server decide. One request at a time on a small card is not a failure. A queue that completes beats concurrency that crashes. Throughput is usually higher anyway, because you stop losing partial work.

#4. Quantise harder, and know what it costs

If the weights alone won't fit, quantisation is the real fix. A smaller quantisation stores each weight in fewer bits, so the same model occupies less memory at some cost in quality.

The cost isn't linear, and that's the useful part. Moving from 16-bit to 8-bit is close to free in practice. Moving to 4-bit is usually still good, and it's where most people should sit on consumer hardware. Below 4-bit the degradation becomes noticeable on reasoning-heavy work, and at that point a smaller model at a higher quantisation is generally the better trade than a large model crushed down.

#5. Offload, or pick a smaller model

You can split a model between GPU and system RAM, and it works, but set your expectations. Anything running on CPU is dramatically slower than the same work on the card, and a model mostly in system RAM will feel broken rather than slow.

For a card that genuinely can't hold the model you want, a smaller model usually beats a large one offloaded. It will also finish. It's an unglamorous conclusion and it's almost always right.

#6. The order I actually work through

Check nvidia-smi. Kill anything stale. Re-run.

That clears it maybe a third of the time, and it costs 2 minutes. If the card is genuinely busy with your own process, I halve the context window next, because it is the single change with the best ratio of memory freed to quality lost. Only after that do I touch quantisation, and only then do I consider a smaller model.

The mistake I made early on was going straight to a smaller model. It felt decisive. It was also the most expensive option in quality terms. Twice I could have stayed on the bigger model by trimming a context window I'd set to the maximum for no reason at all.

Measure after each step. The error tells you how much it wanted and how much was free, so you can see whether you are closing the gap or wasting an afternoon.

#Match your symptom to the fix

What you're seeing Likely cause Go to
Fails only on long prompts KV cache from context length 2
Fails when 2 people use it Concurrency 3
Fails immediately on load Weights don't fit 4
Worked yesterday, fails today Stale process or fragmentation 1
Free memory looks sufficient Fragmentation 1

#When you've tried everything

Check the error numbers again after each change rather than guessing whether it helped. If "free" has moved and the request still fails, you're making progress and need another increment. If "free" hasn't moved at all, the setting you changed isn't the one being used — a common outcome when an environment variable is set in one shell and the server is running from another.

Watch for a second GPU too. On a multi-card machine a process pinned to GPU 0 will fail while GPU 1 sits empty, and CUDA_VISIBLE_DEVICES decides which one your process can see. If nvidia-smi shows idle capacity that your process never touches, that variable is usually why.

Frequently asked questions

Why does it fail mid-generation rather than on load?

Because the KV cache grows as the response is produced. The weights fit at load time; the cache is what runs you out later, which is why long outputs fail and short ones succeed on the same model.

Does restarting really help?

It clears fragmentation and releases anything leaked by a crashed process, so yes — temporarily. If restarting is the only thing that works, you're too close to the limit and one of the other fixes is overdue.

Is 4-bit quantisation noticeably worse?

For most everyday work, not really. The drop becomes easier to notice on reasoning-heavy and code tasks, and it gets steep below 4-bit. A smaller model at 4-bit usually beats a bigger one at 2-bit.

Can I use system RAM instead?

You can offload layers to it, but generation speed falls off a cliff. It's a reasonable stopgap and a poor permanent setup.

Written by

Vemuri Abhiram

Founder & Editor

Vemuri Abhiram is the founder and editor of Underread, where he writes about AI tools, software, and practical technology. He covers hands-on tool reviews, detailed guides, product comparisons, and emerging technologies.

More from Underread, once a week.

The guides worth reading and the tests worth trusting. One email, every Friday.

No spam. Unsubscribe in one click.