How to Fix “CUDA out of memory” When Running a Local LLM
The error names your free memory and the amount it wanted. Those 2 numbers decide which of the 5 fixes you need.
On this page
- Where your VRAM actually goes
- 1. Find out what's already holding memory
- 2. Cut the context window first
- 3. Reduce concurrency
- 4. Quantise harder, and know what it costs
- 5. Offload, or pick a smaller model
- 6. The order I actually work through
- Match your symptom to the fix
- When you've tried everything
The model loaded fine yesterday. Today it dies part-way through a generation:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 24.00 GiB of which 1.873 GiB is free.
Read the numbers before you touch anything, because they tell you which problem you have. If the gap is small, you need to trim. If the model is twice the size of the card, no amount of trimming will help and you need a different approach entirely.
#Where your VRAM actually goes
| Consumer | Scales with | Can you change it? |
|---|---|---|
| Model weights | Parameters × quantisation | Yes, by quantising |
| KV cache | Context length × concurrency | Yes, easily |
| Activations | Batch size | Yes |
| Other processes | Whatever else is running | Yes, kill them |
#1. Find out what's already holding memory
Before blaming the model, check what else is on the card. A stale process from a crashed run holds its allocation until it exits, and a desktop session can quietly consume a surprising amount:
nvidia-smi
Look at the per-process list at the bottom rather than the total. I've lost an hour to this twice now, both times because a Python process from a notebook I'd closed was still resident and still holding 8 GB. The fix was killing a PID, not changing a single parameter.
If free memory is far lower than you expect and no process explains it, restart the machine. Fragmentation is real. Fragmentation accumulates, and a card that can't find 2 contiguous GB will fail even when the total free looks sufficient.
#2. Cut the context window first
This is the highest-value change and the one people reach for last. The KV cache scales with how much context you allow. It's allocated for the maximum you configured, not for what you're actually using. So a 128k window costs you the memory even on a 500-token prompt.
Most work does not need the maximum. Setting a window that matches your actual prompts frees memory immediately with no change in output quality for anything that fits:
OLLAMA_CONTEXT_LENGTH=8192 ollama serve
Work out what you need rather than guessing. Guessing high is what caused this. Take your longest realistic prompt, add your longest expected response, add a margin, and set that. If you're not sure what your prompts actually run to, measure before you choose — the context length guide covers counting tokens properly.
#3. Reduce concurrency
Every request in flight needs its own cache. Two parallel requests at a long context can cost more than doubling the model size, which is why a server that was stable for 1 user falls over the moment a second one arrives.
Cap it explicitly rather than letting the server decide. One request at a time on a small card is not a failure. A queue that completes beats concurrency that crashes. Throughput is usually higher anyway, because you stop losing partial work.
#4. Quantise harder, and know what it costs
If the weights alone won't fit, quantisation is the real fix. A smaller quantisation stores each weight in fewer bits, so the same model occupies less memory at some cost in quality.
The cost isn't linear, and that's the useful part. Moving from 16-bit to 8-bit is close to free in practice. Moving to 4-bit is usually still good, and it's where most people should sit on consumer hardware. Below 4-bit the degradation becomes noticeable on reasoning-heavy work, and at that point a smaller model at a higher quantisation is generally the better trade than a large model crushed down.
#5. Offload, or pick a smaller model
You can split a model between GPU and system RAM, and it works, but set your expectations. Anything running on CPU is dramatically slower than the same work on the card, and a model mostly in system RAM will feel broken rather than slow.
For a card that genuinely can't hold the model you want, a smaller model usually beats a large one offloaded. It will also finish. It's an unglamorous conclusion and it's almost always right.
#6. The order I actually work through
Check nvidia-smi. Kill anything stale. Re-run.
That clears it maybe a third of the time, and it costs 2 minutes. If the card is genuinely busy with your own process, I halve the context window next, because it is the single change with the best ratio of memory freed to quality lost. Only after that do I touch quantisation, and only then do I consider a smaller model.
The mistake I made early on was going straight to a smaller model. It felt decisive. It was also the most expensive option in quality terms. Twice I could have stayed on the bigger model by trimming a context window I'd set to the maximum for no reason at all.
Measure after each step. The error tells you how much it wanted and how much was free, so you can see whether you are closing the gap or wasting an afternoon.
#Match your symptom to the fix
| What you're seeing | Likely cause | Go to |
|---|---|---|
| Fails only on long prompts | KV cache from context length | 2 |
| Fails when 2 people use it | Concurrency | 3 |
| Fails immediately on load | Weights don't fit | 4 |
| Worked yesterday, fails today | Stale process or fragmentation | 1 |
| Free memory looks sufficient | Fragmentation | 1 |
#When you've tried everything
Check the error numbers again after each change rather than guessing whether it helped. If "free" has moved and the request still fails, you're making progress and need another increment. If "free" hasn't moved at all, the setting you changed isn't the one being used — a common outcome when an environment variable is set in one shell and the server is running from another.
Watch for a second GPU too. On a multi-card machine a process pinned to GPU 0 will fail while GPU 1 sits empty, and CUDA_VISIBLE_DEVICES decides which one your process can see. If nvidia-smi shows idle capacity that your process never touches, that variable is usually why.
Frequently asked questions
Why does it fail mid-generation rather than on load?
Because the KV cache grows as the response is produced. The weights fit at load time; the cache is what runs you out later, which is why long outputs fail and short ones succeed on the same model.
Does restarting really help?
It clears fragmentation and releases anything leaked by a crashed process, so yes — temporarily. If restarting is the only thing that works, you're too close to the limit and one of the other fixes is overdue.
Is 4-bit quantisation noticeably worse?
For most everyday work, not really. The drop becomes easier to notice on reasoning-heavy and code tasks, and it gets steep below 4-bit. A smaller model at 4-bit usually beats a bigger one at 2-bit.
Can I use system RAM instead?
You can offload layers to it, but generation speed falls off a cliff. It's a reasonable stopgap and a poor permanent setup.
Written by
Vemuri Abhiram
Founder & Editor
Vemuri Abhiram is the founder and editor of Underread, where he writes about AI tools, software, and practical technology. He covers hands-on tool reviews, detailed guides, product comparisons, and emerging technologies.
You might also like
How to Fix “The requested webhook is not registered” in n8n (404)
Why an n8n webhook works while you watch it and 404s afterwards, and the 5 causes ranked by how often they're the answer.
Vemuri Abhiram5 min read
How to Fix Cloudflare Error 524 on Long LLM Requests (Without an Enterprise Plan)
Why a slow model call dies at exactly 100 seconds behind Cloudflare, and why streaming fixes it when raising timeouts can't.
Vemuri Abhiram6 min read
How to Fix “operator does not exist: vector <=> vector” in Supabase and pgvector
Why a pgvector similarity search fails with error 42883 inside a Supabase function, and the fix with the fewest side effects.
Vemuri Abhiram5 min read
More from Underread, once a week.
The guides worth reading and the tests worth trusting. One email, every Friday.
No spam. Unsubscribe in one click.

