Skip to content
Underread
Guides

How to Stop an LLM Returning Invalid JSON (and What to Do When It Still Does)

Prompting for JSON is the weakest fix available, and it's the one most people reach for first. There are 3 better ones.

By Vemuri Abhiram6 min read
Screen showing structured data and code side by side
Screen showing structured data and code side by side
On this page

Your pipeline parses model output into a structure, and once in every 30 or 40 calls it falls over:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Print the raw response and you'll usually find the JSON is there and perfectly valid — wrapped in a markdown code fence, or introduced by a cheerful sentence explaining what the model has produced for you.

#The 4 fixes, from strongest to weakest

# Fix Reliability Needs provider support
1 Schema-constrained output Highest Yes
2 JSON mode High Yes
3 Tool or function calling High Yes
4 Prompt instructions plus repair Lowest No

#1. Constrain the schema, don't request it

The strongest option passes a schema to the API and has the decoder enforce it while tokens are generated. The model is not choosing to comply; the invalid tokens simply aren't available to it. That's a categorically different guarantee from asking nicely in a prompt, and it's why this belongs first rather than last.

from pydantic import BaseModel

class Invoice(BaseModel):
    number: str
    total: float
    currency: str

resp = client.responses.parse(
    model="gpt-4.1-mini",
    input=[{"role": "user", "content": text}],
    text_format=Invoice,
)
invoice = resp.output_parsed

Two things change once you do this, and the second matters more than it first appears. Parsing stops failing, obviously. But your schema also becomes the single definition of the contract, so adding a field is 1 edit rather than a prompt change plus a parser change plus a test that's checking a string.

#2. JSON mode, when a schema isn't available

If the provider supports a JSON mode but not a full schema, enable it. It guarantees syntactically valid JSON, which removes the fences and the preamble, though it won't guarantee your fields exist or have the right types — so you still validate after parsing.

The one trap worth knowing is that models can emit valid JSON that runs until the token limit and gets cut off mid-structure. Syntactically it was going to be fine; it just never finished. Give it enough headroom and check finish_reason before you trust the result, because a truncated response and a malformed one look identical to your parser.

#3. Tool calling as a side door

Even where a response schema isn't offered, tool or function calling usually is, and arguments are validated against the schema you define. Declare 1 tool whose parameters are the structure you want, force the model to call it, and read the arguments. It's slightly indirect but it gets you the same enforcement, and on older models it's often the most reliable option available.

#4. Prompt instructions, and the repair step

Prompting is the weakest fix, and it's the one most people reach for first because it's the easiest to type. You can improve the odds — ask for raw JSON with no code fences, give 1 example of the exact output, and set a low temperature — but you cannot make it a guarantee, and anything that depends on a model choosing to comply will eventually meet the call where it doesn't.

So keep a repair step regardless of which fix above you use. Strip the common wrappers first, since a fence or a preamble costs nothing to remove and accounts for most failures:

import json, re

def parse(raw):
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        pass

    fenced = re.search(r"```(?:json)?\\s*(.*?)```", raw, re.S)
    if fenced:
        return json.loads(fenced.group(1))

    start, end = raw.find("{"), raw.rfind("}")
    if start != -1 and end > start:
        return json.loads(raw[start : start + (end - start) + 1])

    raise ValueError("no JSON found")

If that still fails, retry once with the parse error fed back in. One retry is the right number: a second rarely succeeds where the first didn't, and an unbounded loop turns a bad response into a bill. There's more on keeping that kind of loop honest in the rate limit guide.

#5. Measure the failure rate before you optimise

Count parse failures as their own metric. Not errors in general — this specific one.

Without that number you're guessing, and I've seen teams rebuild a whole extraction pipeline over a failure rate that turned out to be under 1 in 500. That's a rate a single retry already absorbs. It isn't worth a rewrite.

The number also tells you when you're done. If schema-constrained output takes you from 3% failures to effectively zero, you can stop. If it doesn't move at all, the problem was never the format and you're looking at truncation or a schema the model can't satisfy.

#6. Make the schema easy to satisfy

A schema the model can't satisfy will fail no matter how you constrain it.

Keep fields optional where the source data genuinely might not contain them. Use enums for anything with a fixed set of answers, because an enum removes the guessing entirely. And avoid deeply nested structures where a flat one would do, since every level of nesting is another place for a long response to get truncated before it closes.

Ask for less. You'll get it more reliably.

#Match your symptom to the fix

What you're seeing Likely cause Go to
Output wrapped in ```json fences Prompt-only approach 1 or 4
"Here is the JSON you asked for..." Same 1 or 4
Valid JSON, wrong field names No schema enforcement 1
JSON cut off mid-object Hit the token limit 2
Fails on long inputs only Truncation, not formatting 2

#When it's still failing

Log the raw response on every parse failure, before any repair runs. Debugging this without the original string is guesswork, and the failures are rare enough that you won't reproduce them on demand.

Check finish_reason at the same time. If it says the response hit the length limit, no amount of parser work will help — the model was cut off and the fix is a bigger max_tokens or a smaller requested structure. If the model consistently refuses to produce the shape you want, the schema may be asking for something it can't satisfy, such as a field that doesn't exist in the input. A model asked to extract a VAT number from a document with no VAT number has no good option, and a clearly optional field with a null default is a better answer than a stricter prompt.

Frequently asked questions

Why does the model add a code fence when I asked it not to?

Because fenced JSON is overwhelmingly what it saw in training when JSON appeared in a chat response. A prompt instruction is competing against that prior, which is why constraining the output at the API wins.

Does schema-constrained output cost more?

Not meaningfully in tokens. The saving is in retries and failed runs, which is where the real cost of unreliable parsing sits.

Can I use this with streaming?

Yes, though you only get a complete, valid object at the end. Partial JSON mid-stream is expected and parsing it as it arrives needs an incremental parser.

Should I validate even with a schema?

Yes. Schema enforcement guarantees the shape, not that the values make sense. A date in the future or a negative total will pass the schema and still be wrong.

Written by

Vemuri Abhiram

Founder & Editor

Vemuri Abhiram is the founder and editor of Underread, where he writes about AI tools, software, and practical technology. He covers hands-on tool reviews, detailed guides, product comparisons, and emerging technologies.

More from Underread, once a week.

The guides worth reading and the tests worth trusting. One email, every Friday.

No spam. Unsubscribe in one click.