LLM structured output when the provider ignores your schema

Structured output is the guarantee that a model returns parseable JSON matching a schema you supplied, rather than prose you have to salvage. Every major provider now advertises it. The failure mode nobody warns you about is the one where a provider accepts your response_format without complaint, returns HTTP 200, and hands back markdown anyway.

We hit this building recal, a local-first assistant that lets people bring their own model keys. Bring-your-own-key means we do not control which endpoint the answer comes from, so a schema that works on one provider and silently degrades on another is not an edge case for us, it is Tuesday. What follows is the ladder we shipped and the specific things that broke.

Key takeaways

• Strict schema mode is a contract with real requirements: OpenAI's Structured Outputs needs every field listed in required and additionalProperties: false, so an "optional" field has to be expressed as a null union.
• JSON mode and strict schema mode are different guarantees. JSON mode promises valid JSON. Only strict mode promises your JSON.
• Some OpenAI-compatible endpoints accept response_format and ignore it. You get 200 and prose. Feature detection by "did the request error" does not catch this.
• The reliable middle rung for BYOK is plain JSON mode with the shape carried in the prompt, not strict schema.
• Prompt framing changes the outcome on stubborn models. "Produce a brief" gets you markdown; "output only a JSON object, put the narration inside the string fields" gets you JSON.
• Whatever rung you land on, parse leniently. The last line of defence is treating unconstrained text as the answer instead of raising a parse error.

Why does response_format json_schema fail on some providers?

There are three distinct failures wearing the same coat, and separating them saved us a lot of guessing.

The request is rejected. You send response_format: {type: "json_schema", ...} and get a 400 back, often complaining about an unknown parameter. This is the honest failure. The endpoint does not implement the feature, it says so, and you can fall back on the spot.

The schema is rejected. The endpoint implements strict mode but your schema violates its rules. OpenAI's documentation is explicit that "all fields or function parameters must be specified as required" and that developers must set additionalProperties: false, because Structured Outputs only supports generating keys defined in the schema. If you wrote a schema the normal JSON Schema way, with optional fields simply omitted from required, it will be refused.

The fix is to stop thinking of fields as optional and start expressing optionality in the type. Our answer object has a required markdown string and a title that may be absent, so the strict variant declares both as required and gives title the type ["string", "null"]. That is the documented pattern for emulating an optional field, and it is why we carry two schema shapes for the same object: a plain one for endpoints that take ordinary JSON Schema, and a strict one for OpenAI.

The request is accepted and ignored. This is the one that costs you a day. The endpoint returns 200, no error anywhere, and the body contains markdown. We hit this on a coding-specialised endpoint that advertised OpenAI compatibility: it accepted the response_format field and obeyed only the prompt. Nothing in the response says "I ignored that."

The consequence is architectural, not cosmetic. Any feature detection built on "did the request error" is blind to the third case, so you cannot discover it at runtime from the response alone. You have to know it per endpoint, or make the failure survivable.

What do you do when structured output is not supported for a model?

Stop treating it as a boolean. We model it as four rungs of descending strength, resolved per endpoint when the user picks a model rather than negotiated per request.

Rung	Mechanism	Guarantee	Where the shape lives
Native strict schema	Provider's schema-constrained decoding	Output matches your schema	The API request
JSON mode	Provider forces a syntactically valid JSON object	Valid JSON, any shape	The prompt
Prompted JSON	Instruction only, no API constraint	None	The prompt
Lenient parse	Extraction from whatever came back	None	Your parser

The first three are request modes. The fourth is not a mode at all, it is the parser you run regardless of which of the first three you used, because any of them can still surprise you.

The counterintuitive part is that the middle rung is the one that earns its keep in a bring-your-own-key world. Plain JSON mode asks for much less: make the output a syntactically valid JSON object. Far more OpenAI-compatible endpoints honour that than honour strict schema, and it fails loudly rather than silently when it is unsupported. The shape then rides in the prompt, where every model can read it. That is a weaker guarantee on paper and a stronger one in practice, which is an uncomfortable thing to admit about an architecture but it is what the traffic showed us.

For endpoints where we cannot rely on either, including one major provider whose native format we deliberately do not special-case, we drop to the prompt floor and lean entirely on framing plus parsing.

Why does prompt framing change whether you get JSON?

Because on a model that ignores response_format, the prompt is the only lever left, and not all instructions pull it equally hard.

Our first version asked the model to produce a re-entry brief and described the JSON shape further down. Compliant models did the right thing. The stubborn ones read "produce a brief", decided a brief is a document, and returned beautifully formatted markdown that failed to parse.

Rewriting it fixed them without touching the compliant ones. The instruction now leads with the output contract and demotes the human-readable part to a field:

Produce a recal re-entry brief STRICTLY as a single JSON object. Output ONLY the JSON object, no markdown, no headings, no code fences, no prose before or after it. The short human-readable narration goes INSIDE the string fields; the object's structure is fixed and must not change.

Two details in there are load-bearing. First, the imperative framing puts the format before the task, so "what am I making" resolves to "a JSON object" and not "a brief". Second, the wrapper names the string json explicitly and prints the literal shape. That second one is not stylistic: OpenAI's JSON mode documentation warns that "the API will throw an error if the string 'JSON' does not appear somewhere in the context", and other providers inherited the same requirement along with the API surface. A wrapper that says reply with only a single JSON object matching {...} satisfies it as a side effect, which is a nice accident to rely on deliberately.

How do you parse output you could not constrain?

Assume the constraint failed and make that survivable. Our finalizer tries three things in order.

Strip a fenced block first. Models that mostly comply often wrap the object in triple backticks with an optional json tag, which is valid behaviour under a prompt-only instruction and invalid JSON to a strict parser.

Failing that, take the first balanced { ... } in the text. Prompted output is not grammar-constrained, so a model may add a sentence of preamble before the object. Scanning for a balanced brace span recovers the object without regex fragility.

Failing that, treat the entire response as the answer body. This is the part worth arguing for, because it looks like giving up. When a provider ignores your format request and returns a genuinely good markdown answer, raising a parse error throws away a correct response over a formatting disagreement. The user asked a question and the model answered it. Our lenient path puts that text into the markdown field and moves on, so the worst realistic outcome is a missing title rather than a failed request.

The rule we ended on: constrain as hard as the endpoint allows, then parse as if you had not constrained it at all.

What does this cost you?

Honesty requires the other column.

Carrying four rungs means carrying four code paths and two schema variants for one object, and the strict variant looks wrong to anyone who knows JSON Schema until they read the comment explaining why title is a null union. That is real complexity for a two-field object.

Lenient parsing also hides problems. A provider that quietly stopped honouring JSON mode will look fine in your metrics, because the floor catches it and the answers keep flowing. If you take this approach, log which rung actually produced each parse, or you lose the ability to notice a regression.

And none of it gives you validation. Landing on the prompt floor means the shape is a suggestion. If your downstream code needs a field to exist, it still needs to check, because a rung that cannot constrain also cannot promise.

FAQ

Is JSON mode the same as structured outputs? No. JSON mode guarantees the response parses as JSON. Structured Outputs with strict: true guarantees it matches your schema. OpenAI's own guidance is to prefer Structured Outputs when it is available, and the gap between the two is exactly the shape.

Why does my strict schema get rejected when it is valid JSON Schema? Strict mode is a subset, not the full specification. Every property must appear in required, and additionalProperties must be false. A schema that is perfectly legal elsewhere will be refused if it leaves a field out of required.

How do I make a field optional under strict mode? You do not. You keep it in required and widen its type to include null, such as ["string", "null"], then treat null as absent in your own code.

How can I tell if a provider is ignoring response_format? Not from the status code, which is the whole problem. Send a request whose answer would naturally be prose, then check whether the body parses as JSON. If it comes back as markdown with a 200, the field was accepted and discarded. Do this once per endpoint and record the result rather than probing on every call.

Does prompting for JSON hurt answer quality? In our testing the compliant models were unaffected by the stricter framing, and the stubborn ones went from unparseable to usable. The risk is different: an imperative format-first instruction can make a model terse, so put the narration requirement explicitly inside the schema description rather than hoping for it.

Do I still need this if I only use one provider? Much less. A single first-party endpoint with native strict schema is the easy case, and most of this ladder is the price of not controlling the endpoint. If you later add a second provider or let users bring their own, the cost arrives then.

The short version

Structured output is not a feature you have or lack, it is a strength you resolve per endpoint. Ask for the strongest constraint the endpoint genuinely honours, carry the shape in the prompt whenever the API will not carry it for you, and write the parser as though every constraint above it silently failed. The providers that return prose behind a 200 are not going away, and the only defence that survives them is the one that does not depend on the provider telling you the truth.

---

Sources: OpenAI's Structured Outputs guide, read 2026-08-06, for the strict-mode requirements, the null-union pattern for optional fields, the JSON mode comparison, and the requirement that the string "JSON" appear in context. Provider-specific behaviour described here is our own observation from running recal's bring-your-own-key path against multiple endpoints, not a documented claim by those vendors.

Written with AI assistance and reviewed against the primary source linked above by the recal team.