What the parser may forgive

A structured-output pipeline has at least three jobs: describe the required shape, turn model text into a program object, and decide whether the result is safe to use. Those jobs are easy to tangle together.

In BAML, the output type enters the prompt through ctx.output_format. The model may still add surrounding text or make minor formatting mistakes. SAP then attempts to map the response to the declared type. BoundaryML's error-handling guide says the parser tolerates minor errors and thought tokens, while ambiguous failures can still raise BamlValidationError.

Recovering an obvious trailing comma can replace brittle cleanup code. Supplying a missing invoice total would be something else entirely. A parser can repair representation; it should not manufacture a business fact.

class Receipt {
  merchant string
  total float
  currency string
  items Item[]

  @@assert(total_is_nonnegative, {{ this.total >= 0 }})
}

class Item {
  name string
  quantity float @assert(quantity_is_positive, {{ this > 0 }})
  price float
}

function ExtractReceipt(image: image) -> Receipt {
  client ReceiptModel
  prompt #"
    Extract the receipt exactly as shown.
    {{ ctx.output_format }}
    {{ image }}
  "#
}

The schema gives SAP a target. The assertions reject a negative total or nonpositive quantity. Neither rule proves that the merchant, price, or currency matches the image.

Correct types can contain bad answers

BoundaryML's article "Structured Outputs Create False Confidence" argues that constrained generation can hide uncertainty by forcing an answer into an inappropriate object. Its receipt examples illustrate the idea, but the article is a vendor-authored experiment, not a neutral benchmark. A team should reproduce the behavior with its own models and documents before generalizing from it.

The underlying problem is easy to demonstrate without that experiment. 0.46 and 1.0 are both valid floats. A well-formed URL can cite the wrong source. A support ticket can match an allowed category while missing the customer's actual request.

BAML offers two ways to express validation rules:

• @assert and @@assert are strict. A failed top-level assertion raises BamlValidationError; inside containers, a failure can remove the invalid member.
• @check and @@check keep the returned data and expose whether the check passed. That is useful when a human or later stage should review the value.

Assertions suit invariants the application must enforce, such as a nonnegative amount, an identifier pattern, or uniqueness across a set. Checks suit warnings and review signals. Questions that depend on external evidence usually need deterministic comparison or a separate verification step.

Retry according to the failure

A blind retry hides the reason a call failed and may repeat the same expensive mistake. Record the failure class first.

1. If the provider request failed, use the appropriate network or rate-limit handling.
2. If SAP cannot map an ambiguous response to the type, save the raw response and parse error. Retry or call a dedicated fixup function only when the operation is safe and bounded.
3. If an assertion failed, retain its name. Prompt changes may help, but this is a known invariant violation, not generic malformed JSON.
4. If a soft check failed, return the value with a review flag instead of silently promoting it to trusted data.
5. If downstream evidence contradicts a parsed value, classify it as a semantic extraction error and add the case to the test set.

This split also improves agent traces. Provider errors, parse errors, validation failures, and wrong-but-valid answers should not share one "LLM failed" counter. They require different fixes and have different production risks.

Keep schemas and failures in version control

BAML files live with the application, so function signatures, prompts, clients, assertions, and tests can pass through normal code review. Test blocks run in the editor or through baml-cli test. They support hard assertions, soft checks, latency conditions, and filters for CI runs.

Happy-path documents are not enough. Add malformed responses, missing fields, contradictory values, out-of-domain images, prompt-injection text embedded in documents, and cases where the correct result is a refusal or review state. Test the parser's tolerance as well as the model. An overly generous coercion can be as risky as a parser that breaks on harmless punctuation.

Provider portability makes those tests more useful. BAML's openai-generic client accepts an overridden base_url, API key, model name, and provider options. The same typed function can therefore run against an OpenAI-compatible endpoint, including ISH API, while the schema and acceptance rules stay fixed.

client<llm> ReceiptModel {
  provider "openai-generic"
  options {
    base_url "https://api.ish.chat/v1"
    api_key env.ISH_API_KEY
    model "YOUR_MODEL_ID"
  }
}

Holding the prompt, inputs, schema, and tests steady matters when comparing models. Otherwise a provider switch and a test rewrite become one experiment, and the result explains neither.

Where BAML stops

BAML can turn flexible text into typed objects, surface named validation failures, and keep tests beside prompts. It cannot inspect a receipt and establish that the total matches the pixels. It cannot decide whether a citation supports a claim or whether an extracted action is authorized.

Use SAP as the representation layer. Put hard business invariants in assertions, review signals in checks, and evidence comparisons in deterministic code where possible. Save difficult failures as versioned tests. When a clean object reaches the application, its shape may be trustworthy. Its contents still have to earn that trust.

Primary sources

• BAML open-source repository
• Schema-Aligned Parsing comparison
• BAML checks and assertions
• BAML testing functions
• BAML error handling and fixup guidance
• BAML openai-generic client