IFStruct gives no credit for almost valid

IFStruct has 2,000 prompts that request JSON or YAML under a supplied schema. Each response must parse, use the requested code-block style, omit forbidden commentary, choose the correct top-level shape, return the specified number of items, and satisfy required keys, field types, enums, ranges, and other constraints. Any error fails the sample.

That rule resembles software more closely than a semantic grader that awards partial credit. An invoice with the right meaning and one invented notes field may still break validation or send data down an unreviewed path.

The Apache 2.0 licensed public IFStruct dataset covers invoices, support tickets, travel itineraries, scientific experiments, terminal-session notes, and other entities. Each prompt carries a schema, so the task is to follow changing contracts rather than memorize one response template.

The reward changed what it measured

The published recipe served a BF16 version of LFM2.5-350M through llama.cpp. Before training, it passed 452 of 2,000 items, or 22.6%. JSON scored 18.0%, YAML 27.2%, wrapped objects 28.5%, and bare lists 16.6%.

Training drew about 500 rows from NVIDIA's structured-output instruction dataset. To cover requirements seen in IFStruct without training on its test rows, the authors appended a fenced-code instruction to 40% of prompts and converted a separate 20% into top-level array tasks.

A LoRA adapter updated about six million parameters, around 1.66% of the model. Three programmatic rewards graded every completion: correct parsing and raw-versus-fenced form, top-level field count, and JSON Schema validity. Schema validation received the largest weight. The run sampled eight completions for each prompt group using the open-source TRL GRPO trainer.

Afterward, the same 2,000-item evaluation and serving stack produced 594 passes, or 29.7%. JSON jumped from 18.0% to 31.9%. Bare lists moved from 16.6% to 29.7%. YAML barely changed, from 27.2% to 27.5%, while wrapped objects gained 1.2 points.

The intervention emphasized fenced and raw JSON plus bare arrays, and those categories moved most. Nothing in this test demonstrates a broad improvement in reasoning or extraction.

Seventy percent failure is not a footnote

The 7.1-point gain is a 31% relative improvement over the starting pass rate. It also leaves 1,406 failed samples.

Missing required fields remained the largest error category. Wrong item counts and type mismatches were also common. One output can trigger several violations, so raw error totals do not equal failed samples. They still show that better surface form did not solve nested constraint satisfaction.

Performance varied widely by entity. Event-ticket booking reached 57.9%, while camera reviews reached 6.0%. Interview transcript segments fell from 26.2% to 16.2%. The average hides both the targeted success and local regressions.

The report does not include multiple random seeds, confidence intervals, a held-out comparison with supervised fine-tuning, or application-specific cost. Its 1,518 millisecond average latency belongs to one stated local setup, not a general deployment claim. The authors also say this notebook does not reproduce the training pipeline for Liquid AI's separate IFStruct release model.

The experiment supports a limited conclusion: inexpensive task-specific post-training can improve exact schema behavior in a small model. It does not show that GRPO beats simpler methods, that its gain transfers to another domain, or that 29.7% supports unattended use.

What a production team should copy

Start with schemas from the real application, including nested objects, item limits, enums, escaped strings, unknown fields, and bare arrays. Freeze a held-out split before designing rewards so the test cannot gradually become part of training.

Rewards should be executable and separate. Parsing, output shape, field count, and schema validity measure different failure modes. Independent logs reveal whether the model learned cosmetic formatting or the contract itself.

Run a supervised baseline and several seeds. Record the model, dataset revision, LoRA targets, decoding settings, server, and validator revision. Cheap training is most useful when the result is also cheap to reproduce.

Production responses still need external validation. Reject unexpected fields, cap lengths, apply business rules, and retry or escalate invalid output. Grammar-constrained decoding can prevent malformed syntax, but it cannot establish that a valid value is accurate or authorized.

A minimal boundary can be boring on purpose:

result = schema.safe_parse(model_output)
if not result.success:
    return retry_with_error_summary(result.errors)

record = apply_business_rules(result.data)
if not record.approved:
    return send_for_review(record.reasons)

return commit(record.value)

The model proposes data; ordinary code decides whether that data is well formed, acceptable, and safe to commit. Rejected outputs can feed the next evaluation set after secrets are removed and the examples are kept out of the held-out test.

Our BenchMIRT review examines how a benchmark name can promise more than its score measures. The katalint article makes a related engineering case for machine-checkable constraints when prose instructions are easy to violate.

A 29.7% pass rate can justify another experiment. It cannot justify removing the validator.