Three LLMs preserved about 84% of facts but reversed only 4.4% to 6.8% of controlled news framing
A rewrite can keep names, dates, quantities, and events intact while leaving an article's point of view almost untouched. In a new controlled test of news framing, DeepSeek-V4-Flash, Qwen-Plus, and Kimi-K2.6 kept factual preservation close to 0.84. Yet they reversed only 0.044 to 0.068 of the framing interventions inserted by the benchmark.
The two scores measure different editorial risks. Factual fidelity does not tell you who remains foregrounded, whose action is hidden in passive voice, or which fact still controls the lead. A request to make coverage "neutral," remove bias, or reconcile reports needs to test those choices directly.
The paper is a preprint and its benchmark is synthetic, so its result cannot settle a question about journalism or any model provider. It does reveal a hole in common evaluation practice. Checking whether the facts survived does not establish that a frame was removed.
The test hides a known edit, then asks the model to undo it
The benchmark starts with 60 English news articles. Each was canonicalized to 180 to 260 words and represented by six to ten atomic facts. The study applied three kinds of fact-preserving intervention:
- Lexical framing changed evaluative wording while leaving sentence and paragraph order fixed.
- Agency framing changed grammatical subjects, voice, and attribution prominence without changing who performed an action.
- Salience framing moved information within a paragraph, the lead, or the whole article.
Each operator was applied at low, medium, and high strength, producing nine variants per source and 540 framed articles. Direction was balanced across favorable and unfavorable framing. Another 60 clean articles tested whether models would invent a framing problem where the benchmark had inserted none.
GLM-5.2 generated the variants under constraints that prohibited new facts, altered entities or quantities, and new causal claims. The generator also recorded an edit map. During evaluation, the tested model saw only the framed article. It first identified the framing type and direction, then rewrote the article without access to the source, atomic fact list, true operator, or edit map.
Hiding the source turns "neutralization" into an inverse problem. After the rewrite, the evaluator checks whether the atomic facts remain and how many known framing edits were actually reversed.
Good factual retention concealed weak reversal
DeepSeek scored 0.840 on factual preservation, Qwen 0.834, and Kimi 0.837. Their intervention reversal rates were 0.044, 0.068, and 0.054, respectively. The best fact score and best reversal score came from different models.
Detection alone did not explain the gap. Macro-F1 for recognizing the three framing types plus a clean class was 0.247 for DeepSeek, 0.236 for Qwen, and 0.374 for Kimi. Kimi also produced a false positive on 75.4% of clean articles, compared with 25.0% for Qwen and 16.7% for DeepSeek. The most sensitive detector was also the least calibrated on clean articles.
Exact recognition of both framing type and direction occurred in 18.9% of aligned reconstructions. Within that subset, pooled reversal rose from 0.049 to 0.071, a 45% relative increase. The absolute result remained low: 75 of 1,052 inserted edits were reversed, leaving 92.9% in place even after correct recognition.
The location of the frame also mattered. Lexical changes were easiest to detect. Salience changes were hardest to reverse, with rates from 0.012 to 0.038 across the three models. Agency reversal ranged from 0.061 to 0.091. At high intervention strength, every model still remained below 0.09 reversal.
Native thinking modes moved reversal by +0.025 for DeepSeek, -0.021 for Qwen, and 0.000 for Kimi on matched high-strength cases. The extra inference had no consistent effect across the three models.
"Same facts" is not the same article
News framing research has long treated selection and salience as part of how an issue is defined. The original Media Frames Corpus annotated framing across policy issues, while the BASIL corpus paired lexical and informational bias across reporting on the same events. Both resources treat organization and emphasis as possible carriers of bias even when statements remain factual.
That distinction is easy to lose in an AI workflow. A system may compare extracted claims before and after a rewrite, find no contradictions, and mark the job complete. The benchmark shows why that check is too narrow. A lead can still foreground praise. Passive voice can still blur responsibility. The facts can be identical while the reader receives a different account of who mattered.
An automatic "neutralize" button hides a political decision. Neutrality is not a source text waiting to be reconstructed in ordinary newsroom use. This benchmark can score inversion because it knows exactly which controlled transformation produced each article. A live system does not have that ground truth, and reasonable editors can disagree about the baseline.
Build editing systems around recorded changes
The study suggests a safer, auditable workflow for assisted editing:
- Preserve the original article and extract a checkable fact inventory.
- Ask a detector to label lexical stance, agency choices, and information order separately.
- Give the editor a precise requested change, such as restoring an actor to the subject position or moving a buried qualification into the lead.
- Generate the rewrite, then diff it against both the original and the requested edit map.
- Recheck facts and attribution independently. Do not let the rewriting model grade its own success.
- Keep the human editor responsible for whether the requested frame change is legitimate.
A model's explanation of its rewrite is not an independent audit. Direct model self-reports correlated at only 0.04 with observed harmful behavior. The problem also resembles the same simulated electorate producing scores of 0.308 or 0.686 under different setups. Plausible output can conceal a measurement choice with large consequences.
For work done through ish.chat, comparing the original, requested edit, and final version side by side is more useful than asking for a generic fairness score. An api.ish.chat pipeline can keep detection, rewriting, fact checking, and change verification as separate calls with separate logs.
The benchmark needs its own audit
The preprint does not provide a dedicated limitations section or a linked release artifact. It reports 60 canonicalized English sources, but it does not document their selection in enough detail to judge representativeness. GLM-5.2 generated the controlled variants, and the paper says all 540 passed its validation constraints, yet the published text does not describe an independent human validation study.
The factual score is based on deterministic lexical and numeric compatibility with the source inventory. It may miss paraphrases or preserve text that a journalist would judge misleading. Only three model families were tested with one frozen task interface, tools and web access disabled. The HELM evaluation framework is a useful comparison here because it treats scenario, adaptation method, metrics, and uncertainty as separate parts of a model claim.
The finding is narrow: in this controlled setting, the systems often retained factual pieces without undoing the presentation change. Any product claiming to remove news framing should measure both.
