What the researchers changed

The study began with 4,779 respondents from the 2024 American National Election Studies who reported either voting for one of the two major candidates or not voting. Each respondent became a simulated persona.

The researchers varied two phrasings of the vote question, four persona designs, six open-weight instruction-tuned models, three response-handling methods, and two weighting approaches. The models included two sizes each from the Llama, OLMo, and Qwen families. Free-text answers were parsed either by regular expressions or a separate model, while another branch requested structured JSON.

That produced 288 total configurations. The team ran each of the 144 configurations involving stochastic model generation with five seeds, yielding 3,440,880 simulated responses. Its published reproduction code and prompts expose the design behind the charts.

Two metrics compared simulations with ANES responses. Weighted F1 measured individual-level classification across Harris, Trump, and non-voter. Weighted total variation distance, or TVD, measured the gap between aggregate distributions. Higher F1 is better; lower TVD is better.

Reliability did not rescue validity

Repeated runs of a given configuration were generally stable across the five seeds. That sounds reassuring until the configurations are compared with each other. The best setup reached F1 0.686 and TVD 0.097. The worst reached F1 0.308 and TVD 0.489.

The result separates two ideas that AI evaluations often blur. Reliability asks whether the same setup produces similar results when repeated. Sensitivity asks whether reasonable changes to the setup alter the conclusion. The simulated survey was reliable within a configuration, yet sensitive enough across configurations to change what a user might believe about the electorate.

The American Association for Public Opinion Research makes the same distinction in its 2026 report on responsible AI integration in survey research. It recommends checking validity, performance, sensitivity, and reliability, with evaluation criteria set before production use. Repeatability alone does not show that the model measured the intended construct or represented the intended population.

We saw a related problem in OmniaBench's overall agent scores, where an aggregate concealed large subgroup differences.

There was no single best model

The model ranked first depended on the metric. Qwen-30B had the best mean F1 among the model choices, while OLMo-32B had the best mean TVD. The two metrics were strongly correlated across configurations, but they did not select the same winner.

The subgroup results complicate the ranking further. Qwen-30B was strongest overall by F1, yet the Llama models and Qwen-8B performed better for conservative and extremely conservative respondents. The paper reports that Qwen-30B substantially underestimated non-voter rates among moderate and conservative groups.

Calling one model "best at public opinion" would discard variation that a public-opinion system needs to represent. Qwen3-VL-30B-A3B-Instruct and OLMo 3.1 32B Instruct have different training histories and response behavior. Their ordering changed with the measurement target inside this one task.

The result also echoes Company Bench's replay finding: when an evaluation label or rank changes after a defensible methodological choice, the methodology belongs in the headline.

Better personas can smuggle in the answer

Persona composition had a large association with performance. Demographic-only direct personas averaged F1 0.434. Direct personas that added attitudes averaged 0.624. Those attitudes included political ideology and party identification, variables closely tied to vote choice.

That gain does not establish that a model can predict an unseen electorate. In a real polling gap, the most predictive attitude may be missing, outdated, or exactly what the researcher is trying to estimate. A system validated on 2024 variables and 2024 vote choice may fail in a future election after candidates, events, and coalitions have changed.

The paper calls this the context drift fallacy. It also identifies a ground-truth fallacy: ANES is a carefully designed human survey, but it still contains survey error and is not identical to verified behavior. Comparing a simulation with ANES measures agreement with that survey, not access to a flawless political reality.

What a credible synthetic survey should disclose

The paper's TS2E framework adapts total survey error to model-generated respondents. A review needs the following details:

• State whether synthetic responses are used for pretesting, augmentation, imputation, or full replacement of human respondents.
• Define the target population and explain how real people became simulation units and personas.
• Record the exact model, version, training cutoff where known, inference settings, prompt, output format, and response parser.
• Run multiple seeds, but also vary plausible prompts, persona formats, models, and post-processing choices.
• Report individual and aggregate metrics, then break them out for relevant subgroups.
• Name the reference dataset's own limitations and test on a later or otherwise unseen setting.

Teams using ish.chat to explore survey questions or api.ish.chat to run structured experiments should preserve prompts, model identifiers, raw outputs, parsing code, and evaluation results. Without that record, sensitivity cannot be measured after the fact.

Synthetic respondents may help pretest questionnaires, explore assumptions, or stress-test an analysis before human fieldwork. They should not become cheaper citizens whose answers stand in for consent, participation, or current opinion. In this experiment, changing the machinery around the same personas more than doubled the worst F1 score. That change came from the simulation design, not the public.