Frontier LLMs Fail a Basic Aggregate-Probability Test, arXiv Paper Finds

Researchers say estimates built from finer-grained persona prompts can align more closely with human reference data than a model’s direct answer for the full population.

A paper posted to arXiv on July 16 reports that state-of-the-art language models often fail a statistical self-consistency test that should hold if their prompt-based estimates behave like conditional probabilities.

The researchers evaluate whether a model’s estimates for subpopulations can be combined into a valid estimate for an overall population. Under the law of total probability, prior-weighted estimates for mutually exclusive subgroups should reproduce the population-level estimate. The paper finds widespread violations of that expectation across multiple problem domains and frontier models.

The result matters for teams using LLMs to estimate population behavior, preferences, risks, or survey-like distributions from natural-language prompts. A single prompt asking for an aggregate estimate may be less dependable than a workflow that asks about constituent groups and reconstructs the aggregate from those answers.

The researchers use binary trees to repeatedly divide a population into increasingly fine-grained subpopulations. They prompt models with verbal descriptions of those groups, collect estimates at each level, and aggregate the responses back to a population-level result. They then compare those reconstructed estimates with direct population-level estimates and with results from partitions of different granularities.

The central pattern is what the authors call the “macro fallacy.” In their persona-prompting study, estimates reconstructed from more detailed subpopulation responses often aligned better with human reference data than estimates produced directly for the overall population.

That finding suggests the models may contain useful associations about individual demographic or behavioral groups without reliably carrying those associations through to a coherent aggregate answer. In engineering terms, the failure is not necessarily a lack of local signal. It is an aggregation and calibration problem exposed by changing the resolution of the prompt.

The paper says the effect persisted when researchers varied the structure of the binary tree and the estimation task. It also reports that implicit prompting could partially recover the effect, though the abstract does not specify the prompting formulation or how much recovery it produced.

For product teams, the immediate implication is practical rather than theoretical. If an LLM is being used to generate a top-down estimate, such as the expected preference of a customer base or the likely behavior of a target population, it may be worth testing a bottom-up version of the same task. That version would partition the population into defined subgroups, obtain conditional estimates, weight them using known population shares, and compare the resulting aggregate with the direct answer.

That process introduces its own risks. The partition itself can encode assumptions, subgroup weights must be credible, and repeated prompting increases cost and latency. A bottom-up reconstruction can also create a false impression of rigor if the selected subgroups are incomplete or poorly defined. But it provides a concrete consistency check that does not require a labeled benchmark for every prompt.

The authors describe statistical self-consistency as an “unsaturated, reference-free” evaluation criterion. That is potentially useful because conventional LLM evaluation often depends on curated answers, external judges, or benchmark datasets that do not cover a particular deployment context. A consistency test instead asks whether multiple answers from the same model agree with a probabilistic identity.

There are important limits to the current evidence. The arXiv submission is not identified as peer reviewed, and the available abstract does not name the frontier models tested, provide parameter counts, quantify the size of the violations, or enumerate the problem domains. It also does not establish whether independent teams can reproduce the macro fallacy across models, prompts, languages, or real production estimation tasks.

Still, the evaluation design identifies a failure mode that is easy to understand and potentially cheap to probe. Organizations relying on LLM-generated aggregate judgments should not assume that a plausible top-line answer is statistically compatible with the model’s own subgroup-level reasoning.