Combining unreliable evidence
· 33 min read
Digests are written primarily by LLMs with only readability edits by me. If this one reads like slop, please shoot me an email at max.zhuang@berkeley.edu.
Schuster, Gautam and Markert asked thirteen open language models questions about people, organizations and other things that do not exist. In one example the question is Sarah Kim’s date of birth, and two tables in the prompt give different dates: one from a government civil registry, the other from a social-media user. Because the person is invented, the models have no memory to fall back on, and on their own they lean toward the government table. The authors then showed the social-media table twice, under the same user’s handle, identical in every cell and in its source label; only the heading naming it Table B or Table C differed. Nothing new had been said. The probability the models gave the social-media answer rose by 30 points. That is an average over all thirteen models and 7,440 such conflicts, each shown in both orders. Showing the same table under two different handles, which could at least be two people, raised it by 34 points. Writing both handles at the top of one table, which makes the same claim from the same two people, raised it by 6. Schuster, Gautam and Markert
A correct combiner would have behaved differently in both cases. The repeated table adds no evidence, so it should leave the answer where it was. The single table naming both users carries exactly the claim of the two separate tables, so it should move the answer as much as they did. The models responded to how many times the table appeared, and hardly at all to how many sources stood behind it.
| What was added | Shift toward the social-media value | What a correct combiner does |
|---|---|---|
| The same table twice, under one user | 30.0 | Should not move: the second table repeats the first, from the same user. |
| The same table under two different users | 33.9 | May move, if the two users are independent. |
| One table naming both users | 6.2 | Should move as much as two separate tables from the two users would: the same two users make the same claim. |
Average shift toward the social-media value, points of probability · 13 open models, 3B to 72B · fictional facts
This digest asks how a network combines several pieces of evidence that can each be wrong, and whether it notices when they are not independent. Evidence here means any set of inputs that each carry information about a target and can each be mistaken: passages retrieved into a prompt, the features of one example, readings from several sensors, labels from several annotators, votes from several sampled answers or several models. Two pieces are not independent when their errors are related: two copies of one report, two passages descended from one source, two samples from one model whose errors fall on the same hard problems.
The answer comes in three parts. First, the correct rule counts each piece for what it adds given the pieces already seen, so a copy adds nothing. Second, a network trained on examples applies whatever version of that rule its training data implied. Dependence that was fixed and present throughout training is therefore accounted for, as far as the network’s capacity, regularization and training allow, and the network does not double count it. Dependence that changes from one input to the next is another matter: by the argument and the computations below, the network can discount it only when four things hold. The dependence is visible in the input, meaning something in the input differs when the pieces are dependent. Training varied that visible feature. Training rewarded a different answer when it was present. And the architecture lets it change how much the other inputs count. Otherwise the network gives each piece the weight it earned in training. Third, one fact limits every method (networks, majority votes and label aggregators alike): the reports themselves cannot reveal how their errors are related, so nothing can infer the dependence from agreement alone without assuming something about its form.
The correct rule, and what a copy is worth
Start with two witnesses, each right 70% of the time about a yes-or-no question, and no reason beforehand to favour either answer. The natural way to combine them works in odds. A witness who is right 70% of the time says “yes” seven times for every three when the answer is yes, and three times for every seven when it is no, so hearing “yes” multiplies the odds of yes by 7/3, about 2.33. That multiplier is the witness’s likelihood ratio: how much more probable the report is if the answer is yes than if it is no. From even odds, one “yes” gives odds of 2.33, a probability of 70%. If a second witness also says yes, multiplying again gives odds of 5.44, a probability of 84.5%.
The second multiplication assumed something. It used the second witness’s likelihood ratio on its own, as if the first report had not been heard. That is right only if the two witnesses’ errors are unrelated once the true answer is fixed, which is what conditionally independent means. If the second witness is repeating the first, the second “yes” was certain whichever answer is true, so its likelihood ratio given the first report is 1, and the right answer stays at 70%. Counting the copy anyway overstates the belief by 14.5 points. Five identical reports counted as independent give 98.6%, and ten give 99.98%, while the truth stays at 70%. The numbers in this section, from the two witnesses onward, were computed for this digest from the definitions.
A wrong belief is not always a wrong decision
Naive Bayes is the classifier that does what the second multiplication did: it multiplies every input’s stand-alone likelihood ratio, as if all the inputs were conditionally independent. It is known to classify well even when they are not, because a distorted probability need not change which answer comes out on top.
Domingos and Pazzani worked through a case with two equally likely classes, one attribute that exactly copies another, and a third, independent attribute. Naive Bayes counts the copy twice. It still picks the same class as the correct rule everywhere except in two narrow regions of the possible combinations of the class probabilities the attributes imply. By this digest’s calculation those regions cover about a tenth of the combinations. In them, the third attribute nearly balances the first, so counting the copy twice tips the sum. Domingos and Pazzani
When the only evidence is one report and its copies, equal priors mean the copies can never flip the decision, because multiplying the odds again leaves them on the same side of even. With an uneven prior, one copy can. Start at 20% for yes. One witness’s yes gives 36.8%, still a no. Counted twice, it gives 57.6%, a yes. So double counting almost always distorts the belief. It changes the decision only when the prior is uneven, the costs of the two kinds of error are uneven, or the evidence is nearly balanced. This digest is about the belief, because the belief is what a model reports as its confidence and what any later step builds on.
How far off the naive sum is
In general each piece should contribute its likelihood ratio given the pieces already seen. Define the dependence among the pieces, under one answer, as the logarithm of how much more likely the pieces are to occur together than independence would predict. Adding up the stand-alone ratios in log-odds, the scale on which they add, is exact when the pieces are conditionally independent, and more generally when their dependence is the same under both answers. Zhang’s Theorem 1, written in log-odds, says the naive sum is off by the dependence when the answer is yes minus the dependence when the answer is no. Zhang
The theorem has a consequence that is easy to miss: agreement between reports is not the problem. Two accurate, independent witnesses agree more often than chance because both track the truth (58% of the time, against 50%, for the two 70% witnesses), and that agreement is the evidence. What costs something is the difference between the two answers in how much more likely the observed agreement is than independence would predict.
For a copy the difference is large. Two matching “yes” reports from a witness and its copy are 1/0.7 times as likely as independence predicts when the answer is yes, and 1/0.3 times as likely when it is no. Dividing the first by the second gives 3/7, which exactly cancels the 7/3 the copy would otherwise have added.
A partial dependence earns a partial discount. Suppose the two witnesses’ errors have a correlation of 0.5. For two equally accurate witnesses this is the same as the second copying the first half the time and otherwise reporting on its own. Their agreement then justifies 75.3%. That is what 1.32 independent witnesses would give, in the sense that multiplying the odds by 7/3 a total of 1.32 times, rather than twice, reaches the same belief. At an error correlation of 0.2 the pair is worth 1.64 witnesses, and at 0.7 it is worth 1.17.
With more than two witnesses, the correlation between pairs no longer settles the answer, because the same pairwise correlation can come from different mechanisms. Take ten witnesses, each right 70% of the time, all saying yes. Independent witnesses would justify 99.98%. With an error correlation of 0.2 between any two, the answer depends on what produces the correlation.
- Shared difficulty: each question has its own error rate, averaging 30% across questions but higher on some and lower on others (drawn from a beta distribution chosen to give that 0.2 correlation), and the witnesses err independently given that rate. Then ten agreeing witnesses justify 96.5%.
- A common source: a fifth of the time all ten repeat one shared report, and otherwise they report independently. Then ten agreeing witnesses justify 73.0%.
Under the common source, unanimity past three witnesses becomes evidence that the shared report was used rather than evidence for the answer. Nine of ten agreeing (99.9%) is therefore stronger evidence than ten of ten.
- Independent witnesses
- Shared difficulty
- Common source, a fifth of the time
- Exact copies of one
Probability justified when all agree · each witness right 70% of the time · even prior
The reports cannot reveal their own dependence
Every method in the rest of this digest runs into one fact. Consider three pairs of witnesses. The first pair is two independent witnesses, each right 70% of the time. The second is two witnesses right 60% of the time whose errors correlate at 0.125. The third is two witnesses right only half the time whose errors correlate at 0.16. All three pairs produce exactly the same reports at the same rates: both say yes 29% of the time, both say no 29%, and they split 42% of the time. Yet when both say yes, the first pair justifies 84.5%, the second 67.2%, and the third 50.0%, since witnesses right half the time carry no information at all.
Nothing in the reports tells the three pairs apart. What does is outside them: labelled outcomes, showing when each witness was wrong; knowledge of where each source came from; or an assumption about the form the dependence takes. Methods described later that learn dependence from unlabelled votes work by making such an assumption, typically that only a few pairs of sources are dependent and the rest are independent and better than chance. Labels are expensive for this purpose. Pinning down an error correlation near 0.2 to within 0.1 either way, with 95% confidence and each source’s accuracy already known, takes about 1,000 labelled items that both sources of a pair have reported on. Ten sources have 45 pairs, so unless every source reports on the same items, the labels needed multiply.
What a trained network computes
A network is not handed likelihood ratios. It is trained on examples to predict a label from all of its inputs at once, usually by minimizing log loss, the average negative log-probability it gives the true label. Its last layer adds contributions in log-odds, which looks like naive Bayes. Does it end up double counting the way naive Bayes does?
It does not, because its contributions are fitted to the inputs as they occurred together in training, not estimated one input at a time. The smallest case shows how. Logistic regression gives each input one weight, adds the weighted inputs, and turns the sum into a probability. Trained on one cue, coded +1 or −1, that is right 70% of the time, it learns a weight of 0.847, which is the cue’s log-likelihood ratio, the logarithm of 7/3. Now duplicate the cue in every training example. Started from zero, the model learns two weights of 0.424, which sum to 0.847, and predicts 70%, the truth. (With a ridge penalty, which charges the fit for the sum of its squared weights, the two weights also split evenly, and their sum approaches 0.847 as the penalty shrinks.) Gradient descent gets there because two identical columns receive identical gradients at every step: the pair behaves like a single cue learned at twice the rate, and its total converges to the single cue’s weight. So, at convergence, a trained model does not double count a dependence that was present throughout its training.
These logistic-regression results are exact fits to the whole training distribution, computed for this digest. That duplicated inputs share one weight is Zou and Hastie’s Lemma 2, proved for least squares with any strictly convex penalty. The same argument, that averaging the two weights leaves the fit unchanged and lowers the penalty, holds for log loss (this digest’s extension). Zou and Hastie
Four limits apply: what the model can represent, what its penalty favours, what the data leave open, and what training reaches.
The first is that the readout, the last layer that turns the network’s internal features into log-odds, must be able to represent the right answer. Copies of one cue leave the correct log-odds a sum, one term per cue, so one weight per input can match them. A common source that nothing in the input reveals does not. Take ten sources that a fifth of the time all repeat one shared report. The correct belief is 84.5% when six say yes and four say no (a split rules out the shared report, so the net two votes count as independent), and 73% when all ten say yes. No single weight per source gives both, because the right contribution of each source depends on what the others said. In a computation for this digest, logistic regression fitted to that training distribution reports 64% for the split and 95% for unanimity, wrong in both directions on the data it was trained on. The best of ten randomly started networks with three hidden units, which can let one input’s contribution depend on the others, recovers both answers to within a tenth of a point.
The second is that regularization pulls the fit toward treating inputs as independent. As the ridge part of the elastic net’s penalty grows (the elastic net is a least-squares fit penalized for both the absolute and the squared size of its weights), Zou and Hastie show that the least-squares fit becomes a rule that “totally ignores the dependence between predictors and treats them as independent variables”. A strong penalty also lets a duplicated feature keep more total weight than one copy would. For example, under a ridge penalty in least squares, with each feature scaled so that its squares sum to the number of examples, two copies carry 4/3 of the single feature’s weight when the penalty equals the number of training examples (computed for this digest).
The third is that how exactly redundant inputs share their weight is not fixed by the training data. The data fix only the total; the initialization and the penalty decide the split. The split is invisible while the inputs stay redundant and decides the answer once they come apart. A logistic regression trained on an exact copy and then shown two independent cues that disagree reported 29%, 56% and 63% from three random initializations, where the truth is 50% (computed for this digest).
The fourth is that a network trained on finite data may not reach the rule its data imply, and which cue it learns first can decide what it uses. Hermann and Lampinen gave a five-layer network, trained on 256 examples, an easy cue and a harder cue that was right 90% of the time, the two independent given the label. They tested it on data in which one cue at a time was made uninformative. The network relied more on the easy cue than on the harder one once the easy cue was right somewhere between 70% and 80% of the time: their text gives 80% as a level at which it already did, and the fitted lines in their figure cross near 70%. The correct rule relies on the harder cue alone until the easy one is right more than 90% of the time (this digest’s arithmetic). The network weighted the cues by how easy they were to learn rather than by how reliable they were. Hermann and Lampinen
When the dependence changes at test time
Since the weights are fitted to the dependence the training data had, a change in that dependence at test time miscalibrates the model in the direction of the change. A model trained on two independent 70% cues and then shown a cue together with its copy says 84.5% where the truth is 70%; a model trained on ten independent cues and shown ten copies of one says 99.98%. A model trained on copies and then shown two independent cues says 70% where the truth is 84.5%. These are again exact computations for this digest; no published study that this digest could find tests this in a trained classifier.
The model cannot be blamed for not noticing, because there is nothing to notice. Two agreeing cues look the same whether one copied the other or not. Only the probability of the label given that input has changed, and that is the fact from the previous section in another form: the reports alone cannot reveal their dependence.
When a network can learn to discount
Suppose instead that the dependence varies from one input to the next, so that two cues are sometimes copies and sometimes independent. Then the right weight depends on the input, and the question is what it takes for a network to learn it. A small computation for this digest answers it in the simplest case. A network with one hidden layer of 16 units was trained on an even mix of copied and independent pairs of 70% cues, with ten random initializations for each condition. The one exception is the network given a flag that was always off, which was trained on independent pairs only.
| What the network was given | Both cues agree, independent (truth 84.5%) | Both cues agree, copied (truth 70.0%) |
|---|---|---|
| A flag marking copies, varied in training | 84.5% | 70.0% |
| No flag | 75.3% | 75.3% |
| A flag that was always off in training | 84.5% | 76% to 87%, by initialization |
| The varied flag, but in logistic regression | 75.3% | 75.3% |
With a visible flag that training varied, the network learned the discount to within a tenth of a point across all ten initializations. Without the flag it gave both kinds of input the same 75.3%, which is the right answer averaged over the training mix and wrong for each kind. With a flag that never varied in training, it had no gradient telling it what the flag meant, and its answer on flagged copies depended on its initialization. And logistic regression could not use the flag at all, because the flag has to change how much the two cues count, which a model that only adds a fixed amount per input cannot express.
So, by this digest’s argument and the computation above, a network can learn to discount dependent evidence only when four things hold. The dependence is visible in the input: something in the input differs when the pieces are dependent. Training varied that visible feature. Training rewarded a different answer when it was present. And the architecture can let it change the weight of the other inputs. In text, a verbatim copy is visible in this sense. Two reports in different words that descend from one upstream source are not, unless something in the text says where each came from.
A transformer has the machinery both to see a verbatim copy, by comparing positions, and to let what it sees change how much each passage counts. Its output, up to a final rescaling, is a sum of what each layer and attention head wrote, read off as logits. Attention itself computes a weighted average over positions, so it registers shares rather than counts. Duplicating one passage among others raises that passage’s share. With no position information, no causal mask (the rule that a position can attend only to earlier ones) and no token that takes up most of the attention, duplicating every passage changes nothing: the head’s output is identical.
A head that places most of its attention on the first token behaves differently. That token’s large, fixed score dominates the sum that the attention weights are divided by, so each added copy takes about the same share as the last, mostly from the first token, and the copies’ combined share grows almost linearly until it rivals the first token’s (this digest’s arithmetic). Such heads are common: beyond its bottom two layers, Llama-2-7B attends heavily to the first token in every head.
Either way, a head that only averages over passages has no way to register that two positions hold identical text. Detecting a copy takes comparing positions with each other. Pretrained models do build such comparisons where their training rewarded them: GPT-2 small has heads that detect repeated tokens and a head that suppresses predictions copied from earlier in the context. Elhage et al., Barbero et al., Xiao et al., Wang et al., McDougall et al.
No study that this digest could find trains a transformer on reports with varying, visible dependence and tests whether it learns to discount the copies while still counting independent agreement. The behaviour of pretrained models is measured, and it is the subject of the next section.
What language models do with repeated evidence
The repeated-table experiment satisfies the first condition completely, and transformers have the machinery for the fourth. The copy matched the original in every cell and carried the same source label, so the dependence was as visible as it can be. The thirteen models, from 3 to 72 billion parameters across the Qwen2.5, OLMo-2, Llama-3 and Gemma-3 families, counted it almost like a second source. By this digest’s reading, what was missing was the training signal. The authors added an instruction asking the models to weigh each source’s credibility and adding: “Simple repetition of information should not influence your choice.” It weakened the effect, but in twelve of the thirteen models not enough to restore the original preference for the government source.
They then fine-tuned Gemma-3-4B to give the same answer with and without a repeated table, training on 1,500 conflicts with newspaper and person sources only. Used together with a shorter prompt that asks only that the sources’ credibility be assessed, without the sentence about repetition, the fine-tune cut the shift caused by a repeated table by 78%. That was measured on entities and source types the training had not seen (government against social media), averaged over seven training runs. After fine-tuning, though, a second, differently named social-media source moved the model about as little as a repeat did (read off the paper’s figure). What it learned therefore looks like indifference to agreeing tables in general, whatever their source, rather than detection of copies. Schuster, Gautam and Markert
Why would a model count copies by default? The four conditions suggest an answer, which is this digest’s interpretation rather than a tested result. A language model is trained on ordinary text, where a statement that appears more often is more often what comes next, and nothing in that training pays for treating a second, identical occurrence as uninformative. The network computes what its training implied, and its training implied that repetition is evidence.
Five further measurements describe how much repetition moves a model and what competes with it: how fast further copies stop adding, how little extra passages dilute a claim, order, the model’s own prior, and near-copies. A last paragraph turns to defenses.
Further copies add less than the first. Kurfali and Östling placed three conflicting answers to a question with no true answer in a 5,000-word document, across eight open models. Showing one answer’s sentence twice, one extra copy, raised its odds of being chosen between 3.3 and 26 times. Showing it five times, four extra copies, raised them between 2.3 and 63 times. If each copy added a fixed amount of evidence, as counting independent reports would, five showings would give four times the log-odds boost of two showings. By this digest’s arithmetic on their table, the ratio of the two boosts was between 0.66 and 1.69, against the 4 that a fixed amount per copy predicts, and below 1 for two of the eight models.
Passages written separately for one false answer, by the attack known as PoisonedRAG, appear to keep adding for longer. They are separate generations of one claim, so they sit close together in wording, but they are not verbatim copies. When an attacker wrote between one and five of the five passages retrieved for a question from Natural Questions, a set of real search queries, PaLM 2 gave the attacker’s answer 48%, 76%, 84%, 90% and 100% of the time. That is between half a unit and 1.2 units of log-odds per added passage (this digest’s arithmetic on their table), though each attacker passage there also displaced a genuine one. The contrast between verbatim copies and separately written passages inside a conflict comes from different papers, models and tasks; no study has compared them in one design with conflicting evidence. One study that compared them without a conflict found that copies and paraphrases of a document did not measurably improve correctness while distinct documents did; inside a conflict, the near-paraphrases above kept adding. Kurfali and Östling, Zou et al., Ross et al.
More passages dilute a planted claim far less than its share of the context would predict. Widening a BERT-based fact verifier’s evidence from 5 to 10 sentences left the damage from one poisoned sentence unchanged, a loss of 23.87 points of accuracy against 23.69. On HotpotQA, questions that need two documents, PaLM 2 gave the attacker’s answer 41% of the time with five attacker passages among fifty retrieved, down from 99% when those five were all it read (with access to the retriever, 43% against 94%).
Which passage wins depends strongly on how relevant it is to the question: in one study, passages in the most question-similar quarter won 62% to 69% of conflicts and those in the least similar quarter 27% to 38% (read off the paper’s figures). Du, Bosselut and Manning, Zou et al., Wan, Wallace and Klein
Order can move answers as much as count does, or more. The repeated-table study showed every conflict in both orders, so its headline averages over order; studies that do not shuffle the order measure order along with count. In sentiment classification with four labelled examples in the prompt, a 2.7-billion-parameter GPT-3 was moved far more by the last example’s label than by the count of positive examples. With the counts held fixed, the last label changed the share of test reviews it called positive by 45 to 57 points. Going from one positive example to three, with the last label fixed, changed it by 7 to 12. (This digest’s arithmetic on values read off the paper’s figure; one model, no variance reported.) With one passage holding the answer among twenty, GPT-3.5-Turbo answered correctly 75.8% of the time when it came first and 53.8% when it came tenth. At tenth, it did worse than the 56.1% it reached with no passages at all. Zhao et al., Liu et al.
The model’s prior, its own answer before reading the passages, pulls against them. When GPT-4o’s own answer was right and the single passage it was given was wrong, it adopted the passage 60.8% of the time; Claude Opus did so 31.3% of the time. For the three models whose token probabilities could be read, GPT-4o, GPT-3.5 and Llama-3, the more probability a model gave its own answer, the less often it adopted a contradicting passage. With fictional facts, as in the repeated-table study, the model has no stored answer to resist, so only its preference among kinds of source stands against the count (this digest’s inference). Wu, Wu and Zou
Qwen2.5-7B, undefended, did not discount those near-copies either. PoisonedRAG’s passages, which sit close together in wording and in the retriever’s representation, and passages written in deliberately different styles succeeded about equally against it on HotpotQA, 91% and 92% of the time (one run, 100 questions). Chen et al., “Divide and doubt”
Defenses that let each passage cast one vote treat passages as independent votes, so a few planted passages that agree count as several votes. One certified defense asks the model about each passage separately and answers from the keywords that many of those replies share. For each question it then proves, where it can, that a given number of corrupted passages could not change the answer. Its guarantees assume that honest passages outnumber malicious ones, because it counts passages rather than sources. With its default settings on one news-question dataset, it proved a correct answer for 38% of questions against one corrupted passage among ten and for none against two (read off the paper’s figure); stricter settings certified more. Xiang et al.
Voting over samples and models
Voting is the other common way to combine fallible answers: sample several answers from a model, or ask several models, and take the most common one. Its promise comes from Condorcet’s jury theorem. If independent voters choosing between two answers are each right with the same probability above one half, the majority is right more often than any one of them, and with enough voters it is almost never wrong. Forty-one such voters, each right 76.8% of the time, give a correct majority 99.99% of the time, even if every wrong voter gives the same wrong answer.
Brown and colleagues released 10,000 sampled answers per problem from Llama-3-8B-Instruct on 127 grade-school math problems from GSM8K, and they make the comparison possible. One sample is right 76.8% of the time. A vote over 41 samples is right 86.7% of the time, and a vote over 1,001 samples 88.2%. The vote helps, and it stops far short of independent voters. These figures are this digest’s reanalysis of the released samples. Brown et al.
- Independent yes-or-no voters, equally accurate on every problem
- Measured vote, Llama-3-8B
Share of 127 GSM8K problems the vote gets right · one sample is right 76.8% of the time
The reason is not that samples copy each other. Given the problem, each sample is a fresh draw. What correlates them is the problem: the same problems are hard for every sample. In the same reanalysis, if one sample gets a problem wrong, another sample on the same problem is wrong 59.6% of the time, against 23.2% for a sample on a randomly chosen problem. Lingjiao Chen and colleagues proved that on each problem the vote converges to the answer the model gives most often, so in the limit it is right exactly where the right answer is the model’s most likely one. With two possible answers, they also proved that more samples help on those problems and hurt where a wrong answer is more likely. Here the right answer is the most common one on 112 of 127 problems, which is where the vote stops. Chen et al.
Wrong answers also coincide, which matters because a wrong answer that many samples share can become the most common one, and then more votes entrench it. In this digest’s reanalysis of Brown and colleagues’ samples, which also include Llama-3-70B, two wrong samples on the same problem give the same wrong answer 36% of the time for Llama-3-8B and 77% of the time for Llama-3-70B. Across 71 models on four-option MMLU questions, when two models are both wrong they pick the same wrong option 60% of the time, against 33% by chance. Kim et al.
Panels of different models do not escape this. Kohli had nine judge models from seven companies label 1,000 sentence-pair inference items; each judge disagreed with the human majority on 28% to 36% of them. The panel’s majority was right 72.0% of the time, and the best single judge 71.8%. Kohli estimated that nine judges with these error patterns, erring independently within each level of item difficulty, would give a correct majority about 94% of the time. When all nine agreed, they were wrong 9.1% of the time, where independent judges would be wrong about 0.02% of the time.
Kohli also summarized the panel as worth about 2.2 independent judges. That figure comes from the judges’ average pairwise error correlation, 0.39, through the standard formula for the precision of an average of equally correlated estimates: nine divided by one plus eight times 0.39. By this digest’s reading, the formula measures how much averaging the judges reduces their noise, not how much evidence their vote carries, although Kohli presents it as the number of independent votes the panel contains. The ten witnesses above show the difference. At their correlation of 0.2 the formula makes ten witnesses worth 3.6, while their unanimous vote is worth what 3.9 independent witnesses would give under shared difficulty, and what 1.2 would give under a common source. Raising the temperature or rewording the prompt left Kohli’s 2.2 unchanged, and asking for written reasoning lowered it to 1.9. On two other inference datasets and a preference benchmark the same measure gave 2.0 to 2.5. This is one single-author preprint on one panel, and its numbers should not be carried to other panels. Kohli
When the voters are equally accurate and their errors are related only through shared question difficulty, picking the most common answer is still the best rule. What fails is the promise attached to it, that more voters drive accuracy toward certainty and that unanimity is near-certain. Ai et al.
When the voters differ in accuracy, the plain rule itself can fail. Pooling 100 samples each from the 8B and 70B models gives 96.1% on these problems, below the 96.9% of the 70B samples alone, because on one problem the 8B model’s confident wrong answer outvotes the 70B model’s right one (this digest’s reanalysis).
Aggregators that learn reliability from agreement
Weak supervision, which labels training data with cheap, fallible rules instead of people, and crowdsourcing face the same problem with an extra difficulty: there are no labels, so the aggregator must estimate how reliable each source is from how often it agrees with the others. The Dawid–Skene model for annotators assumes the sources err independently given the true label, and so does Snorkel’s label model for programmatic labelling rules unless it is given, or learns, which rules are correlated. Under that assumption, agreement between copies can only be explained as reliability, so copies look more reliable than they are, and in the two cases below, perfectly reliable.
Snorkel’s authors state the extreme case. Five labelling rules that always vote together and are right half the time sit beside five independent rules that are right 99% of the time. A label model that assumes independence has its best fit at 100% accuracy for the five copies and 50% for the five good rules. Worked through for this digest, that fit labels every item by the copies’ coin flip, each label held with confidence 1.0, while a plain majority vote of the ten would be right 73.8% of the time (ties broken at random). The authors use the example to argue for modelling the dependence. Ratner et al.
One duplicate can be enough. In a simulation for this digest, five independent sources each right 70% of the time were joined by a sixth 70% source entered twice. Fitted by expectation maximization, the independence model rated the duplicated source 100% accurate and the others 58%. It then followed the duplicated source on every item and was right 69.9% of the time, against 83.8% for the correct rule. It put a confidence of 0.9 or more on wrong labels for 30.1% of all items. Majority vote stayed at 83.8%, and with the sixth source entered five times it fell only to 72.4%. This is one simulated run on synthetic sources, meant to show the mechanism, not a rate for real systems.
Real systems are fragile in the same direction, though less cleanly. Lee and colleagues used an earlier study’s scores from up to 33 reward models and judge models on 100 sampled answers per question, for five math, science and general-knowledge benchmarks, with the answers generated by a smaller and a larger model: ten settings in all. The verifiers’ scores were correlated given correctness. The Dawid–Skene model picked worse answers than a plain average of the scores in all ten settings, by 9.7 points on average (this digest’s arithmetic on the paper’s tables). A model that allows structured dependence was also below the plain average in all ten, by 4.8 points, so the paper does not isolate dependence as the cause.
In another study, two copies of a bad labelling rule added to twelve rules for movie-review sentiment dropped the F1 score of a classifier trained through Snorkel from about 68 to about 20 (read off the paper’s figure, ten seeds). The same models also degraded when given three independent useless rules, so that experiment measures fragility to bad sources as much as to copies. Lee et al., Rühling Cachay, Boecking and Dubrawski
Naive Bayes and these aggregators fail in different ways. When each source’s reliability is fixed, as in naive Bayes, ignoring dependence mostly inflates confidence and changes few decisions. When the reliabilities are learned from agreement, ignoring dependence can hand the whole decision to the copied source, which is worse than not weighting the sources at all. In the simulation, a plain vote degraded gradually as copies were added, while the learned weighting collapsed at the first one.
Representing the dependence, and what it costs
There are four ways to give an aggregator what the reports cannot, and each has its own price.
The first is to estimate the dependence from co-occurrences: how often each pair of sources errs together on the same items when labels exist, or votes together when they do not. This needs many items that both sources of every pair have voted on, and real data often lack them. In nine of seventeen public crowdsourcing datasets, the median pair of workers who labelled any item in common shared only one to six items (computed for this digest on the seventeen datasets released with Li, Rubinstein and Cohn’s model of correlated annotators). It can also cost more than it returns, as the next two results show. Tree-augmented naive Bayes, which lets each input depend on one other input, beat naive Bayes on 17 of 25 datasets with smoothed estimates and lost on 14 of 25 without smoothing (this digest’s count from the paper’s tables). Adding more than a handful of sensible dependencies to a label model, even ones chosen using the true labels, lowered downstream AUC by up to about 4 points on one dataset and 8 on another (read off the paper’s figure).
Where it works, the measured gains on real weak-supervision tasks are modest, 0.6 to 4.6 points of F1 or accuracy in single runs. These methods learn the dependence from unlabelled votes by assuming that only a few pairs of sources are dependent. The large gains found so far all come where near-duplicate sources exist: among eleven prompts to a language model, used as labelling rules, that included three near-identical ones, learning the dependence from the prompts’ votes raised F1 on a relation-extraction task from 52.1 to 63.9, over five seeds. Friedman, Geiger and Goldszmidt, Rühling Cachay, Boecking and Dubrawski, Bach et al., Varma et al. 2019, Su et al.
The second is to read the dependence from how the sources were built. Two labelling rules that call the same piece of code, or two prompts whose wording is nearly the same, are dependent for a reason anyone can see. Static analysis of labelling-rule code found dependencies that learning which rules depend on which from their votes missed and added 2.9 and 3.8 F1 points on two image and video tasks. On the relation-extraction task above, reading the dependence from the similarity of the prompts’ wording did about as well as learning it from their votes, 64.8. This route needs no co-occurrence data, and it catches only the dependence that is visible in the sources’ description. Varma et al. 2017, Su et al.
The third is to assume the worst. Covariance intersection, from sensor fusion, combines two estimates without knowing how their errors correlate. Provided each estimate’s own stated uncertainty is honest, the result is never overconfident, whatever the correlation is. Two equally uncertain estimates combine to the uncertainty of one, which is exactly right if they are copies and gives up the gain if they were independent. The price is paid every time the sources were in fact independent. Denœux’s cautious rule, a related worst-case rule for combining evidence that treats every pair of sources as possibly overlapping, shows both sides. In a simulation of ten Gaussian features, nine of them sharing one correlation, it scored an AUC of 0.941 against 0.986 for the independence rule when the features were independent. When they correlated at 0.9, it scored 0.830 against 0.798 (recomputed for this digest). Uhlmann, Denœux
The fourth is to fit the combination to labelled outcomes, which is what the networks earlier in this digest did. With labels, training builds the dependence that the labelled data contain into the weights. A single refitted scale on naive Bayes’s log-odds removes most of its miscalibration in a simulation where nine features share one correlation, though not the extra misclassifications the dependence causes (computed for this digest). With few labels, the fitted combination can do worse than ignoring the labels. In the verifier comparison above, logistic regression fitted to the verifiers’ scores with labels for 5% of the questions was the best method in four of the ten settings, but it fell below a plain average of the scores in four of the five settings with the smaller answer-generating model. Lee et al.
None of the four creates information the sources did not jointly hold. For the nine judges, with the gold labels available, weighting them by accuracy closed less than 1% of the gap to what independent judges would reach. Weights computed from their error correlations closed a fifth of the gap on one of four datasets, and on another scored below a plain majority. Kohli
Dependence is not harmful in itself. The witnesses earlier were all equally accurate; with unequal accuracy, known dependence can help. Two estimates with error variances of 1 and 4 whose errors correlate at 0.9 combine, if the correlation is known, to a variance of 0.54. That is below the 0.80 that two independent estimates of those variances would give, because the noisier estimate gets a negative weight (computed for this digest). Clemen and Winkler make the general point, and warn that the gain is hard to realize when the correlation must be estimated. The damage comes from dependence that is ignored or unknown. Clemen and Winkler
What is settled and what is open
Settled: the correct rule, and Zhang’s identity for how far the naive sum is off. A model trained with log loss to convergence on enough data accounts for dependence that was fixed and present in its training, within the limits its capacity and regularization set. The reports alone cannot reveal their dependence. A majority vote over samples that are independent given the question converges, question by question, to the most common answer, so its accuracy stops at the share of questions whose most common answer is right, however many samples are added.
Measured, in a narrower setting than its importance suggests: that language models count a visible copy almost like a second source. The cleanest evidence is one study of thirteen open models, from 3 to 72 billion parameters, on fictional facts presented as tables. A second study, of eight open models, found that repeated copies raise an answer’s odds with sharply diminishing returns after the first extra copy. Whether larger proprietary models behave the same way has not been tested in this design. Shown analytically and in simulation, but measured only loosely: aggregators that learn reliability from agreement without labels can hand the decision to a duplicated source. Where copies of a random rule were added to a real task and their estimated accuracy read off, the independence model overrated them (about 0.63 to 0.68 against a true 0.5) rather than taking them for perfect. Other real benchmarks show fragility in the same direction without isolating copies.
Open, with the cheapest experiment for each:
- Does a transformer trained on reports with varying, visible dependence learn to discount copies while still counting independent agreement? Train a small transformer on synthetic witness reports, half copies and half independent, with and without position information, over three seeds, and test whether its answers on copies match the correct 70%. This digest found no published study that has run this.
- Can a model learn to discount copies while still counting independent agreement worded differently? Fine-tune with pairs that contrast a verbatim copy with an independent source making the same claim in other words, and test both on held-out entities.
- Do verbatim copies saturate faster than paraphrases and independent passages inside a conflict? Hold one conflict fixed and add k verbatim copies, k paraphrases, or k independently written passages, in shuffled order, in one study.
A network combines evidence by the rule its training implied, as far as its training reached it, and by this digest’s argument it notices that evidence is not independent only where the dependence was visible in the input, varied and rewarded in training, and expressible by the architecture. No combiner, trained or not, can learn the dependence from agreement alone. For a common kind of dependence, two passages that trace back to one source, the dependence is usually not visible. For verbatim copies it is visible, and transformers can compare positions, but by this digest’s reading the training of the open models tested so far has not rewarded discounting them.