Glossary
Majority voting
Majority voting selects the value reported by the largest number of sources under a defined counting rule.
Truth discovery
Majority voting is a baseline method for reconciling conflicting claims. After claims are aligned to one subject and property, each source contributes a vote and the most frequently reported value wins. The method is transparent, but its meaning depends on the unit counted: web pages, publishers and independent original observations can yield different majorities. A tie requires an explicit policy.
It is useful when sources are similarly reliable, reasonably independent and answering the same question. Those conditions should be tested rather than presumed. If many pages copy one report, simple counts amplify that report. If a specialist authority disagrees with many general directories, the numerical majority may be less informative than the authority's direct evidence.
For defensible use, record the eligible sources, deduplication rule, votes and exceptions. A majority does not establish truth; it states a reproducible aggregation result under specified assumptions. Truth discovery methods may model source reliability and dependence, while majority voting deliberately omits those distinctions and therefore serves best as a comparison point.
Example
Four directories list one address for Halden Therapeutics AG (fictional), but three copy a single filing; counting original observations changes the vote.
When voting is a sound baseline
Voting has a theoretical justification as well as a practical one. Condorcet's jury theorem shows that if each voter is independently more likely than not to be right, the probability that the majority is right rises toward certainty as voters are added. The result is exact under its assumptions and treacherous outside them: it requires that errors be independent and that each voter be better than chance on the question asked. Both conditions fail for web sources that copy each other and for hard questions on which most sources are wrong, as the discussion of source dependence makes clear. A vote is therefore trustworthy to the degree that those two conditions have been checked.
In crowdsourcing and labelling work, Snow and colleagues found that a handful of non-expert annotations per item, combined with a correction for annotator bias, could approach the quality of expert annotation on several language tasks. The finding is a useful reminder that voting often works well when the voters are many, comparable and independent, and that the improvement over simple voting comes from modelling the voters, which is the step that Dawid–Skene and Raykar and colleagues formalise.
Specifying the vote
A voting rule has to settle five things before it can be reproduced. What is counted: pages, domains, publishers or original observations. Who may vote: only sources that cover the property, only sources that were current on the date, only those not excluded by a quality rule. How ties are broken: by recency, by source priority, by leaving the value unresolved. Whether abstention counts: a source that is silent on a property has not voted for the absence of a value. And whether a margin is required: a 51 to 49 result and a 98 to 2 result are not equally informative, and the margin is worth recording.
Recording the rule, the electorate and the tally with each result converts a vote into evidence that a reviewer can re-run. The article on comparing truth-discovery methods without gold labels uses voting as the reference point against which more elaborate methods are measured, which is its most defensible role.
A worked example
Seven pages report the number of sites in a fictional programme run by Halden Therapeutics AG (fictional). Five say 12 and two say 14. The five that say 12 trace to a single press release; the two that say 14 are the sponsor's own registry entry and a regulator listing, which were compiled independently of one another. By page count the answer is 12 by a wide margin. By original observation it is 2 to 1 in favour of 14. The registry entry, being the primary record, carries more weight still. The example is not an argument against voting but against counting the wrong unit: the same rule applied to deduplicated lineages gives a different and better-supported result.
Related terms and boundaries
Voting is the simplest member of the family of truth discovery methods and the one against which the others are judged. It is limited by source dependence, which inflates counts, and by differences in source reliability, which it ignores by design. Where the voters are human reviewers and the question is categorical, the Dawid–Skene model replaces equal votes with estimated error rates, and inter-annotator agreement reports how consistent the voters were. None of these converts a tally into verification.
Checks a reviewer can run
Ask what unit was counted and how copies were removed. Ask how ties and missing votes are treated. Recompute one vote from the stored claims. Compare the vote result with the result of a method that models dependence on the same sample, and read the cases in which they differ.
Common misreadings
- Counting copied pages as independent votes.
- Ignoring a source's scope or authority.
- Failing to specify how ties and missing values are handled.
Appears in
- Majority voting vs truth discovery
A count of agreeing sources is a useful baseline; truth discovery also considers how those sources behave.
References
- Li et al. (2016), A Survey on Truth Discovery, ACM SIGKDD Explorations 17(2).
- Snow et al. (2008), Cheap and Fast, But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks, EMNLP.
- Raykar et al. (2010), Learning From Crowds, Journal of Machine Learning Research 11.
- Wikipedia, Condorcet's jury theorem.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Majority voting" bear on a decision of yours?
This entry sits under truth discovery. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.