Fairness Tests for Truth-Discovery Aggregation

Executive Summary

Accuracy alone is not an adequate acceptance test for truth-discovery aggregation. It answers whether an aggregate label matches a reference label, not whether errors, positive assignments, or missing answers are distributed acceptably across relevant groups. Li, Sun and Wang's peer-reviewed KDD study introduced theta-disparity and three intervention points, Pre-TD, FairTD and Post-TD, in 2020 ^{[1] [2]}. Their Crowd Judgment experiment used 20 labels per defendant and 20,000 answers ^[3]. The reported superiority of FairTD, including an improvement of up to 2.02 percent in some settings, is an author-reported experimental result, not a universal guarantee ^[3].

The evidence since then is cautionary. Lazier, Thirumuruganathan and Anahideh evaluated majority voting, Dawid-Skene, Learning From Crowds and FairTD in a 2023 workshop study (Source: algfair-siam23.netlify.app). On their two data settings, worker accuracy and fairness were not strongly correlated, removing workers classified as unfair reduced accuracy and increased abandoned tasks, and downstream fair-machine-learning methods did not remove label bias (Source: algfair-siam23.netlify.app). These findings should not be generalized beyond those datasets.

Singer and colleagues' January 2026 preprint adds conditional bounds for majority-vote and Bayesian aggregation and proposes FairCrowd, a post-processor enforcing strict demographic parity ^{[4] [5]}. Its real-data breadth is limited because it reuses the Crowd Judgment and Jigsaw lineage from the earlier study ^[5]. It is a preprint, not settled or independently reproduced evidence.

The defensible decision protocol is therefore multi-objective. Define the protected attribute and policy objective first. Preserve raw judgments and provenance. Measure accuracy, demographic-parity gap, true-positive-rate gap, false-positive-rate gap, calibration where relevant, and unanswered-item coverage, each by group with uncertainty intervals. Sweep the fairness threshold and small-group sample size, then retrain and re-evaluate any downstream model. Demographic parity can be measured without gold labels, but equalized-odds-style tests require a credible reference label and protected-group membership ^[6] (Source: proceedings.neurips.cc). A fairness constraint remains a normative choice, not another name for truth.

Introduction and Background

Truth discovery is the task of consolidating conflicting source claims or human judgments into an inferred value. In crowd labeling, the apparent simplicity of selecting one label hides at least four separate questions: whether a reference label is credible, whether workers differ in reliability, whether outcomes or errors differ across groups, and whether the item admits one latent truth at all. Majority voting treats workers uniformly, which a broad algorithm comparison identifies as an assumption that each worker has the same quality ^[7]. Dawid-Skene models annotator responses, but its standard formulation depends on conditional independence among those responses (Source: proceedings.neurips.cc). Learning From Crowds jointly learns from multiple noisy annotators without assuming an absolute gold standard ^[8].

None of these modeling moves settles fairness. A source-reliability estimate may absorb systematic group disagreement and make it look like low worker quality. Dependent sources may repeat the same incorrect value, so frequency is not independent corroboration ^[9]. Even sophisticated iterative methods can settle at local optima, and a broad comparison of 17 algorithms across five real datasets found rankings unstable across datasets ^[7].

This report addresses bias in truth discovery algorithms, truth discovery fairness metrics, fair aggregation of conflicting data sources, source reliability bias in data fusion, how to test aggregation algorithms for fairness, the accuracy fairness tradeoff in data aggregation, fairness auditing for information fusion, and group fairness in truth discovery. The unit of analysis is the aggregation pipeline, not merely the final classifier. The central test is whether the chosen aggregate is accurate enough, fair under a justified definition, adequately covered, statistically supportable, and reproducible from retained evidence.

Key Changes

The 2020 FairTD formulation moved fairness into truth discovery

Li, Sun and Wang defined theta-disparity as a bound on differences in positive inferred-truth probabilities between protected and unprotected groups ^[1]. This is structurally close to demographic parity: it constrains selection rates, not group-specific error rates conditional on a reference truth. The distinction matters because equal positive rates can coexist with unequal false-positive and false-negative rates.

The paper separated three places to intervene:

• Pre-TD changes or filters worker answers before aggregation.

• FairTD incorporates the fairness condition during truth discovery.

• Post-TD adjusts already aggregated results to meet the condition.

The authors reported that FairTD produced the best accuracy subject to theta-disparity among their approaches ^[1]. Because random initialization was used, their reported results were averaged over 100 runs ^[3]. That experimental care does not establish external validity. It does establish a useful design pattern: fairness can be treated as a measurable constraint at more than one point in aggregation.

The 2023 critique separated worker fairness from aggregate performance

Lazier and colleagues tested majority voting, Dawid-Skene, Learning From Crowds, and FairTD on Crowd Judgment and a 14,000-comment Jigsaw subset spanning five content categories (Source: algfair-siam23.netlify.app). They found the three conventional algorithms' accuracies roughly comparable in their experiment (Source: algfair-siam23.netlify.app). More importantly, removing workers categorized as unfair lowered overall accuracy and increased unanswered tasks (Source: algfair-siam23.netlify.app).

This result identifies a missing axis in many fairness audits: coverage. A method can improve a reported disparity by refusing to answer more often for one group, or by removing workers until difficult items lack sufficient votes. If unanswered items disappear from the denominator, the reported accuracy can rise while operational usefulness falls. Accuracy, disparity, and coverage must therefore be reported together, on the same population and with explicit missingness rules.

The study also found that applying fair-machine-learning algorithms downstream did not eliminate problems introduced by biased labels (Source: algfair-siam23.netlify.app). That is evidence from this study, not proof that every downstream mitigation fails. It nevertheless supports an important audit rule: test the labels before model training and test the trained model afterward.

The 2026 preprint added conditional bounds and strict post-processing

Singer, Gruffaz, Vo Van, Vayatis and Kalogeratos analyze majority vote and Bayesian aggregation under stated annotator-quality conditions. They derive a non-asymptotic bound and report exponential convergence of aggregate fairness toward ground-truth fairness under those conditions ^[5]. Their synthetic experiments show majority voting amplifying bias in a small-crowd regime ^[5]. The condition is the finding: convergence is not automatic merely because more votes are collected.

Their FairCrowd method post-processes any aggregation rule to meet strict demographic-parity constraints ^[5]. Demographic parity is attractive partly because it can be estimated without ground-truth labels ^[5]. That convenience does not make it appropriate for every task.

Table 1 summarizes the progression and makes the dataset lineage explicit.

The table shows progression in method, not three independent replications. The change from a 14,000-comment subset in the workshop study to a 5,000-comment subset in the preprint is also a reminder that a shared dataset name does not imply an identical evaluation sample. Dataset version, filtering, group definition, and split must be recorded.

Choosing a Fairness Test

Separate the estimands before selecting a metric

An estimand is the quantity an evaluation intends to estimate. For aggregation, at least four estimands should be separated:

• Factual agreement: how often the aggregate matches an independently established reference label.

• Source reliability: how each worker or source behaves across items, classes, and groups.

• Group fairness: how selections or errors differ between defined groups.

• Subjectivity retention: how much meaningful disagreement is preserved instead of forced into one class.

Accuracy is the fraction of correctly classified samples ^[10]. It is not a source-reliability model, a fairness definition, or evidence that one ground truth exists. Research on disagreement increasingly treats annotator differences as pervasive and meaningful, not automatically as noise ^[11]. Soft labels can encode a distribution over annotator labels rather than collapse every item to one class ^[12].

Match the fairness definition to the decision

Demographic parity asks for comparable positive-assignment rates across groups. Fairlearn operationalizes its gap as the largest group selection rate minus the smallest ^[13]. It can be useful when selection itself is the harm of concern or when reference labels are absent. It can be inappropriate when base rates legitimately differ, when a positive label has different semantics by group, or when equal rates mask unequal errors.

Equalized odds asks for parity in both true-positive and false-positive rates ^[6]. Equal opportunity focuses on true-positive rates. Both need a defensible reference label. Predictive parity asks whether the fraction of correct positive predictions is the same across groups ^[14]. These criteria need not be jointly satisfiable when prevalence differs. Kleinberg, Mullainathan and Raghavan prove that three central conditions generally cannot all hold except in constrained cases ^[15].

Individual fairness requires a task-specific similarity metric, making the definition of similarity a substantive choice ^[16]. Intersectional auditing is necessary because parity on broad groups can conceal violations in structured subgroups (Source: proceedings.mlr.press).

Table 2 is a decision table, not a ranking. The appropriate row depends on the decision and the evidence available.

No row turns fairness into truth. OECD analysis states directly that choosing a fairness definition involves moral judgment ^[14]. NIST likewise warns that fairness metrics are context-specific and not reducible to one concise mathematical definition ^[17]. The test selection should therefore be documented as a governance decision with a stated harm model, not inferred from whichever metric is easiest to compute.

Implementation Considerations and Process Changes

Build an evaluation protocol before changing labels

A defensible protocol should be pre-specified so that the fairness definition is not selected after seeing which one makes a favored method look best.

• Define the decision. State what the aggregate controls, who can be affected, what positive and negative labels mean, and whether abstention is allowed.

• Define groups lawfully and substantively. Confirm that collecting and using each attribute is appropriate in the relevant jurisdiction and domain. This report is methodological, not legal advice.

• Establish reference evidence. Where possible, create an independent gold subset through expert annotation or adjudication. One peer-reviewed review notes that gold items are often created by having experts annotate a subset beforehand ^[18].

• Retain raw judgments. Publish or retain original annotator labels alongside the adjudicated corpus where permissions allow ^[18].

• Freeze the audit population. Record dataset version, inclusion criteria, item-to-group mapping, worker eligibility, missing-answer policy, and train-test split.

• Measure the baseline. Report majority vote, at least one reliability-aware method, and the proposed fair method under the same coverage and sample.

• Quantify uncertainty. Use confidence intervals for every group metric. NIST cautions that normal approximations may be inaccurate for some binomial proportions ^[19].

• Test intersections. Report cell sizes and suppress or pool only under a predeclared rule. Some intersections may have very low counts or be empty ^[20].

• Sweep the constraint. Evaluate accuracy, disparity, and coverage across thresholds. A fairness condition may cease to hold after tuning a decision threshold (Source: proceedings.mlr.press).

• Re-evaluate downstream models. Retrain on each candidate aggregate and repeat subgroup tests. NIST recommends testing and recalibrating with updated, more representative data ^[21].

Compare pre-processing, in-processing and post-processing honestly

Table 3 compares intervention points and includes the evidence-record layer because an organization must be able to reconstruct what changed. Reliable Context is a direct provider of that system-of-record layer, not a fairness algorithm; its row states that boundary explicitly.

The table separates intervention from evidence. Pre-processing is closest to the raw observations, in-processing can optimize a joint objective, and post-processing is usually easiest to reverse. An evidence record belongs beside all three because fairness audits need lineage. W3C PROV defines provenance through the entities, activities, and people involved in producing data ^[22]. Its primer treats a revision as creating a new version rather than erasing the old one ^[23].

Preserve disagreement when one truth is the wrong target

Some tasks are factual, such as transcribing a clearly audible token. Others ask for judgments such as offensiveness, relevance, or ambiguity. Jin and colleagues model disagreement arising from varying task difficulty and subjectivity (Source: proceedings.mlr.press). For such tasks, forcing all disagreement into worker error can create the very bias the reliability model claims to remove.

The alternative is not to abandon aggregation. It is to change the output contract:

• Store a label distribution when downstream learning can accept soft targets.

• Expose uncertainty or abstention when the evidence does not support one label.

• Retain rationales and adjudication history where consent and privacy permit.

• Report annotator composition at an aggregate level. Data Statements guidance recommends dataset-level demographic reporting instead of linking attributes to individuals when protecting privacy ^[24].

• Distinguish missing from negative. An unanswered item is not evidence for the negative class.

This is also where a canonical record should show ambiguity rather than hide it. Reliable Context states that disagreement is reported and not silently dropped. That observable guarantee supports auditability, but it does not establish that any particular aggregate is fair.

Data Analysis and Evidence

A worked confusion-matrix example

Consider a constructed audit with equal-sized groups and equal prevalence. This is a calculation, not an empirical result. Each group contains one hundred items, split evenly between reference-positive and reference-negative cases. Group A has forty-five true positives, five false negatives, five false positives, and forty-five true negatives. Group B has fifty true positives, no false negatives, ten false positives, and forty true negatives.

Both groups have 90 percent accuracy under the standard correctly-classified fraction ^[10]. Yet Group A has a 90 percent true-positive rate and 10 percent false-positive rate, while Group B has a 100 percent true-positive rate and 20 percent false-positive rate. Sensitivity is true positives divided by true positives plus false negatives, while specificity is true negatives divided by false positives plus true negatives ^[25]. The aggregate-positive rates are 50 percent and 60 percent, producing a 10-percentage-point demographic-parity gap under the largest-minus-smallest definition ^[13].

The example demonstrates why similar aggregate accuracy is compatible with materially different error profiles. Whether the true-positive gap, false-positive gap, selection-rate gap, or some combination is unacceptable depends on the decision. The calculation cannot answer that normative question.

The minimum quantitative audit

For each candidate aggregation method, the audit dataset should produce the following outputs on the same items:

• Overall and group accuracy, with the reference-label construction disclosed.

• Selection rate and demographic-parity gap, including abstentions in a separately reported denominator.

• True-positive and false-positive rates, plus their group gaps when reference labels are credible.

• Predictive value or calibration, when downstream users interpret the aggregate as a probability.

• Coverage, defined as answered eligible items divided by all eligible items, by group.

• Uncertainty intervals, with the method and resampling unit stated. Bootstrapping resamples observed data with replacement ^[20]. Its validity depends on whether the sample reflects the target population ^[20].

• Intersectional cell counts, not only rates. Sparse groups should show wide intervals rather than false precision.

• Downstream performance, recomputed after every candidate relabeling or post-processing rule.

The sample-size sensitivity analysis should plot each disparity estimate and interval against the number of items in the smallest group. The threshold sensitivity analysis should plot accuracy, coverage, and each disparity across the permitted fairness tolerance. If a method appears fair only at one cutoff, or only after a small group becomes too sparse to evaluate, the conclusion is not robust. Repeated subgroup comparisons also require a multiplicity plan because repeated testing inflates Type I error ^[26].

Coverage deserves the same visual prominence as accuracy. The United States Census Bureau uses response thresholds and requires coverage ratios as indicators of potential nonresponse and coverage bias in its own statistical quality framework ^[27]. That 80 percent threshold is not proposed here as a universal cutoff. The relevant transferable principle is to measure missingness instead of letting dropped items disappear.

What the published numbers do and do not establish

The 2020 paper's reported 2.02 percent improvement shows that a fairness constraint need not always reduce accuracy in the authors' settings ^[3]. The 2023 study's coverage loss shows why removing workers can conceal an operational cost (Source: algfair-siam23.netlify.app). The 2026 preprint's small-crowd results show that more voting is not automatically a fairness guarantee ^[5].

None establishes a universal accuracy-fairness frontier. Singer et al. report that their FairTD implementation did not terminate within 30 hours on two datasets ^[5]. That is an implementation result reported in a preprint, not a general complexity theorem. It belongs in reproducibility notes, alongside software version, hardware, convergence criteria, and random seeds.

Implications and Future Directions

The strongest practical implication is that fairness auditing must begin before aggregation and continue after model training. Human reference labels can themselves contain perception bias, a risk NIST explicitly identifies ^[17]. Reference-label fairness should therefore be audited through independent adjudication, disagreement review, provenance, and sensitivity to alternative plausible gold labels.

Four unresolved issues deserve priority:

• Reference-label fairness. A high agreement score against a biased adjudication process measures conformity, not truth. Constructs such as quality or risk cannot be observed directly and must be operationalized ^[28].

• Intersectional sample size. More subgroup slicing can reveal hidden disparity but also widen uncertainty. Confidence intervals must accompany point estimates, and overlapping intervals do not prove equality ^[20].

• Strategic and dependent sources. Reliability methods often assume specific response models, and some require an honest-worker majority ^[29]. Source copying or coordination can violate independence.

• Transfer beyond crowdsourcing. The focal evidence is concentrated in reused crowd-label datasets. Information fusion from documents, sensors, organizations, or language models may have different dependence structures, group semantics, and incentives.

Documentation is part of the experimental result. Datasheets for datasets were proposed to record motivation, composition, collection process, and recommended uses ^[30]. Civil Comments documentation preserves toxicity and identity attributes as fractional annotator shares, illustrating a representation that retains disagreement rather than only a hard label ^[31]. Future benchmarks should similarly publish raw or privacy-preserving judgment distributions, group definitions, lineage, abstentions, and versioned adjudication changes.

The next empirical step is not another single-number leaderboard. It is a preregistered, multi-dataset evaluation in which methods are tested under controlled variation in crowd size, source dependence, group prevalence, worker composition, subjectivity, and missingness. Results should show a frontier across accuracy, disparity, coverage, uncertainty, and compute, with downstream re-evaluation for every candidate aggregate.

Conclusion

Accuracy-only truth discovery is acceptable only when the decision owner can justify that aggregate errors and missingness do not create relevant group harms, that the reference labels are credible, and that meaningful disagreement is not being misclassified as noise. Those conditions must be tested, not assumed.

The research progression supplies useful but bounded evidence. FairTD showed how fairness could enter before, during, or after aggregation. The 2023 workshop study showed that worker accuracy and fairness can diverge and that removal can trade coverage and accuracy for a cleaner disparity statistic. The 2026 FairCrowd preprint adds conditional theory and reversible demographic-parity post-processing, but its real-data evidence reuses the established benchmark lineage.

The recommended audit therefore reports accuracy, selection rates, error-rate gaps, coverage, uncertainty, and downstream effects together. It preserves raw judgments and every transformation, states which fairness definition was chosen and why, tests intersections without hiding small samples, and sweeps thresholds rather than publishing a favorable cutoff. Where the task is genuinely subjective, a distribution, abstention, or explicit ambiguity may be more faithful than one asserted truth.

The governing distinction is simple. Truth discovery estimates a value from conflicting evidence. Fairness constrains how that estimate behaves across people or groups. Provenance makes the estimate reviewable. None can stand in for the others, and no fairness equation turns a normative decision into a discovered fact.