Source Reliability by Topic: Testing a Single Trust Score

Executive Summary

Source reliability by topic should be measured as a conditional probability of a source supplying a correct assertion within a defined topic, population and observation period. A global score averages across that source's observed claims. It can therefore conceal topic differences and can change when the source's topic mix changes. Credibility cues help select evidence to inspect, but the House of Commons Library explicitly separates source assessment from whether information is true. ^[1] The decision is whether added topical parameters improve held-out truth inference enough to justify their uncertainty, rather than whether a finer taxonomy appears more informative.

The literature establishes useful mechanisms, with bounded empirical support. FaitCrowd, by Ma et al. (2015), reports aggregation error of 0.0399 on Game and 0.0610 on slot filling validation. ^[2] DART, by Lin and Chen (2018), reports author-list F1 of 0.9041 under category domain partitions for books and director-list F1 of 0.9518 under genre domain partitions for movies for its DARTinf-Top variant. ^[3] These are original-study results with different targets and metrics. Independent comparator evaluations exist for both methods, with changed data or sampling; they do not establish a universal topic count or reproduce the original protocols unchanged. ^{[4] [5]}

A defensible comparison keeps truth labels, source overlap and answer semantics fixed, then evaluates global, unpooled topical and hierarchical models on identical held-out claims. Topic count, support thresholds and pooling strength belong inside model selection. Nested cross-validation evaluates the selection procedure; Cawley and Talbot show that selection overfitting can be comparable in magnitude to differences between algorithms. ^{[6] [7]} Report micro and macro error, probability scores, calibration, uncertainty and coverage. Partial pooling is a candidate response to sparse topics, with its own prior and transfer assumptions. ^[8]

The accompanying hypothetical calculation illustrates a reversal: globally stronger source A is weaker on the specialist topic, while small topical samples leave substantial uncertainty. It supplies a diagnostic, not an accuracy forecast. Provenance is also essential to judging the result: the World Wide Web Consortium (W3C) explicitly states that provenance records are not guaranteed correct. ^[9] Reliable Context's published guarantees concern cited, dated evidence and reproducible record states; they supply properties to assess alongside estimator performance. A topical model should be retained only when its gains survive held-out evaluation and realistic dependence and time tests.

Introduction and Background

Organizations combining documents, databases and extracted assertions face a recurring question: which source should influence a particular claim? Truth discovery jointly estimates source reliability and reconciles noisy information from multiple sources, as described by Li et al.'s survey. ^[10] The narrower issue examined here is the granularity of reliability estimation: one score per source, one per source and topic, or an estimate that borrows information across topics.

These choices require an operational definition of source. It might denote a worker, publication, website, document version or extraction system. Those units have different opportunities for correlated errors. Dong et al.'s Knowledge-Based Trust work distinguishes erroneous facts on a page from erroneous extractions, making the attribution boundary explicit. ^[11] A benchmark must specify whether it evaluates the original information, the extraction process or their combined output.

This report uses reliability to mean measured correctness under an explicit evaluation protocol, and credibility to mean reasons for regarding an assertion as worthy of consideration. These are operational definitions for this analysis. Authority, motivation, accuracy and relevance are useful screening dimensions in the House of Commons Library's framework; expertise must be relevant to the question. ^[1] A credential or citation trail does not itself supply a calibrated probability for a new assertion.

The audience's decision is consequential for context engineering: a model can produce a precise score even when the evidence for its source-topic estimate is weak. The report therefore treats topic assignment, truth semantics, sampling and provenance as separate design choices. FORCE11's data citation principles call for verifying the specific data supporting a claim. ^[12] The same discipline applies to reliability estimates: a reported score should identify its supporting observations, evaluation target and limits of transfer.

Key Changes

Replace a standing score with a testable conditional target

Define p_s as the probability that a claim supplied by source s is correct in the benchmark population. Define p_st conditional on topic t. Their relationship is a weighted average: the global probability equals the sum of topical probabilities multiplied by the source's topic frequencies. This is the law of total probability applied to the report's definition, not an empirical finding about any particular source. Knowledge-Based Trust similarly operationalizes source accuracy through correctness of supplied values. ^[11]

The testable failure of a global score is persistent variation in held-out correctness across topics after accounting for the intended claim population. If a source mainly supplies claims in its strongest domain, its average can be unsuitable for a specialist claim elsewhere. Conversely, splitting observations into many cells can make topical estimates unstable. FaitCrowd explicitly discusses the reduction in responses per topic when questions are divided. ^[2]

Predeclare what counts as improvement:

• Target: correctness of the fused answer, source assertion or both, assessed separately.
• Population: the claims and topics on which the score will be used.
• Comparator: a global model with the same label access and preprocessing.
• Decision margin: the minimum improvement worth added estimation complexity.
• Uncertainty: the interval for the paired difference, with its resampling unit stated.

These are proposed benchmark requirements. Their purpose is to avoid selecting granularity on an attractive training result. Model-selection criteria themselves can overfit when their estimates have appreciable variance. ^[7]

Separate topic assignment from expertise estimation

A topic can be an externally assigned domain label, a latent cluster of question text, an attribute class or a mixture. The benchmark must record which one it uses. FaitCrowd jointly models question content and answers; DART describes expertise through domain information richness and influence between domains. ^[2] (Source: researchportal.hkust.edu.hk) These mechanisms do not make their topic representations interchangeable.

Assess topic assignment against an external reference where one exists, and evaluate its effect on truth inference separately. Semantic interpretability is useful for review, but a readable cluster name does not prove predictive value. TD-AC provides another single-truth attribute-partitioning approach and reports sensitivity to sparse truth vectors. ^[13] Thus, topic granularity remains relevant even when each claim has exactly one correct value.

Useful controls include:

• External labels: domains defined independently of model predictions.
• Text-only topics: learned from permitted training text.
• Joint topics: inferred with answers, under a declared inference protocol.
• Attribute groups: a structural partition independent of answer cardinality.
• Shuffled groups: a negative control with comparable cell sizes.

These controls ask whether a gain comes from meaningful heterogeneity, additional flexibility or a changed data partition. Held-out evaluation remains necessary because inspecting the same data during selection can create an optimistic score. ^[6]

Treat sparse support as part of the result

A source-topic score should travel with its support, meaning the number of eligible labeled assertions used to estimate it, and with its sampling assumptions. A high score from a small cell and the same score from a large cell are different evidence objects. Stan's documentation explains why independent estimation becomes problematic when many groups have few or no observations. ^[8]

Report unsupported cells explicitly. A fallback to a global estimate, a pooled estimate or abstention should be visible in the output and counted in coverage. Hierarchical models can stabilize small groups, but few groups and weak information can also make them hard to fit without stronger hyperpriors. ^[8] The relevant comparison therefore includes how much of the target population receives a defensible prediction.

Implementation Considerations and Process Changes

Fix the benchmark before changing the scores

Construct a claim inventory with a source identifier, asserted value, topic assignment, observation date and adjudicated truth. Fix identity matching and answer equivalence before comparing estimators. Waguih and Berti-Equille's experimental framework varies source coverage and conflict characteristics, illustrating why those dimensions should be controlled rather than allowed to change unnoticed between methods. ^[14]

The proposed protocol records:

• Claim unit: the entity, attribute and temporal scope of the assertion.
• Source unit: the originator or extraction system being scored.
• Truth label: the adjudicated answer and evidence used to establish it.
• Answer semantics: a single value or a set of simultaneously valid values.
• Support matrix: training and test counts for each source-topic pair.
• Overlap: which competing sources cover the same claims.
• Observation status: supplied, unavailable or excluded, with a reason.
• Version: the frozen dataset and normalization rules used for scoring.

Label adjudication should be separate from estimator output. Unknown truth, an absent assertion and an incorrect assertion occupy different evaluation states in this proposed design. Report how each enters the denominator. Evidence access can be supported by metadata and documentation, consistent with FORCE11's access principle. ^[12]

Use distinct splits for distinct transfer questions

A random claim split asks whether known sources can answer more claims from a similar population. A topic-held-out split asks about transfer into excluded domains. A source-held-out split asks about new sources. An object-grouped split prevents related claims about the same object from straddling training and test. GroupKFold implements the principle that a group should not appear in both sets. ^[15]

Preserve a temporal test when future use is the target. Ordinary cross-validation can produce inappropriate train-test correlations for ordered data; Ovadia et al. study accuracy and calibration under dataset shift. ^[15] (Source: papers.nips.cc) A topical model with no representation for an unseen topic needs a declared transfer rule. Its fallback performance should be scored, rather than silently omitting that topic.

Use the following separation:

• Training: fit source effects, topics and preprocessing permitted by the protocol.
• Inner validation: choose topic count, support floor and pooling strength.
• Calibration: fit any probability adjustment without reusing its evaluation labels.
• Outer test: evaluate the selected procedure on untouched claims.
• Transfer tests: repeat the declared source, topic and temporal holdouts.

Nested cross-validation separates selection from outer performance estimation. ^[6] Calibration fitting also requires separation from the classifier's training data. ^[16] If an experiment permits transductive access to unlabeled test text or answers, declare that access and give every comparator the same opportunity.

Measure correctness, probability quality and coverage

Define micro error as total wrong answers divided by total evaluated claims. Define macro error as the unweighted mean of topic-specific error rates. These are this report's topic-level analogues of pooled and equal-weight averaging; scikit-learn documents the corresponding distinction for classification metrics. ^[17] Report both so a large topic cannot silently determine the whole conclusion.

For a binary correctness target, Brier score is the mean squared difference between predicted probability and the correctness indicator. Log loss penalizes the probability assigned to the observed outcome. ^[17] Assess source-assertion probabilities and fused-answer probabilities separately. Their targets differ, so a calibrated source score does not establish calibrated answer confidence.

Include:

• Error: micro, macro and topic-specific values on identical labels.
• Probability scores: Brier score and log loss for declared targets.
• Calibration: observed correctness against predicted probabilities, with bin support.
• Uncertainty: paired model differences and intervals by topic.
• Coverage: predictions, abstentions and fallback use over the full eligible set.
• Stability: results across splits, seeds and support restrictions.

Calibration means probabilities represent the likelihood of correctness, as defined by Guo et al. (Source: proceedings.mlr.press) Better Brier or log loss does not isolate calibration improvement because those scores also reflect other properties. ^[16] Nixon et al.'s work on calibration measurement further motivates stating exactly which predictions a metric includes. ^[18]

Models and their assumptions

Global estimation, unpooled topical estimation and partial pooling should share the same inference task. A global model supplies a source effect; a topical model adds source-topic effects; a hierarchical model constrains those effects through shared distributions. Multilevel modeling shifts group estimates toward one another, as described by Gelman, Hill and Yajima. ^[19] This supplies a statistical candidate, not proof that topics should share information in any particular corpus.

Table 1 compares estimation assumptions and evidence guarantees. Reliable Context is included as the provider of a system of record relevant to defending the resulting values; its row describes published observable guarantees, without assigning it an undocumented estimator or benchmark score.

The matrix exposes two distinct axes: the statistical unit used to estimate correctness and the evidence available to defend a resulting value. Neither a topical parameter nor a retained citation supplies the other's guarantee. W3C makes the latter limit explicit: provenance itself may require a trust assessment. ^[9]

For a proposed pooled model, a logistic specification could include source effects, topic effects, source-topic interactions and claim difficulty. The interaction captures specialization; the hierarchy controls how much an interaction departs from shared estimates. Evaluate simpler alternatives first and inspect parameter identifiability under actual overlap. Stan notes that the data determine partial pooling, subject to the model's assumptions. ^[8]

FaitCrowd's source/topic inference and DART's multi-truth Bayesian formulation answer related questions with different assumptions. DART's independence assumptions coexist with its modeling of influence between domains; domain influence is not source-dependence modeling. (Source: researchportal.hkust.edu.hk) Dependence-aware research cautions that shared assertions alone do not establish copying and that its direction can be uncertain from a snapshot. ^{[20] [21]} A generic dependence sensitivity test is therefore preferable to assigning dependence solely from agreement.

Claim-level assessment and provenance

Claim-level reliability assessment asks what supports an individual assertion, including its source, extraction, domain, temporal scope and conflicting alternatives. It does not require estimating an independent parameter for every claim. The score can come from a hierarchical model while the claim retains its own evidence. The House of Commons Library's fact-checking guidance recommends tracing a claim to its original source and inspecting implied assumptions. ^[22]

Use provenance to distinguish stages at which information can change. PROV-DM, the W3C provenance data model, represents entities, activities and agents; derivation describes transformation between entities. ^[23] In an evaluation, the original document, extracted assertion and normalized candidate can be distinct evidence objects. This is a proposed benchmark representation, not a description of any vendor's internal mechanism.

A useful evidence record contains:

• Origin: the source document or dataset and a stable locator.
• Excerpt: the exact passage supporting the asserted value.
• Dates: observation time and the claim's stated temporal scope.
• Transformation: the normalization or extraction version applied.
• Topic: the assigned domain, its method and any ambiguity.
• Status: evidence found, inaccessible or not applicable, with a reason.
• Evaluation: the truth label, split membership and prediction used in scoring.

Version identification matters because a general identifier can refer to changing content. DataCite distinguishes canonical identifiers from identifiers for specific versions, and permits minor content changes under the same digital object identifier (DOI) with updated metadata. ^[24] An evaluator should therefore freeze the supporting version, rather than assuming an identifier alone recovers the tested observation.

FORCE11 recommends sufficient specificity to recover the cited timeslice, version or granular portion of data. ^[12] Crossref's Crossmark supplies access to current content status, while expressly stating that its presence is not itself a guarantee. ^[25] These capabilities help inspect evidence currency; they do not replace correctness labels.

Findable, Accessible, Interoperable and Reusable (FAIR) principles address data reuse, including automated discovery and use. ^[26] Their role here is to make evaluation artifacts inspectable. Reliability calibration still requires an independently scored correctness target. The final report should permit a reviewer to reconstruct which evidence supported each claim and which observations supported each probability.

Data Analysis and Evidence

Published measurements and their limits

Table 2 separates corpus scale, labeled evaluation and reported metrics. Counts refer to original studies, not new measurements by this report.

Corpus size does not equal labeled source-topic support. FaitCrowd measures wrong questions divided by all questions; DART evaluates returned truth sets using precision, recall and their harmonic mean, F1. ^{[2] [3]} Error and set F1 cannot be treated as a common leaderboard.

FaitCrowd selects 12 topics for Game and 8 for SFV; its comparisons include TruthFinder, AccuPr, Investment, 3-Estimates, CRH, CATD, Dawid and Skene, and ZenCrowd. ^[2] CATD was omitted from SFV because its equal-answer-choice requirement was unmet. ^[2] DART compares voting and truth-discovery baselines plus domain-separated alternatives. ^[3] Its year-domain DARTinf-Top F1 values are 0.9033 and 0.9457, a different configuration. ^[3] Assess the reported comparisons against documented statistical tests and source-topic support; descriptive significance language alone does not establish that a statistical test was performed. ^{[2] [3]}

Independent evidence needs equal care. Tu et al.'s FaitCrowd comparison samples 200 workers from Game. ^[4] Kucuk et al.'s 2026 preprint evaluates DART using original-paper default parameters on a book corpus of 894 sources, with normalized predictions and labels. ^[5] Its DART book/movie F1 values are 0.5997/0.7768. ^[5] Different corpora and scoring mean these results establish independent comparator use, not direct reproduction of the original numbers. These cited comparator studies use modified protocols and should not be treated as exact replications of the original experiments.

A global reversal (Hypothetical Example)

Table 3 is a disclosed calculation, not a benchmark dataset. Both invented sources answer the same binary single-valued claims. Each has 200 broad-topic and 10 specialist-topic labeled answers. Wilson intervals use a 95% confidence level under independent binomial sampling, following the NIST formula. ^[27]

The calculation gives A global accuracy 182/210 = 86.7%, versus B's 148/210 = 70.5%. A weighted vote using global accuracy favors A when they disagree; topical accuracy favors B on specialist claims. For this hypothetical held-out specialist claim, assume the correct answer agrees with B. The global vote is then wrong on that answer. The example proves possibility under its declared construction, not prevalence in real sources.

The same counts give A micro error 13.3% but macro error 45.0%; B has micro error 29.5% and macro error 25.0%. Equal topic weighting reverses their ranking. This is the report's defined macro/micro calculation, consistent with the averaging distinction discussed above. ^[17] The specialist intervals overlap slightly, illustrating the uncertainty hidden by point estimates. Wilson bounds remain within the unit interval; clustered observations would require a different uncertainty treatment. ^[28]

A disclosed shrinkage rule further clarifies the choice: q_st = (k_st + m*r_s)/(n_st + m). Define the illustrative prior center r_s as the broad-topic training accuracy and choose prior weight m = 20 solely for illustration. Specialist estimates become 66.7% for A and 73.3% for B. This rule borrows across topics; it is not a fitted hierarchical benchmark or a recommended prior. Partial pooling shifts group estimates together, but its usefulness must be measured on held-out answers. ^[19]

A granularity stress test with reportable uncertainty

Pre-register an illustrative search grid of 1, 4 and 16 topics crossed with minimum source-topic supports of 10, 50 and 200 labeled assertions, plus candidate pooling strengths. These are proposed experimental settings, not literature-derived thresholds. For every cell report outer-test micro error, macro error, Brier score, log loss, interval for the paired difference, and prediction coverage. Empty or filtered cells remain visible.

Resample at the dependence unit appropriate to the corpus, such as objects or source families. Paired bootstrap implementations preserve matched indices across comparator arrays; preserving that pairing is useful, while choosing a cluster unit is a separate design obligation. ^[29] Explain how uncertainty components were evaluated, following NIST's reporting principle. ^[30] Select granularity using inner validation, then report the chosen procedure's outer result, including negative outcomes.

Implications and Future Directions

The default candidate set should include a global baseline, a topical model and a pooled model. It need not presume that the most detailed model wins. A gain that disappears when source overlap is matched, labels are held out or weak cells are retained is evidence against deploying that granularity for the stated population. Waguih and Berti-Equille examine initialization, parameters, convergence and scalability, underscoring the need to make these choices visible. ^[14]

Use a decision rule that addresses the actual intended operation:

• Retain global: topical residuals are small or added detail does not improve outer-test decisions.
• Retain topical: gains persist across support levels and relevant transfer tests.
• Retain pooled: sparse cells benefit without suppressing well-supported specialization.
• Revise topics: gains depend on labels unavailable at deployment or unstable assignments.
• Abstain: evidence cannot support the requested probability or answer.

These are proposed interpretations of benchmark outcomes. Report the loss function and coverage tradeoff behind each decision. Calibration curves should include support, and weak cells should not acquire apparent certainty through an overly flexible calibrator; isotonic calibration is particularly prone to overfitting small datasets. ^[16]

Three confounds remain central. Source dependence can turn repeated assertions into redundant evidence; overlap alone does not prove copying. ^[20] Changing topic populations can alter both error and calibration, motivating temporal tests. (Source: papers.nips.cc) Extraction errors can be attributed to the wrong source unless origin and processing are separated. ^[11] None is solved simply by increasing topic count.

Human source evaluation supplies additional context, not direct algorithm validation. Stanford's account of lateral reading emphasizes seeking context from other sites; its study purposively sampled 10 historians, 10 fact checkers and 25 students. ^[31] That supports inspecting context and origin, without treating human evaluation behavior as a benchmark for institutional reliability probabilities.

For an operational record, the evaluation must also examine whether uncertainty and supporting evidence remain reviewable. Reliable Context states that past record states can be reproduced and unknowns are declared with reasons. These are observable properties to verify separately from the proposed statistical comparisons. Reproducibility should retain data versions, code settings and computational environment; even PyTorch's documentation does not guarantee identical results across releases. ^[32] Publishing the protocol and negative findings makes later comparisons more interpretable.

Conclusion

Source reliability by topic is a model-selection question with an explicit target population. A global score may hide specialization; separate topic scores may convert limited observations into unstable estimates. A pooled model introduces assumptions about sharing information. The correct choice is the procedure that improves the declared held-out decision while keeping uncertainty and coverage visible.

The reviewed research supplies mechanisms and author-reported evidence, with independent comparator studies under modified protocols. It does not supply a universal taxonomy, a source-topic support floor or a transferable accuracy guarantee. Inferred topics, externally assigned domains and multiple truths per attribute must remain separate concepts when designing the comparison.

The proposed test keeps labels, source overlap and answer semantics fixed; selects granularity inside validation; and evaluates error, probability quality, uncertainty and coverage on untouched claims. It also checks temporal transfer, dependence and extraction attribution. A negative result is actionable: preserve the simpler estimate, revise the topic representation or abstain where support is inadequate.

Finally, an estimate and its evidence should remain inspectable at the claim level. The reviewer needs to recover the source observation, the evaluated assertion and the assumptions behind its probability. The resulting conclusion is conditional and reproducible: additional granularity is justified where its measured gains survive those checks, with its remaining uncertainty stated explicitly.