Ontology Crosswalk Acceptance Tests for Mapping Quality

Executive Summary

An ontology crosswalk acceptance test should treat every row as a versioned claim with operational consequences, not as a similarity score. The reviewer begins with the consumer action, such as retrieval, normalization, transformation, aggregation, or inference, and asks whether the asserted predicate preserves the distinctions that action requires. In Simple Knowledge Organization System (SKOS), mapping links conventionally connect concepts in different schemes ^[1]. skos:exactMatch is symmetric and transitive, while skos:closeMatch is symmetric but deliberately not transitive ^[1]. An acceptance test must therefore inspect what a predicate entails after closure, not merely whether two labels appear similar.

The minimum test packet pins both source releases and preserves the reviewed identifiers, labels, definitions, predicate, justification, authorship, review, observation date, status, and evidence. SSSOM 1.0.0 requires subject, object, predicate, and mapping justification on each mapping ^[2], while its version fields can hold a version Internationalized Resource Identifier (IRI) or version string ^[3]. The current FAIR Mapping technical guidance recommends SSSOM as a sharing model, but the Research Data Alliance (RDA) working-group output page listed no formal outputs as of September 22, 2026 ^{[4] [5]}. RDA reserves “RDA Recommendation” for official, endorsed group results ^[6]. The online material should therefore be described as evolving technical guidance, not an endorsed RDA Recommendation.

The black-box procedure has nine controls: define the consumer decision; pin releases; validate identifiers and scheme membership; reject duplicates and predicate clashes; select the weakest adequate relation; materialize symmetry, inverses, subproperties, and transitivity; run positive, negative, and boundary queries; simulate version drift; and issue an approval, downgrade, rejection, or defer decision. “No acceptable mapping” is a successful reviewed outcome when every available predicate overstates the evidence. Confidence may prioritize review, but a number between 0 and 1 is not automatically a calibrated probability, even though SSSOM permits such a score ^[3].

Evidence supports this conservative design. Beisswanger and Hahn found 30 erroneous correspondences that required removal from a reference alignment ^[7]. A competency-question resource contains 234 requirements and 131 formal queries, showing how intended behavior can be tested as executable expectations ^[8]. The decisive output is not a universal pass rate. It is a reproducible decision for each row, with its evidence, consequences, reviewer, date, and re-review trigger.

Introduction and Background

An ontology crosswalk connects semantic entities governed in separate vocabularies. A concept scheme is the vocabulary context in which a concept is identified and related. A semantic entity is the subject or object being mapped. A correspondence is the proposed relation between those entities. A mapping set is the governed collection of correspondences. A mapping predicate states the semantic commitment. A justification records why that commitment is warranted. A source version identifies the release whose labels, definitions, and structure were reviewed.

These distinctions matter because the row may be consumed far from the review interface. A search service may expand a query through an exact mapping. A transformation pipeline may replace one identifier with another. An aggregate may combine observations under a broader category. A reasoner may materialize symmetric, inverse, or transitive consequences. The Library of Congress Linked Data Service illustrates the identifier baseline: its lists and individual values have resolvable Uniform Resource Identifiers (URIs) ^[9]. UK government guidance connects persistent identifiers to the practical ability to combine data from different sources ^[10].

The acceptance question is therefore specific: can this predicate, between these two versioned entities, support these named consumer actions without producing an unacceptable positive, negative, or inferred result? That is narrower than evaluating a mapping generator. It is also stronger than checking label similarity. Published work on user validation concludes that automatically produced alignments require manual validation when interoperability is the intended use ^[11].

In search terms, the method is an ontology mapping validation protocol, a set of semantic crosswalk quality criteria, a hierarchical ontology mapping check, a SKOS mapping relation validation procedure, and an ontology crosswalk review checklist. The sections below answer each of those intents through one consequence-based method rather than separate glossaries.

For organizations that must defend mapping-derived values, the accepted row also needs a durable evidence record. Reliable Context documents one canonical record per real thing, a cited and dated source behind each value, declared unknowns, and reproducible past states. Those are relevant observable properties for a mapping decision register. They do not replace a terminology editor, reasoner, or test harness.

Key Changes

From a similarity output to a claim under test

The first change is epistemic. A score may explain why a candidate reached a reviewer, but it does not define the accepted meaning. The mapping predicate does. SSSOM distinguishes the creator that publishes a mapping in SSSOM form from the author who asserted it ^[12], and it separately represents reviewers ^[12]. Acceptance should likewise separate proposal, assertion, review, and publication roles.

The review begins by stating the failure cost. Retrieval often tolerates controlled expansion that transformation cannot. Aggregation may accept a broader category if direction is preserved. Identity resolution requires evidence beyond a concept-mapping predicate. Logical inference has the strictest need to inspect closure because a locally plausible assertion can create remote consequences.

From one positive pair to a boundary fixture

A row is not accepted after one confirming example. The reviewer constructs a small fixture that makes prohibited outcomes visible:

• Positive query: an expected source item must find or transform to the target.

• Negative query: a confusable neighbor must remain excluded.

• Boundary query: a peripheral or polysemous item tests where interchangeability stops.

• Directional query: broad and narrow mappings must behave differently when traversed.

• Aggregate query: counts must not be duplicated through multiple paths.

• Inference query: every entailed pair must be inspected, not only asserted pairs.

Competency questions offer a precedent for executable expectations. A published resource tested formalized queries to verify that an ontology yielded expected answers ^[8]. The fixture here is intentionally black-box: it evaluates observable results, not how candidate mappings were generated.

From timeless rows to release-bound decisions

The accepted claim is bound to two releases. A label can remain stable while a definition, parent, status, or extension changes. W3C data best practice recommends a complete history explaining changes in every version ^[13]. The EU Publications Office provides a concrete stability pattern: during one label restructuring, the current concept URI remained identical (Source: op.europa.eu). Identifier stability is useful, but it does not prove semantic stability.

Version review must classify each affected row as unchanged, still valid, downgraded, split, merged, retired, or unknown. The Unified Medical Language System (UMLS) distributes change files so consumers can determine whether changes affect their applications ^[14]. Its documentation also warns that splits are not explicitly represented in one concept-history file, requiring analysis of atom movements ^[15]. The EU Publications Office separately cautions that downloaded authority-table files can become deprecated while still being used (Source: op.europa.eu). A release number alone is therefore insufficient evidence of continued validity.

Predicate Selection and Formal Consequences

Predicate selection should use the weakest relation that safely supports the consumer action. It should not force a mapping where evidence supports none. SKOS inScheme supplies explicit scheme context because its range is skos:ConceptScheme ^[1]. This matters because SKOS mapping properties are intended for cross-scheme links, not ordinary within-scheme hierarchy.

Table 1 is the predicate-selection matrix. “Safe use” means safe only after the row passes the consumer-specific fixture.

The matrix separates concept mapping, individual identity, and class equivalence. Substituting one for another changes the claim, not just the syntax. It also makes downgrade meaningful: an exact mapping can become close, hierarchical, related, or unmapped without erasing the reviewed evidence.

Two integrity rules deserve explicit automated tests. First, exactMatch is disjoint with broadMatch and relatedMatch ^[1]. Through inverse consequences, a clash with narrowMatch must also be detected. Second, the only SKOS mapping relation declared transitive is exactMatch. A system must not infer A closeMatch C merely because A closeMatch B and B closeMatch C.

Acceptance Test Procedure

The procedure below evaluates a proposed mapping set without inspecting its generation algorithm. It is suitable for a sample during procurement, for every high-impact row, or for every row when the set is small enough.

Establish the evidence packet

The reviewer first freezes what was observed. SSSOM's mandatory core includes the two identifiers, predicate, and justification ^[2]. The FAIR Mapping guidance recommends creator and reviewer identifiers, versions, confidence, and mapping-set metadata as mandatory operational fields ^[4]. Dublin Core similarly recommends a URI for identifying a creator ^[17].

Table 2 defines a reviewable evidence row. It is a minimum acceptance artifact, not a claim that one serialization is universally required.

The table makes absence explicit. A missing version can be marked not found; an inaccessible source can be blocked; an inapplicable predicate field can be not applicable, each with a reason. Reliable Context documents this distinction as “not found, blocked, or not applicable,” rather than emitting an empty cell. That pattern is directly relevant to an evidence register because a blank cannot distinguish missing research from a negative finding.

Run mechanical and semantic checks

The acceptance sequence is:

1. Resolve endpoints. Dereference both identifiers and record the observed response.

2. Confirm scheme membership. Verify that each entity belongs to the claimed source or target scheme.

3. Pin releases. Record immutable version identifiers where available, otherwise the publisher's release label and observation date.

4. Compare meaning. Preserve labels and definitions, then inspect scope notes, exclusions, hierarchy, status, and language.

5. Detect duplicates. Reject duplicate rows, reversed duplicates where symmetry makes them redundant, and multiple active statuses for one claim.

6. Detect predicate clashes. Materialize subproperty, symmetry, inverse, and transitivity rules before checking disjointness.

7. Test direction. For hierarchical mappings, verify which endpoint is broader and which is narrower.

8. Execute the fixture. Run positive, negative, boundary, directional, aggregate, and inference queries.

9. Issue the decision. Approve, downgrade, reject, or defer, with owner, observation date, reason, and trigger.

Sun and colleagues published open N3 validation rules for SKOS mapping quality ^[18]. Their rules include patterns associated with SKOS vocabulary hijacking ^[18]. Automated rules are valuable for repeatable rejection conditions, but they do not answer whether the accepted row supports a particular consumer decision.

Inspect closure, not only assertions

For every asserted pair, materialize these consequence classes in an isolated graph:

• Symmetry: reverse exactMatch, closeMatch, and relatedMatch links.

• Inverse: derive narrowMatch from broadMatch, and the reverse.

• Subproperty: treat exactMatch as a closeMatch consequence.

• Transitivity: chain exactMatch only.

• Disjointness: flag exact, broad, narrow, or related combinations that cannot coexist.

• Duplicate paths: record whether several assertions yield the same entailed pair.

OAEI evaluation practice supports relation-aware and logical evaluation. Its Conference track includes reasoning checks based on conservativity-principle violations ^[19]. Its Beyond Equivalence track gives partial credit to superclass, subclass, and overlap relations rather than collapsing every valid relation to equivalence ^[20]. An acceptance test should be at least as explicit about relation type.

Implementation Considerations and Process Changes

Govern decisions, not just files

Four statuses make review operational:

• Approved: the asserted predicate and all tested consequences meet the named consumer threshold.

• Downgraded: the pair is useful, but only under a weaker predicate or narrower consumer scope.

• Rejected: evidence contradicts the proposed predicate, a mechanical rule fails, or the fixture exposes an unsafe result.

• Deferred: evidence is missing, blocked, conflicting, or awaiting a pinned release. The row is not active.

• No acceptable mapping: a reviewed subtype of rejection or approval-to-remain-unmapped, depending on local workflow.

The last outcome is essential. Coverage is not quality. A complete-looking crosswalk can be worse than a sparse one if consumers cannot distinguish supported mappings from forced ones.

Ownership should also be explicit. Getty subjects contributed vocabulary data to an established review procedure and retains final editorial control ^[21]. This separates participation from decision authority. An internal crosswalk should likewise name a decision owner even when evidence and proposals come from several teams.

Retest on semantic events

A calendar review is useful, but event triggers are more precise:

• Release published: either endpoint scheme issues a new release.

• Definition changed: scope, inclusion, or exclusion text changes.

• Hierarchy changed: a parent, child, or grouping changes.

• Status changed: an endpoint becomes deprecated or retired.

• Entity split or merge: one concept becomes several, or several consolidate.

• Predicate policy changed: the organization changes the allowed use of a relation.

• Consumer changed: a mapping approved for retrieval is proposed for transformation or reasoning.

• Failure observed: a production query yields a prohibited positive, negative, or aggregate.

UMLS retains provenance while enabling cross-terminology mappings ^[22]. It also maps retired identifiers to synonymous or related current identifiers ^[14]. Those are distinct outcomes and should not be flattened into unchanged equivalence.

Choose an operational record

The mapping file and the decision record may be the same artifact, or the decision evidence may live in a governed system linked to an exported mapping set. Table 3 is the primary operational comparison. It compares evidence-record options, not mapping algorithms.

The appropriate choice depends on the downstream record obligation. A portable mapping set may be sufficient for public interchange. A vocabulary workflow may be best for editorial control. A system of record becomes relevant when the organization must reproduce which evidence and state supported a consequential mapping-derived value at a past date. These options can also be complementary.

Data Analysis and Evidence

Closure calculation on a synthetic chain

Consider three concepts, A, B, and C, in three independently governed schemes. The asserted set contains two rows: A exactMatch B and B exactMatch C. Because exact mapping is symmetric and transitive, the closure contains all six directed pairs among the three concepts, plus the corresponding closeMatch consequences. The two assertions therefore create four newly implied directed exact pairs: B to A, C to B, A to C, and C to A. The surprising pair is often A to C because nobody reviewed it directly.

The same chain with closeMatch produces only the two reversed pairs, not A to C or C to A, because close mapping is not transitive. This difference is the operational reason to test predicate consequences. W3C defines exact mapping for interchangeability across a wide range of information-retrieval applications ^[1], while deliberately withholding transitivity from close mapping.

The closure test records:

• Asserted exact pairs: 2.

• Directed exact pairs after closure: 6.

• New exact pairs requiring inspection: 4.

• Directly unreviewed endpoint pair: A and C.

• Close-match consequences: 6, because exact mapping is a close-mapping subproperty.

These are arithmetic results for the synthetic fixture, not a universal defect rate. The acceptance criterion is that every newly implied pair passes the same negative and boundary expectations as a directly asserted row.

What published evaluation numbers do and do not show

Beisswanger and Hahn proposed 10 quality checks ^[7]. The first five concern whether an alignment is reusable for system evaluation, while the remainder address completeness and non-trivialness ^[7]. Their discovery of 30 correspondences requiring removal demonstrates that a reference alignment itself needs acceptance controls ^[7]. It does not establish a universal expected rejection percentage because the denominator and error mechanisms are corpus-specific.

Precision is the share of suggested correspondences that are correct, recall is the share of reference correspondences recovered, and F-measure is their harmonic mean ^[26]. These metrics evaluate sets against a reference. They do not by themselves test the consequences of a particular predicate for a particular consumer.

Semantic evaluation adds another caution. Standard precision and recall have limitations under OWL 2 semantics ^[27]. If merged ontologies and mappings are incoherent, published guidance recommends repair before semantic precision and recall are computed ^[27]. When a reference is incomplete, even semantic precision may be misleading, so only semantic recall may be appropriate ^[27].

Human agreement is also evidence, not truth by default. One OAEI uncertain reference alignment derived confidence from agreement among 20 reviewers ^[19]. Expert panels and consensus remain practical ways to construct references ^[11], but acceptance should retain disagreement, rationale, and reviewer identity when they matter.

Review summary calculation

For an actual release, report mutually exclusive row outcomes with a common denominator:

• Approval rate: approved rows divided by all reviewed rows.

• Downgrade rate: downgraded rows divided by all reviewed rows.

• Rejection rate: rejected rows divided by all reviewed rows.

• Deferral rate: deferred rows divided by all reviewed rows.

• Closure surprise rate: newly inferred endpoint pairs that fail review divided by all newly inferred endpoint pairs reviewed.

• Drift impact rate: previously active rows changed after release review divided by all active rows touching changed entities.

Counts should accompany percentages, and deferred rows must remain in the denominator. No cited source establishes a universal pass-rate benchmark, so a report should not manufacture one. Trend comparisons are meaningful only when predicate mix, consumer scope, sampling method, and review policy remain comparable.

Implications and Future Directions

The acceptance method changes procurement and governance questions. A mapping product should be asked to export assertions, provenance, endpoint versions, and review state. A reasoning component should expose materialized consequences. A terminology service should provide resolvable identifiers and release history. A consumer should supply executable positive, negative, and boundary cases. No single score substitutes for those observable properties.

Persistent identity and semantic drift must be handled together. Getty's editorial guidance prioritizes keeping subject identifiers persistent during merges ^[28]. Crossref similarly requires the new owner's DOI prefix in transfer updates ^[29]. These practices preserve addressability through organizational or editorial change, but consumers still need to retest meaning and use.

Mapping governance will also need clearer confidence semantics. A numeric confidence can mean model output, annotator agreement, evidence strength, or calibrated probability. SSSOM allows a value between 0 and 1 ^[3], but a producer should state the estimator, reference population, calibration method, and intended decision threshold before consumers interpret it probabilistically.

Finally, mapping publication should become more explicit about status. GRC Schema's published rules provide a current example: notes identify endpoint versions, and a changed canonical mapping requires a vocabulary version change ^[30]. The general lesson is not to copy one rule set. It is to make acceptance policy, versions, and changes inspectable by consumers.

Frequently Asked Questions (FAQs)

What is an ontology crosswalk acceptance test?

It is a black-box review that decides whether a specific, versioned mapping claim is safe for named downstream uses. It verifies endpoint evidence, predicate semantics, inferred consequences, consumer queries, and version drift, then records approval, downgrade, rejection, or deferral.

What is the difference between exact match and close match?

In SKOS, both are symmetric. Exact match is transitive and expresses high-confidence interchangeability across a wide range of retrieval applications. Close match is not transitive and supports interchangeability only in some applications. Therefore, an A to B to C chain can imply A exactMatch C, but cannot imply A closeMatch C merely from two close mappings.

How should broader and narrower concept mappings be validated?

Validate direction first. For broadMatch, the subject is more specific and the object more generic. Then test roll-up, drill-down, exclusions, and aggregate counts. Materialize the inverse narrowMatch consequence and verify that neither direction is mistakenly consumed as equivalence.

Is SKOS exactMatch the same as owl:sameAs?

No. owl:sameAs states that two individuals are the same resource, while skos:exactMatch is a concept-mapping relation designed for cross-scheme interchangeability. owl:equivalentClass is different again because it asserts equal class extensions. The endpoint kinds and downstream logic determine which category is applicable.

Which ontology alignment evaluation metrics belong in acceptance?

Precision, recall, and F-measure describe alignment-set performance against a reference. Relation-aware and semantic metrics help when predicates and entailments matter. Row acceptance additionally needs consequence inspection and consumer-query tests because aggregate metrics can hide a small number of consequential false equivalences.

What should happen when no predicate fits?

Record no acceptable mapping with a reason and reviewed evidence. Do not leave a blank and do not force the weakest available relation if even relatedMatch would mislead consumers. An unmapped row can be the correct governed decision.

When should an accepted mapping be reviewed again?

Retest when either vocabulary publishes a release, an endpoint changes meaning, hierarchy, or status, a concept splits or merges, mapping policy changes, the consumer action expands, or an observed result violates the fixture. Also record a periodic backstop date for mappings whose sources do not publish machine-readable change events.

Conclusion

An ontology crosswalk acceptance test is a controlled decision about a semantic claim. It starts with the consumer action, fixes the source and target releases, selects the weakest adequate predicate, inspects the predicate's formal consequences, executes positive and negative boundary queries, and records a versioned outcome. This makes approval defensible and makes downgrade, rejection, deferment, and no acceptable mapping equally legitimate results.

The critical distinction is between candidate plausibility and accepted consequence. Labels and confidence scores can prioritize work, but the predicate determines what consumers may infer. Exact, close, broader, narrower, related, identity, and class-equivalence claims are not interchangeable. Their endpoint kinds, direction, symmetry, inverses, transitivity, and disjointness must be tested explicitly.

The final artifact should be reproducible: identifiers, observed labels and definitions, releases, predicate, justification, evidence, author and reviewer roles, fixture results, decision, observation date, and retest trigger. Metrics can summarize a release, but no universal pass rate replaces row-level evidence. A high-quality crosswalk is therefore not the one with the most links. It is the one in which every active link has survived the consequences that its actual consumers will rely on.