Back to Articles|Published on October 31, 2026|30 min read

Reliable Context Article

JSON Evidence Binding Test: Right Value, Right Record?

A 2026 protocol for a JSON evidence binding test: frozen manifest, per-record ledger, RFC 6901 pointer fixtures, matched order and path conditions, and when deterministic assembly beats LLM nesting.

By Reliable Context Research

JSON Evidence Binding Test: Right Value, Right Record?

Summary

  1. 01A JSON document can parse, pass its schema and preserve correct values while attaching them to the wrong record, parent or hierarchy level.
  2. 02Correct binding requires the complete typed record to appear exactly once at its intended location. Repeated values make matching individual leaves insufficient.
  3. 03Diagnose assignment and hierarchy errors separately: input ordering targets assignment errors, while record-level paths target hierarchy errors and can increase truncation.
  4. 04The proposed ledger records independent source identity and expected versus actual locations. Ambiguous or identical records remain untraceable rather than receiving a forced match.
  5. 05The results are author-reported synthetic findings, and transfer to extraction remains untested. Use conventional transformation code when the mapping is fully specified.
Inside this article
  1. 01Executive Summary
  2. 02Introduction and Background
  3. 03Key Changes in How Structured Output Should Be Evaluated
  4. 04Binding Ledger Protocol: Implementation Considerations and Process Changes
  5. 05Provenance, Lineage and Evidence Binding
  6. 06Data Analysis and Evidence
  7. 07Implications and Future Directions
  8. 08Frequently Asked Questions (FAQs)
  9. 09Conclusion

Executive Summary

A JSON document can parse, satisfy its schema and contain every correct string, yet still bind a value to the wrong record, parent or hierarchy level. A JSON evidence binding test asks, for each source record supplied to a model, whether that record's content appears exactly once at the location the grouping rules intend. This report, current as of 31 October 2026, builds a reusable evaluation ledger for that question around one recent paper and a handful of standards.

The anchor is Zhang et al., "Right Values, Wrong Places," revision 2 dated 14 October 2026. It introduces JSONPlacement, a task that supplies every value and every grouping rule, so a misplaced value cannot come from misreading text. The authors separate record-assignment errors from hierarchy-realization errors. They report that whole-record moves make up 98.6% of strictly traceable synthetic misplacements [1]. On their original structure, one 12B open-weight model preserved 97.2% of target values yet bound only 53.4% to the correct path [1]. These are author-reported results in a synthetic setting, not deployment guarantees.

Schema validity does not close the gap. One vendor documents schema-compliant responses through constrained decoding [2]; another states that structured outputs can still contain mistakes [3]; and an independent benchmark of 21 models reports exact leaf-value accuracy of 0.693 to 0.830 on text despite near-perfect schema compliance [4]. A pointer such as RFC 6901 names a location in one document [5]; it does not establish who the value is about.

The proposed protocol has six parts: a frozen source manifest with scoring-only record identifiers, a per-record ledger, matched conditions that vary input order and output representation one at a time, a deterministic baseline, fresh held-out fixtures, and resampling by document rather than by leaf. In the paper, ordering the input by target traversal and writing record-level paths were complementary, and combining them solved every new synthetic instance [1]. Record-level paths also lengthened outputs and increased truncation.

The decision rule is plain. If the mapping from records to hierarchy is fully specified, conventional transformation code is the appropriate tool. If a model must organize, change the representation only after diagnosing which error type dominates. No experiment was run for this report, and transfer to extraction with inferred assignments remains untested [1].

98.6%Share of strictly traceable synthetic misplacements attributed to whole-record moves, as reported by the authors
97.2%Target values preserved by one 12B model on the original structure, as reported by the authors
53.4%Values bound to the correct path by that same 12B model on the original structure, as reported by the authors

Introduction and Background

Context engineers increasingly hand structured text from a language model to another program. Two questions are routinely conflated: whether the output is well formed, and whether each value still belongs to the subject it was supplied with. A syntactic validator answers only the first. JSON Schema is a declarative language for defining structure and constraints for JSON data [6]. It can restrict a value to a fixed set of values [7], although one vendor documents that a returned enum value might differ from the schema only in capitalization [2]. It cannot say which source record a permitted value was drawn from.

This report treats that gap as an information problem, not a sector problem. It is a protocol proposal: nothing in these pages was measured by the author of this report.

Definitions used throughout

  • Value preservation: the share of target values that appear somewhere in the output with the right type and content, regardless of path. The JSONPlacement paper defines it as target leaves matched one-to-one to output values, ignoring paths.
  • Source-record identity: a stable identifier for each supplied record, held by the evaluator. It is an independent input to scoring, not something recovered from the path the model chose.
  • Intended parent and path: the location the grouping rules assign to a record, expressed as a JSON Pointer in a particular output document.
  • Output schema validity: conformance to a schema. It certifies shape and types, and it says nothing about which record a value came from.
  • Correct binding: every content field of a supplied record is written once, with its original name, type and value, at that record's intended path, with no extra or conflicting declarations.
  • Untraceable record: a record whose output location cannot be established without a convenient guess, because its content is duplicated, altered or ambiguous. It is labeled as such and is never assigned a match for the sake of a tidy denominator.

Why string matching cannot certify binding

JSON Pointer (RFC 6901, April 2013) is a string syntax for identifying a specific value within a JSON document [5]. The standard warns that a given pointer is not guaranteed to reference an actual value [5]. A pointer is therefore a locator inside one document. It carries no claim about subject identity, truth, or persistence across a reorganization.

A small transformation example (Hypothetical Example)

Consider a fictional fixture with two source records. The fixture identifiers R01 and R02 are held by the evaluator and are not shown to the model.

[
  {"school": "North", "class": "C1", "student": "S-07", "name": "A. Reyes", "grade": "B"},
  {"school": "North", "class": "C2", "student": "S-07", "name": "B. Ortiz", "grade": "B"}
]

The rule is: nest students under school, then class, then student identifier. The expected pointer for R01's grade is /schools/North/classes/C1/students/S-07/grade, and for R02's grade it is /schools/North/classes/C2/students/S-07/grade. Suppose the model writes R02's whole record under class C1 and R01's whole record under class C2. The output parses and passes a schema. A leaf check on the grade field passes at both paths, because both expected values are the string "B". Only the name field, or the evaluator's record identifier, reveals that each path now describes a different student. If the fixture checked only grades, a bag of correct strings and a valid schema would coexist with two wrong subject-value assertions.

This is the shape of failure the JSONPlacement paper documents: a student in Class 1 written under Class 2 keeps name and grade, but a downstream program now counts the student in the wrong class.

Key Changes in How Structured Output Should Be Evaluated

From valid JSON to value-path binding

The revision of 14 October 2026 is titled "Right Values, Wrong Places: Diagnosing and Fixing Record Misplacement in LLM-Generated JSON" and is by Zhang, Wu, Li, Wang, Yao and Li; older index entries may show an earlier title, and this report uses the v2 title and methods. The JSONPlacement task supplies flat records, grouping fields that jointly determine each record's target path, and content fields whose names determine leaf paths. A simple program generates the reference, so every instance has one unambiguous answer, and the models only reorganize the input, with no retrieval, joins or text extraction [1].

The headline observation is the gap between preserving a value and binding it. On the original structure, no 8 to 12B model solved more than one of 24 instances, while one 12B model preserved 97.2% of values and bound 53.4%.

Two error types, kept separate

A record-assignment error writes a record under another valid record path, such as a Class 1 student under Class 2 [1]. A hierarchy-realization error changes the parent and child relations between containers, for example writing a school at the root instead of inside the schools container [1]. A ledger must keep them apart because the published remedies differ, and labels such as omission, duplication, conflict and hierarchy displacement can overlap and must not be summed as mutually exclusive categories.

Whole-record movement is the dominant pattern in the traceable subset. In four earlier synthetic conditions, 80.8% of misplaced leaves could be traced to a unique destination [1], and 98.6% of those belonged to whole-record moves. If fields erred independently, about 0.06 records would be expected to move whole; the authors observed 6,293.

Diagnose before intervening

The paper derives two interventions from the error patterns and fixes its predictions before generating new instances. Ordering the input by target traversal targets assignment errors; replacing nested output with record-level paths targets hierarchy errors. On new synthetic instances, ordering nearly eliminated records written to other valid paths and whole-record omissions for all three models [1], while record-level paths showed the opposite pattern and lengthened outputs so that more reached the token limit [1].

A dose-response check supports the ordering explanation: for one model, exact matches rose from 0 under full shuffling to 10 under local shuffling and 24 under full ordering. Location in the output does not by itself identify the cause, since both closure errors and key-selection errors can produce out-of-tree records. The authors state that their evidence establishes behavioral patterns, not a unique internal mechanism, and this report draws no conclusion about hidden model mechanisms.

What schemas and constrained decoding do and do not certify

JSONSchemaBench (Geng et al., 27 February 2025 revision) is a benchmark for constrained decoding with 10K real-world JSON schemas [8]. It pairs the schema set with the official JSON Schema Test Suite, a language-agnostic collection that validator implementers use to test their libraries [9]. Its quality component evaluates specific reasoning tasks. It separates coverage, efficiency and quality, which is the right separation, but it does not supply a source-record-binding certification, and this report makes no claim that the paper ignores quality or that constrained decoding generally harms accuracy.

Vendor documentation is consistent with that reading. One vendor states that structured outputs ensure the model always generates responses that adhere to the supplied schema [3] and also that they can still contain mistakes [3]. Another documents schema-compliant responses through constrained decoding [2], with exceptions, including an output that may be incomplete and fail the schema when it hits the token limit [2]. Truncation matters here because the record-path intervention raises it. These are vendor statements about their own features, and each is limited to schema adherence.

Adjacent evidence on syntax versus semantics

Several recent studies separate syntactic compliance from content correctness, with different tasks, so their rates are not comparable to each other or to JSONPlacement.

  • StructHallu-Drift (Hasan, SURGeLLM, 3 July 2026) proposes a taxonomy that disentangles syntactic validity from semantic fidelity [10]. It mutates each of 100 schemas at three levels [10], and among outputs that passed schema validation, the residual semantic hallucination rate was 8.0% [10], and the authors conclude that no single validation layer suffices [10]. Its field-confusion type, a value in the wrong schema field [10], is the nearest analogue, but its task changes the schema and differs from a fixed-schema organization test.
  • Structured Output Benchmark reports near-perfect schema compliance alongside much lower exact leaf-value accuracy (Value Accuracy of 0.693 to 0.830 on the text modality across 21 models; [4]), and it is question-driven extraction, not reorganization of supplied records.
  • Tool-output processing reports a qualitative failure of picking a value for the right key from the wrong record in JSON responses [11]. That is the same family of error in a question-answering setting.
Figure 01
Two placement errors, two targeted interventions
Record-assignment error
  • A record is written under another valid record path, such as a student appearing under the wrong class.
  • Ordering the input by target traversal targets assignment errors.
Hierarchy-realization error
  • Parent and child relations between containers change, such as a school appearing at the root instead of inside its container.
  • Replacing nested output with record-level paths targets hierarchy errors.

Record-level paths also lengthened outputs and increased truncation. The reported interventions were tested on synthetic instances; transfer to extraction remains untested.

A correct string in a valid document can still be asserted about the wrong subject.

Binding Ledger Protocol: Implementation Considerations and Process Changes

The protocol below is a proposal. It translates the paper's design into an auditable procedure and adds elements the paper does not supply, such as array-aware pointer fixtures.

Step 1: Freeze a source manifest

Create a fixture set before any model call. Each source record receives a stable fixture identifier, and the manifest stores the original source text or cells, the explicit grouping rules, and a separately computed expected output. The JSONPlacement artifacts illustrate the practice: dataset and per-task hashes in the run metadata identify the exact archived inputs [1]. A canonical serialization makes your own hash repeatable: RFC 8785 describes an invariant format so that hashing operations are reliably repeatable [12] and requires input without duplicate property names [12] while keeping array element order [12].

Resolve or disclose ambiguous mapping rules before running the model. Identifiers used only for scoring must not alter the model's input, because an identifier-visible arm is a distinct intervention. The paper's unique-identifier diagnostic shows why: making grouping identifiers unique cut records written to other valid paths sharply, yet exact matches changed little because omitted and out-of-tree records remained.

Step 2: Validate parse and schema first, then bind

Run the checks in this order and keep each stage's denominator.

  • Parse and schema compliance: count all attempted outputs, including truncated and invalid ones. The paper counts truncation, invalid JSON and uninterpretable representations as whole-instance failures.
  • Complete-record comparison: compare each supplied record, field names and types included, against the output. Preserve extra, missing and duplicated records instead of aligning them away.
  • Subject-value-path binding: require the right value at the right pointer for the right fixture record. A matching leaf is insufficient when the same value occurs more than once.
  • Conditional diagnosis: classify placement errors only for preserved, traceable records, and report that count beside the all-record denominator.

The paper's scoring is a useful reference. It compares reference and output by full paths, node types and leaf values, ignoring object-key order and applying no output repair. Matching first preserves correct bindings, then matches remaining equal-valued entries, so no output value can satisfy multiple targets. The evaluator also retains every declaration of duplicate keys, which matters because RFC 8259 says object names SHOULD be unique [13] while many implementations report the last name and value pair [13]. A parser that silently keeps the last duplicate can hide a conflicting declaration.

Step 3: Keep the ledger

Table 1 below lists the fields of the proposed binding manifest. Each row of the ledger corresponds to one supplied source record or one source field-value pair.

Ledger fieldContentWhy it is needed
Input version and hash, retrieval dateDigest of the frozen manifest and the date the inputs were retrieved or generatedMakes a rerun comparable
Source record ID and locatorFixture ID, plus the cell, row or span in the original sourceIdentity is an independent input, not inferred from the model's path
Subject and field/valueThe entity the record describes, the field name, the typed valueDistinguishes a value from the subject it is asserted about
Expected parent and pointerJSON Pointer under the grouping rulesNames a location within one particular document
Actual parent and pointerWhere the evaluator found the record's content, or "none"Supports assignment versus hierarchy diagnosis
Value changedYes or no, with the observed valueSeparates a moved record from an altered one
Duplicated or omittedCount of complete occurrences in the outputExtra copies must stay visible, not be aligned away
Traceability statusTraceable, untraceable, or ambiguous, with the reasonUntraced values are reported, not forced into a match
Supporting evidenceThe source text or cell that justifies the expected pathLets a reviewer defend the oracle, not only the score

The table works as a checklist, not a data model. Its design choice is that expected and actual locations are recorded per source record, so a reviewer can answer "which record is this value from?" without re-running anything. Exact tracing requires known target bindings and identifiable source records; when alignment cannot be established, label the record untraceable. Source records with identical complete tuples cannot support unique movement attribution either.

Step 4: Pointer fixtures that survive RFC 6901

Pointer handling is a place where an evaluator can introduce its own errors. Include fixtures for each of the following.

  • Escaped slash and tilde: a key such as "Q3/Q4~adj" (a fictional fixture key) is written with the slash as ~1 and the tilde as ~0, giving the reference token Q3~1Q4~0adj. When decoding, the order matters: the RFC says that applying the substitutions in the stated order avoids the error of turning ~01 into a slash [5].
  • Array indices: a token addresses an array element by zero-based index [5], and the dash token refers to a nonexistent member past the end, which results in an error when evaluated [5].
  • Reordering: RFC 8259 defines an array as an ordered sequence [13], so a permitted reordering changes indices without changing the subject. A pointer change caused by permitted reordering is not automatically a semantic reassignment; key such fixtures by record identity.
  • Index shifts under edits: in JSON Patch, an add shifts later elements one position [14] and operations are applied sequentially [14], so a pointer is meaningful only relative to a document state.
  • Duplicate names: if a name is not unique in an object, the referenced member is undefined [5].
  • Unicode forms: member names match by code points, so differently normalized spellings are different keys [5].

One boundary deserves emphasis. The JSONPlacement target contains only objects and leaf values, with no array alignment involved, so its published results say nothing about array-ordered targets.

Step 5: Run matched conditions

Hold the model, decoding budget and grouping policy fixed, and disclose the settings, as the paper does with its temperature and token budget. Run four prespecified conditions on the same fixtures:

  • Original input order, nested output (baseline).
  • Grouped input order, nested output (targets assignment errors).
  • Original input order, full record paths (targets hierarchy errors).
  • Grouped input order, full record paths (combined).

Input order matters beyond this task: in a long-context study, multi-document question-answering performance could drop by more than 20% when relevant information moved [15], and a synthetic JSON key-value task probed whether models simply retrieve matching tokens from the middle [15]. Disclose output-length changes and context limits. Repeating a complete record path has a measurable length cost, and in the paper's path condition some outputs were truncated. Smaller error counts in a truncation-heavy cell cover fewer answers, so a higher conditional mean with fewer scorable outputs need not represent an improvement.

Use fresh, held-out fixtures for the decision after designing the intervention. Tuning against a test set leaks knowledge into the model choice, so evaluation metrics no longer report on generalization performance [16]; a test set should still be held out for final evaluation [16]. The paper's own fresh validation fixed its predictions in advance and scored every output without retries or repair.

Step 6: Add a deterministic baseline

Where the supplied rules are executable, implement them in conventional code. This checks the fixture oracle and answers whether a language model is necessary for this organization step. Keep downstream factual extraction separately evaluated. The baseline must preserve order only where order is semantically meaningful, and common tools differ on this point.

  • jq: group_by returns an array sorted by the grouping value [17], and unique_by keeps only one element per key [17], so it is not a lossless grouping; unsorted key listing is only roughly in insertion order [17].
  • Python: itertools.groupby needs input already sorted on the key [18], unlike SQL grouping, which works regardless of input order [18]; dictionary order is guaranteed to be insertion order [19].
  • pandas: with sorting disabled, groups appear in their original order [20], and rows keep their order within each group [20].
  • JavaScript: Object.groupBy returns an object with one property per group [21], and integer-like keys are listed first in ascending numeric order [22], so record identifiers such as "10" and "2" can be silently reordered. Other keys follow creation order [22], and Map.groupBy uses the unique callback values as keys [23].
  • PostgreSQL: aggregate input ordering is unspecified by default unless an ORDER BY is written inside the call [24], and the binary JSON type does not preserve the order of object keys [25] and keeps only the last value for duplicate keys [25].

One vendor documents that structured outputs are produced in the same order as the ordering of keys in the schema [3], so key order there reflects the schema, not the data. Because RFC 8259 defines an object as an unordered collection of name and value pairs [13], and parsing libraries differ in whether they expose member order [13], compare object members by key, not position, and compare arrays by position only where the schema declares order meaningful.

Step 7: Report by depth, count, repetition and ambiguity

Stratify results by group depth, record count, repeated values and ambiguity; in the paper, value preservation and binding nearly coincided at 16 records per instance and diverged at larger counts. Report these quantities:

  • Schema-valid output fraction: over all attempted outputs.
  • Correctly bound records: correct records divided by all expected records.
  • Exact-document agreement: the fraction of outputs entirely correct.
  • Extra and duplicate records: counted, not aligned away.
  • Placement errors conditional on preserved, traceable records, beside the all-record denominator.
  • Hierarchy realization versus wrong-record assignment, using the published rules, with mixed or ambiguous failures left visible.

Do not infer record accuracy by multiplying independent field scores, because fields move together. Resample by generated instance or source document, not by leaf count; the paper treats conditions sharing a base instance as paired observations. Eval-statistics guidance points the same way: when questions are drawn in related groups, clustered standard errors are appropriate [26]. For paired conditions on the same fixtures, a paired bootstrap resamples one set of indices and applies it to every sample [27], and for a binary per-document outcome McNemar's test applies, which Dietterich (1998) found to have low type I error [28], with an exact variant based on the binomial distribution [29]. An unpaired bootstrap would instead independently resample the elements of each array [27]. Pass one element per document, not per leaf, to whichever routine is used.

Step 8: Extend to extraction only as a new experiment

Only after the supplied-record test is stable, extend to source-text extraction with independently annotated subject, field, evidence span and path. This is a transfer experiment, not something the original paper validated: its task supplies complete records and explicit grouping rules, and input ordering applies only once grouping fields are explicit. The extraction stage needs its own identity resolution and its own ledger.

Figure 02
Validate first, then test record binding
  1. 01Parse and schema compliance

    Count every attempted output, including truncated and invalid ones, and retain the denominator for this stage.

  2. 02Compare complete records

    Compare supplied records with their field names and types. Keep extra, missing and duplicated records visible.

  3. 03Check subject, value and path

    Require the right value at the right pointer for the right fixture record. A repeated value cannot establish binding through a leaf match alone.

  4. 04Diagnose traceable placements

    Classify placement errors for preserved, traceable records, and report their count beside the denominator for all records.

Provenance, Lineage and Evidence Binding

Readers often ask how provenance differs from lineage, what counts as source attribution for a value, and what an audit trail for values must hold. The literature is less tidy than many guides suggest.

Data provenance versus data lineage

The W3C PROV data model, a Recommendation dated 30 April 2013, defines provenance as information about entities, activities and people involved in producing a piece of data or thing [30]. An entity may be physical, digital, conceptual or another kind of thing [30], a derivation is a transformation of one entity into another [30], and attribution ascribes an entity to an agent [30]. PROV-O expresses that model in OWL2 [31].

OpenLineage describes itself as an open framework for data lineage collection and analysis [32]; its jobs are process definitions that consume and produce datasets [33], and its column-lineage facet records which input columns are used to produce which output columns [34], and distinguishes a direct derivation from an indirect influence where the output column value is impacted by the value of an input [34]. That is flow-level metadata about data moving through jobs. It does not name the JSON Pointer a value occupied inside a particular document.

Authoritative sources do not agree on a crisp distinction. The database-theory survey by Cheney, Chiticariu and Tan (2009) says provenance describes the origins and the history of data [35], and treats the earlier lineage notion as the source tuples that witness an output tuple's existence [35]. A vendor blog states the common practitioner split: provenance is the historical record of data, including where it was created [36], and the two terms answer different questions [36]. That source is a practitioner view, not a standard. A working convention for this report: lineage concerns the flow of data through systems and jobs; provenance concerns the origin and derivation of a particular value; evidence binding is the further requirement that a value be attached to the correct subject record and location in the output.

Where a value came from: why and where provenance

Database provenance offers precise vocabulary for the neighboring question. Buneman, Khanna and Tan distinguish why-provenance, the source data that influenced the existence of the data [37], from where-provenance, the locations in the source databases from which the data was extracted [37]. Where-provenance describes where a piece of data is copied from [35], and it is not invariant under equivalent queries [35]. The binding ledger is a where-provenance record for an LLM transformation step: it maps a source location to an output location. By analogy, an expected pointer is valid only for a stated target structure and hierarchy rule.

Source attribution, fusion and the audit trail

Source attribution for data values has three layers that should be kept apart. The first is the origin: the document, cell or span a value was read from. The second is the binding: the subject and field to which the value was assigned in a given representation. The third is the observation: when and by whom the value was read. A resolvable citation covers only the first layer, and a valid schema covers none.

Knowledge fusion work makes the same separation from the other direction. Dong et al. define knowledge fusion as identifying true subject-predicate-object triples extracted by multiple information extractors [38], and Knowledge Vault computes calibrated probabilities of fact correctness (Source: research.google), reporting about 38 times more confident facts than the largest previous comparable system [39]. These methods decide which claim about a subject is likely true; they presuppose that each claim is already bound to the right subject. A misplacement upstream turns a true value into a false claim before fusion starts. For organizations that fuse private and public information, the practical audit trail is therefore a chain: source locator, subject, field, value, output location, observation date, and any change since.

A system of record is where that chain lives after a value is accepted. Publisher note: Reliable Context, which publishes this report, offers a system of record of this kind. That is a description of the publisher's offering and is not evidence for any claim in this report. A system of record complements the placement test: the test measures whether a transformation step preserved bindings, and the record keeps the chain of custody for accepted values. A reader who needs a vector database, a data catalog or a master data suite should use those tools; they are different products.

Data Analysis and Evidence

The evidence is thin and specific. The JSONPlacement results come from one research group, in one controlled setting, for open-weight models of 4B to 31B parameters. No source in this report measures the proposed ledger, and none supports production claims.

Table 2 below reproduces the baseline value-preservation and value-path binding figures for the original structure, and the exact-match counts for the four matched conditions on new synthetic instances, as reported by the authors.

Measure (JSONPlacement, Original structure)Qwen3.5-9BGemma 4 12B ITMinistral 3 8B Instruct
Baseline value preservation (VP), 24 instances92.62%97.22%95.83%
Baseline value-path binding accuracy (VPA)83.51%53.39%66.49%
Baseline exact matches, original instances0/241/240/24
New instances, exact matches: baseline0/243/240/24
New instances, exact matches: ordered input, nested output24/248/2417/24
New instances, exact matches: original order, record paths0/2412/240/24
New instances, exact matches: ordered input and record paths24/2424/2424/24
Median output tokens, ordered nested to combined (Original)3,480 to 3,6403,541 to 3,9893,123 to 3,533

Source: Zhang et al., Tables 2 and 3 and Appendix E.2, author-reported. Counts are exact matches out of 24 per cell, from runs at temperature 0 with a 16,384-token output budget.

Three readings of Table 2 are supported. First, preserved and bound values diverge widely in the baseline: the gap is 9 points for one model and 44 points for another. Second, the interventions are complementary and model-dependent: input ordering alone lifted one model to 24 of 24, while another needed the combination. Third, the combined condition costs tokens: median output length rose by roughly 160 to 450 tokens per instance on Original, and record paths alone produced truncations in some cells. Nothing here shows that the same gains hold for other models, for extraction, or at production scale; the authors note that the combined intervention has not been validated across multiple real-data sources.

Repeated values deserve a separate comment. The JSONPlacement synthetic values are unique within each instance, while in real data repeated individual values still preclude applying the stricter single-occurrence leaf criterion [1]. The proposed ledger turns that limitation into a reported quantity: count the share of records whose content tuple is duplicated, and stratify by it.

Table 3 below places the controls discussed so far side by side, with what each can and cannot certify about binding.

ControlWhat it can certifyWhat it cannot certify
JSON Schema validationShape, types and permitted values [7]Which source record a value came from
Constrained decoding or vendor structured outputsSchema adherence within a supported schema subset (one vendor lists up to 5000 object properties [3]), subject to documented exceptions such as truncation [2]Value-to-record binding; documented as able to contain mistakes [3]
Tree-shape grammar (author-reported)Fixes container nesting, key names and identifier formats stated in the promptRecord bindings or values; the variant that also fixed the top-level group count is gold-informed [1]
JSON PointerA location in one documentSubject identity, truth, or persistence across reorganization
Binding ledger with oracle (proposed)Per-record expected versus actual location, with traceability statusTruth of the source itself; transfer to extraction
Deterministic assemblyThe executable grouping rules, with order handled explicitlyCases where the mapping is not fully specified
System of record (publisher note: Reliable Context)A system of record should store the source locator, observation date and change history behind each accepted valueIt is not a placement test; it complements one

Table 3 shows that no single control answers the binding question. The gold-informed row deserves attention: a grammar that also fixes the number of top-level groups uses knowledge of the answer, and with it the model in that diagnostic was correct on every instance. Such a constraint is a diagnostic, not a deployable control, whenever the group count would not be known in advance. The final row describes the publisher's own offering and is not evidence about the other rows.

Figure 03
Gemma 4 12B IT: baseline preservation versus binding%
Source: Zhang et al., Tables 2 and 3 and Appendix E.2, author-reported.

Where the mapping is fully specified, use conventional code. Where a model must organize, test binding, not only validity, and let the dominant error type choose the change.

Implications and Future Directions

A decision rule for context engineers

The decision is among three options: retain model-based organization, change the representation, or use deterministic assembly.

  • Use deterministic assembly when the mapping is fully specified and executable. The paper suggests sorting generated records by their grouping fields before organization and notes that a program can assemble the hierarchy [1]. Then conventional transformation code is the appropriate tool, and the model's job, if any, moves upstream to producing the records.
  • Change the representation when a model must organize and the ledger shows a dominant error type. Assignment errors point to input organization; hierarchy errors point to the output form. Ordering presupposes that grouping fields are explicit, so identity must have been resolved earlier.
  • Retain model-based organization when held-out, cluster-resampled results meet a threshold fixed before the run, for the depth, record count and repetition profile that production data will have, and when truncation under any longer representation stays within the budget.
  • Do neither when alignment cannot be established. Label the records untraceable and fix the identity input first.

What remains unknown

  • Transfer to extraction: untested in the paper, which leaves open whether its interventions help where assignments must first be inferred from text.
  • Mechanism: behavioral patterns do not identify an internal cause, and record-level paths change output length, bracket structure and grouping-key repetition together, so effects cannot be attributed to one factor.
  • Public artifacts: the paper's footnote gives a link to an anonymized code repository, and its evidence index lists archive-relative run identifiers and generation commits. This report did not open the repository or run any code, so runnable reproduction should not be promised until the artifact is pinned and exercised.
  • Array-ordered targets: the published task contains only objects and leaf values, so array-aware binding is open.

Frequently Asked Questions (FAQs)

What is a JSON evidence binding test?

It is an evaluation in which each supplied source record has an independent identity, and the evaluator checks that the record's complete content appears exactly once at the location the grouping rules assign. It extends value-level checks by adding the subject and the path.

How do you verify that a value belongs to the right record?

Compare against an oracle computed separately from the model, keyed by fixture record identity, and compare complete typed records, not only leaves. When values repeat, a leaf match is insufficient, and records with identical complete tuples cannot support unique attribution; label them untraceable.

How do you verify integrity between a source and a record?

Hash the frozen inputs and store the source locator and evidence for every expected binding. Rerun against the same hash to reproduce a result. Canonical serialization such as RFC 8785 helps make the hash repeatable. Integrity of the transformation is separate from the truth of the source.

How does this relate to fusing private and public data?

Fusion methods choose among conflicting claims, and they assume each claim is already bound to the right subject. A binding test protects that assumption at the point where a model reorganizes records; it does not decide which source is more reliable.

Conclusion

A correct string in a valid document can still be asserted about the wrong subject. The October 2026 revision of the JSONPlacement paper makes this concrete by supplying every value and rule, so the remaining failure is organization, and by reporting that models can preserve most values while binding far fewer to the right path. Its separation of record-assignment from hierarchy-realization errors, and its matched input-order and output-path interventions, give a testable way to diagnose before changing a system.

The ledger protocol turns those ideas into a procedure: a frozen manifest with scoring-only identifiers, per-record expected and actual locations, RFC 6901 pointer fixtures with escaping and arrays, four matched conditions, a deterministic baseline, held-out fixtures and document-level resampling. Untraceable records stay untraceable.

The evidence has firm limits. The results are author-reported, from one research group, in a setting where transfer to inferred assignments is untested. Nothing in this report was run, and the proposed ledger has no measured performance. Where the mapping is fully specified, use conventional code. Where a model must organize, test binding, not only validity, and let the dominant error type choose the change.

External Sources (39)

Published by

Reliable Context Research

Reliable Context is a system of record for organizations that fuse private and public information and must defend every value they act on: one canonical record per real thing, ambiguity shown rather than hidden, and provenance behind every value.

Our product · Evidence and editorial standards · Contact

Request access

Editorial note

Researched and written by Reliable Context Research with AI-assisted research tools; sources are cited inline and checked before publication. Report an error: https://reliablecontext.com/corrections.