How to make AI answers auditable with provenance

Short answer

An AI answer is auditable when each fact in it points to the document it came from and says how far that fact can be trusted. In a knowledge graph you store this on the facts themselves: the source document and section, when the fact was recorded, and a status such as declared, observed, derived or verified, where a fact can only be marked verified if it links to its evidence. The query that answers a question returns these fields with the rows, so the citations come from the data instead of being written by the model afterwards.

Why "we store the source" is not enough

The usual first step toward traceability is storing a link to the source document. That answers "where did this come from" but not "how much should I trust it". A supplier's declaration on a form, a warehouse scan, an inference from other facts and a lab result can all produce the same relationship, for example (:Batch)-[:CONTAINS]->(:Material). Without a marker, the graph treats a claim and a checked fact as the same thing, until an audit or a fraud shows they were not.

The horsemeat scandal of 2013 is a well-known example. Beef lasagne sold in several European countries contained horse meat that had passed through companies in five countries before it reached the shelf. Each company had records of its direct supplier, as EU food law requires, and those records said "beef" because the paperwork they received said "beef". Nothing in them separated a supplier's word from a checked fact, and it took DNA tests to show the difference.

Four levels of trust

A useful scale comes from Eigenius, a research knowledge-graph database described in an August 2026 paper by Will, Brown and Fuchs, which enforces the status as a commit-time rule. The meanings and examples below are my reading of it for a supply chain:

Status	Meaning	Supply-chain example
declared	Someone asserted it	The supplier's paperwork says the shipment contains palm oil from mill X
observed	A sensor, scan or inspection recorded it	A warehouse scan logged the batch arriving
derived	Computed or inferred from other facts	Origin inferred because the shipment combined three traced lots
verified	An independent party checked it against evidence	A certifier or lab confirmed it against an audit trail

Grading sources is not new. Intelligence analysts rate source reliability and information credibility with the NATO Admiralty Code, medicine rates certainty of evidence with GRADE, and threat intelligence has a 0-100 confidence property in STIX 2.1. If your field already has a scale, use that one. What matters is that every fact carries a value.

How to model it in Neo4j

Start with properties on the relationship:

CREATE (b:Batch {id: 'B-4471'})-[:CONTAINS {
  status: 'declared',
  source: 'supplier-form-2026-117',
  recordedAt: date('2026-08-01')
}]->(m:Material {name: 'palm oil'})

A declared fact has no evidence property yet. Neo4j does not store null properties, so "no evidence" simply means the property is missing.

Neo4j constraints can require a property on every relationship of a type (Enterprise Edition), but not a conditional rule like "verified needs evidence". Enforce that one layer out: in the write path, with an APOC trigger, or at least with a test that fails the build.

// Must always return zero rows: a verified fact without evidence
MATCH ()-[r]->()
WHERE r.status = 'verified' AND r.evidence IS NULL
RETURN r

Then use the status in queries. Ranking supply paths by how many links are only declared shows where the evidence is thinnest, which is where an auditor's time is best spent.

When one fact needs several attestations (declared by the supplier and later verified by an auditor, both kept), move from a property on the relationship to a separate claim node between subject and object, one per attestation. Start simple and change only when a fact needs a history. The W3C PROV-O ontology is the standard vocabulary if you need to exchange provenance with other systems.

What provenance does not fix

A bad verifier. A verified tag is only as good as whoever verified it. A captured or careless certifier gives you a confident wrong answer, which is worse than an honest "declared".

The language model on top. The graph can return the evidence, but the answer layer still has to pass it through. Test that the generated query actually returns the source fields and that the answer names them.

Deep derivations. Derived facts build chains: a conclusion resting on a derivation resting on a declaration. Decide how far back a query walks before it trusts the result.

Facts that were never captured. Provenance tells you where each stored fact came from. It says nothing about what the extraction missed, so completeness needs its own check, such as a hand-labelled sample compared with what the pipeline returned.

Example from a working graph

In the 10-K graph, each relationship a language model extracted from the text of an annual report (competitors, dependencies, acquisitions, regulators) stores the filing, the section and the verbatim supporting quote as properties on the edge. The chat on top turns a question into a Cypher query, and the query returns those fields with the rows, so the answer names the filing and section behind each claim and you can check the source instead of trusting the wording. That only holds because the query step requires the provenance fields in every result. The graph stores the evidence, but the application has to ask for it.

Before an extracted relationship reaches the graph, a check confirms that its quote appears word for word in the cited section. Of 595 proposed relationships, 11 failed because the "quote" was a paraphrase. A second step rejects any record without a quote at all.

The same graph uses a status scale for matching company names. A name is verified when it matches a company in the SEC registry (48 of the 127 company nodes), absorbed when the company was bought by a filer, not in the registry when a check confirmed it is absent, and unmatched when nobody knows yet (16 nodes). "Checked, and it is not there" and "we don't know" are different facts, and the graph keeps them apart.

The answer layer is where it has slipped. Asked which companies it covers, the chat once listed four while the query underneath had returned all five.