How to test a knowledge graph

Short answer

Test a knowledge graph with three kinds of queries, run after every load. Competency questions have a known answer you can check against a source outside the graph. Invariants are patterns your domain forbids, written as queries that must always return zero rows. Count tripwires compare node and relationship counts with the previous load. Neo4j constraints sit underneath for uniqueness and required properties. A passing suite only rules out the errors you thought to test for, so add a new test for every bug you find.

Three kinds of tests

Test	What it is	Software equivalent
Competency question	A question with a known, human-verifiable answer. The graph must reproduce it.	Acceptance test
Invariant	A pattern the domain forbids. The query must return zero rows.	Assertion
Count tripwire	Node and relationship counts compared with the previous load.	Regression test

Database constraints sit under all three. Let Neo4j enforce uniqueness and, on Enterprise Edition, property existence and property types (Community Edition only has uniqueness constraints). Many rules that matter can't be expressed as constraints at all: ownership cycles, parts that must not exceed a total, facts that contradict each other over time. Those live in the test suite.

Competency questions

A competency question is a question your graph must answer correctly, written down before you trust the graph. The idea comes from ontology engineering (Grüninger and Fox, 1995), and in practice it works as an acceptance test. "Which products containing flufenacet were authorised for winter wheat before the 2025 withdrawal?" has a verifiable answer in the Polish plant protection register, in the version the graph was loaded from. If the graph returns something else, the graph is wrong.

Good competency questions have three properties:

• The answer exists outside the graph, in a document, a register or a person's head, so you can check it.
• They come from real users. Questions people asked last month are better than questions the builder finds interesting.
• They cover the hard parts: multi-hop paths, time ("what was valid on 1 March?"), and entities that are easy to confuse.

The same questions do double duty. Before the build they shape the ontology (see Reuse, extend or build an ontology: how to decide). After it they are the acceptance criteria for a pilot.

Invariants

An invariant describes a state that must never exist. The one I'd write first on any graph with ownership or containment:

// No company may own itself, directly or through intermediaries.
MATCH (c:Company)-[:OWNS*1..10]->(c)
RETURN DISTINCT c.name

The depth bound keeps the query cheap on a large graph; raise it if your ownership chains are longer. In a tree of controlled subsidiaries, like the one in a 10-K's Exhibit 21, a cycle is almost always an entity-resolution bug: two different companies merged into one node. It breaks variable-length queries downstream, so catching it at load time is cheap insurance. If your graph also models minority cross-shareholdings, some cycles are real, and the rule needs a threshold such as "controlling stake only". Other examples: every filing has exactly one filer, segment revenues reconcile with the reported total, no date lies in the future.

Wiring invariants into CI takes one line per file: run cypher-shell --format plain -f invariants.cypher and fail the build on any output. With plain format, a query that returns zero rows prints nothing.

Count tripwires

Compare counts per label and relationship type with the previous load and flag changes outside an expected range. A tripwire tells you something changed, not what. It is the cheapest way to notice that a source file came in half empty or a parser silently skipped a section.

What tests don't catch

A green suite means the graph is not wrong in the ways you thought to forbid. Some failures sit outside that.

Missing facts in LLM-extracted graphs. If an extraction model silently dropped a third of the relationships, every invariant still passes. Extraction recall needs a labelled sample and statistical evaluation, which is a different tool for a different failure.

Rotting questions. When the schema changes and a competency question starts failing, the tempting fix is to update the expected answer to match the new output. Check it against the source first, or the test turns into a snapshot of whatever the graph currently says.

Gaps in domain knowledge. Writing a good question takes more domain knowledge than Cypher. On the plant-protection graph, the hard part of every test was knowing what must be true, and that part is agronomy.

Example from a working graph

On the Alphabet seed graph that later grew into the 10-K graph, an invariant that segment revenues add up to the reported total fired on correct data the first time I ran it. For FY2025, Alphabet's segments sum to $402,963M against a reported $402,836M, because $127M of hedging losses are netted at the total line. The data was right; my model had no place for the hedging line. My first draft of that check allowed a 0.1% "rounding tolerance", which would have hidden the $127M difference completely. The filing reports whole millions and its table reconciles exactly once the hedging line is included, so the final check uses no tolerance at all.

Another invariant, "every Company node has a ticker", held for the five hand-picked filers and broke as soon as unresolved competitor mentions from the text were merged in. That one bug produced four new invariants. The rule I work by: one new invariant for every bug found, for as long as the graph lives.

The full walkthrough is in How I Test a Knowledge Graph, and the seed graph with its test suite is on GitHub.