GraphRAG vs vector RAG: when a graph helps

Short answer

Use vector RAG when the answer to a question sits in one or two passages of text and the job is to find them. Add a knowledge graph when the answer depends on connections spread across many documents (who supplies whom, which product contains what, what changed between two years), when you need complete lists rather than the top few matches, or when every answer must point to its source.

Independent benchmarks from 2025 and 2026 back this split, with caveats. On single-fact questions, plain RAG with a reranker ties or beats graph-based retrieval. On multi-hop questions graphs usually win, though the margin shrinks when the vector side gets a reranker or multi-step search. The best results come from hybrids that keep the original text next to the graph. So the choice is rarely either-or. Make it by testing your own questions.

What each one does

Vector RAG splits documents into chunks, turns each chunk into an embedding, and at question time retrieves the chunks closest to the question. The model writes an answer from those chunks. It is quick to build and works well when the question and the answer use similar language.

GraphRAG uses a graph to decide what the model reads. The name covers three different things:

• LLM-extracted graphs with community summaries (Microsoft GraphRAG, LightRAG): an LLM extracts entities and relationships, clusters them and summarises each cluster. Good for "what are the main themes across all of this".
• Passage graphs (HippoRAG 2, RAPTOR): text passages linked through shared entities or summaries. The text stays the unit of retrieval.
• A curated knowledge graph queried directly: entities and relationships from structured sources or checked extraction, each with properties such as dates and sources. The question becomes a query (in Neo4j, Cypher) and the model answers from the rows that come back.

Most academic benchmarks test the first two. Business questions about complete lists, dependencies and sourced facts usually need the third, often with vector search to find the starting point. Keep that in mind when reading the numbers below.

What the research says (2025-2026)

Grouped by question type (Microsoft's own numbers are marked as such):

Question type	What wins	Evidence
Single fact, one passage	Vector RAG ties or wins	Natural Questions: RAG 64.8 F1 vs 63.0 for graph local search (Han et al.). Fact retrieval on novels: RAG with reranking 60.9 vs 49.3 for Microsoft GraphRAG (GraphRAG-Bench, ICLR 2026)
Multi-hop	Graph, by a margin that varies a lot	HotpotQA: HippoRAG 2 63.0 vs RAG 60.0 (Han et al.). Across three multi-hop sets with single-pass retrieval and a 7B model, the best of five graph variants beat dense retrieval by 27 points on average, and multi-round agentic search narrowed that gap (RAGSearch, 2026)
Change over time	Graph over plain RAG; a reranker closes most of the gap	Temporal questions in MultiHop-RAG: vanilla RAG 30.7 vs 50.6-53.3 for community-summary GraphRAG. With a reranker, RAG reached 49.6 and the reranked graph variants 53.5-54.0 (Han et al.)
Summaries across a whole corpus	Graph with community summaries	Summaries over novels: Microsoft GraphRAG 64.4 vs 51.3 for RAG with reranking (GraphRAG-Bench). Microsoft reports 72-83% win rates on comprehensiveness (Edge et al.), judged by an LLM
Questions with no answer in the data	Vector RAG	RAG correctly declined 96.0% of the time, graph global search only 19.3%, answering the rest anyway (Han et al.)
Counting, "list all", top-k	Neither top-k method	On counting, extremum, sorting and top-k questions over a whole corpus, the strongest baseline scored 1.5 F1; HippoRAG 2 was among those tested (GlobalQA). Retrieving k similar chunks cannot return a complete list

Three findings matter more than any single score.

Hybrids beat pure graphs. In Han et al., retrieval from extracted triples alone scored 34.3 F1 on Natural Questions, against 64.8 for plain RAG, and merging vector and graph results beat the best single method by up to 6.4%. A peer-reviewed comparison of 12 graph RAG methods (Zhou et al., PVLDB 2025) found that methods relying only on graph structure do poorly on document QA; its strongest methods, RAPTOR and HippoRAG, hand the model text rather than bare facts. For a curated graph the lesson is the same: keep the source passage, or a pointer to it, with every fact.

Reported win rates are inflated. Much of the GraphRAG literature scores answers by asking an LLM which of two is better. Zeng et al. showed that swapping the order of the answers moves win rates by more than 30%, and a 25-token length difference by more than 50%. With those biases controlled, LightRAG's reported 72% vs 28% win over plain RAG became a slight loss.

Generic, schema-free extraction is lossy and expensive. In Han et al. the graphs were extracted by GPT-4o-mini, with no domain schema and no entity-resolution step described, and only about 65% of the entities needed for the answers made it into the graph. Building the MultiHop-RAG index took 5,560 to 7,702 seconds for the graph variants against 135 for the vector index, and GraphRAG-Bench measured prompts of about 900 tokens per question for vanilla RAG against about 39,000 for Microsoft GraphRAG local search. Microsoft says its LazyGraphRAG brings indexing cost down to that of vector RAG (Microsoft Research), but in GraphRAG-Bench it still scored below plain RAG on fact retrieval (51.7 vs 60.9).

None of this says a graph can't be built with language models. It says that pointing one generic extractor at a folder of documents gives you a graph of unknown completeness. The builds that hold up treat extraction as a pipeline with its own tests: a schema designed from the questions the graph has to answer, structured sources parsed without a model, smaller named-entity and relation-extraction models for the bulk of the text, an LLM for the hard cases and for deciding when two names are the same thing, and a coverage check against questions with known answers (see How to test a knowledge graph). Cheaper extraction can be competitive: in one 2025 preprint, a graph built with classic dependency parsing reached 94% of the performance of an LLM-extracted one (Min et al.).

What the Neo4j-funded NICD study found

The UK's National Innovation Centre for Data compared an agent using vector search alone with one using vector search plus a graph, on 510 questions from the MoNaCo benchmark (paper). Neo4j funded the study, and both the paper and Neo4j's summary say so. The summary says the graph agent "successfully answered over 65.3% of the questions". The paper, scored with an LLM judge, is narrower:

Share of all 510 questions	Vector only	Vector + graph	No retrieval
Attempted	28.1%	65.3%	not stated
Fully correct	11.4%	28.5%	34.1%
Answer with at least one wrong claim	16%	37%	59%

65.3% is the share the graph agent attempted; it got 28.5% fully right. The model with no retrieval produced the most fully correct answers and gave wrong ones to 59% of the questions. The graph was Wikipedia's own structure of articles, sections and links, not an extracted knowledge graph. With the graph the agent attempted far more questions than with vector search alone, but more of its answers contained a wrong claim than were fully correct. Every number in this field needs its denominator.

Where vector RAG is enough

• "What does the document say about X?" Policy lookups, manual search, FAQ-style support. The answer is a passage.
• Summaries of one document or a small set of documents.
• Exploratory questions where a reasonable, sourced paragraph is good enough and nobody needs an exact list.
• Early prototypes. If you don't yet know which questions matter, vector RAG is the cheaper way to find out.

If your RAG system fails on questions like these, the fix is usually better chunking, hybrid keyword and vector search and a reranker, not a graph. On these questions the GraphRAG-Bench authors found that extra graph context tends to add noise.

Where a graph pulls ahead

• Multi-hop questions. "Which of our suppliers depend on the same sub-supplier?" No single chunk contains that answer, so similarity search has nothing to find.
• Complete lists and counts. "Every product containing substance X." Vector search returns the top few similar chunks. A graph query returns all matching rows, and you can check the count against the source.
• Change over time. "Which risk factors are new this year?" needs two versions of the same thing connected. Against plain vector RAG the gap is large, though a reranker closes much of it.
• Answers that must show their source. When each relationship stores the document and section it came from, a query that returns those fields brings the evidence back with the answer. See How to make AI answers auditable with provenance.

The weak spot: turning questions into queries

A curated graph answers only as well as the query it is asked. On CypherBench, with large real-world graphs, the best models of late 2024 produced a query returning the correct result about 61% of the time, and models under 10B parameters stayed below 20%. Over 94% of the best models' queries ran without error, so most failures were valid Cypher matching the wrong part of the graph. A 2026 preprint reports that sending failed queries back for a retry recovered 91.7% of first-pass failures (LAST-CQ), though a query that runs is not the same as a correct answer.

What works in practice: tested, parameterised queries for the question types you know users will ask, free generation only for the rest, the schema and a few examples in the prompt, and an empty or suspiciously small result reported as such rather than papered over by the model.

What about long-context models?

Models that read a million tokens make it tempting to skip retrieval. The evidence is mixed: one study found long context generally beats chunk-based RAG on QA benchmarks (Li et al.), another found no clear winner (LaRA), and Chroma, a vector database company, found performance getting less reliable as input grows (Context Rot). Long context also does not give you complete lists over a corpus that doesn't fit, or a source for each fact.

How to decide on your own data

1. Collect 20 to 30 real questions from the people who will use the system, with correct answers written down by someone who knows the domain. Include a few whose honest answer is "the data doesn't say".
2. Tag each question: single passage, list or count, multi-hop, time comparison, or no answer.
3. Run them through a plain vector RAG baseline with a reranker. It takes a day or two to build and gives you a number to beat.
4. Score each answer: attempted, correct, complete (for lists), sourced, and correctly declined. Score against the written answers, not with an LLM judge alone; if you use one, randomise the answer order.
5. Look at where it fails. If failures cluster on lists, multi-hop and time questions, a graph is likely to help. If they are spread evenly or come from bad source data, a graph won't fix them.
6. Record cost and latency next to accuracy. Two points of accuracy at forty times the index cost may not be the right trade.

If most of your questions are single-passage lookups, stop at step 3 and improve the RAG. When not to build a knowledge graph covers the other simpler options.

Example from a working graph

The 10-K graph answers questions over 25 annual reports from five large companies. "Which companies name each other as competitors" or "who declares a dependency on TSMC" are graph questions: the answer is a set of relationships extracted from different filings, each carrying the filing and section it came from. Of 239 declared competitor relationships, only one pair (Alphabet and Microsoft) name each other. You would not find that by reading the most similar paragraphs.

The same project shows a limit. Asked which companies it covers, the chat listed four and left out Amazon, although the query underneath had returned all five rows. A graph makes answers checkable; it does not make the language model that writes them infallible, so the answer layer needs its own tests.