Vector RAG vs GraphRAG on Toronto City Data

Once the baseline vector RAG system was working, we layered a knowledge graph on top. The graph pulls entities (permit types, documents, requirements, inspections) and relationships (requires, followed_by, costs) out of the content using LLM extraction. The question: does graph context actually improve answers?

How graph mode works

In graph mode, after retrieving chunks by vector similarity, we also query the knowledge graph for entities connected to those chunks. If a chunk mentions a permit type, we pull in all related requirements, steps, fees, and inspections from the graph. That structured context goes into the LLM prompt alongside the raw chunk text.

Results

We ran the same benchmark in both modes. Graph mode helped with some structured questions about requirements and processes (bp-02 documents went from 0.33 to 0.67 relevance, bp-06 inspections from 0.67 to 1.00). But it made others worse (bp-08 apply online dropped from 1.00 to 0.33) where the extra context distracted the model from a straightforward answer.

On average, the scores were nearly the same. The wins and losses cancelled out.

What we concluded

GraphRAG works best for certain query types (process workflows, requirement chains) but not as a blanket improvement. The cost of graph extraction (LLM calls per chunk) and the added latency from graph queries make it hard to justify as the default. We kept vector retrieval as the default and left graph mode available via the ?mode=graph parameter.

Graph stats

The knowledge graph contains 2,716 nodes and 2,860 edges extracted from guide and inspection content. Node types include permit_type, document, step, requirement, fee, inspection, review_stream, and department. Edge types include requires, followed_by, costs, inspected_by, and applies_to.