Turbopuffer vs pgvector: Equal Quality, and a Speed Claim We Got Wrong

Our retrieval runs on Postgres with the pgvector extension, and it works. But Postgres sits on a shared VPS that keeps running low on memory, and the vector index is one of the heavier tenants on that box. So we asked a simple question: could a purpose-built vector database do the same job faster, cheaper, and off the server that keeps running out of RAM?

We tested Turbopuffer, an object-storage-native vector search engine, against our pgvector setup. Same 100-question benchmark across 16 categories of Toronto.ca, same embeddings, same generation model, scored by a neutral judge from a different model family. Six configurations in all: pgvector and Turbopuffer, each with and without our graph layer, plus a Turbopuffer hybrid mode that adds keyword search. Retrieval was the only thing that changed.

Results

Judge scores are averages over 100 questions, each from 0 to 1. Latency is the median retrieval time. “Same chunks” is the overlap with what pgvector retrieved for the same question. The pgvector latency is a sequential scan, for the reason given in the correction above; the quality columns are unaffected, because a sequential scan is exact search and returns the right rows, just slowly.

The short version

Quality was a tie. Every vector configuration scored between 0.93 and 0.97 on every judge dimension, close enough to sit inside the judge’s own margin of error, and Turbopuffer vector search returned 97 percent of the same chunks as pgvector. Both backends were finding the same evidence. That part of this article has held up under everything we have measured since.

The speed half was wrong, and it was wrong in an instructive way. We were not comparing two vector indexes. We were comparing a vector index against a Postgres table with no usable index at all, which answers every query by reading all 58,000 rows and comparing each one. That is exact search, so it returns the right chunks, which is why nothing in the quality columns looked unusual and why the error survived for two months. It is simply slow.

Measured properly, the gap is roughly 35 milliseconds against 28. Turbopuffer is still ahead at the tail, where its 95th percentile is about four times tighter, and it still runs off a server that has too little memory. Those are real reasons. A 20x speedup was not one of them.

Two numbers for pgvector appear in this article and they measure different things. The database executes the search itself in about 5 milliseconds, which is what Postgres reports for the query plan. The application sees about 35, because it also encodes a 768-dimension vector into the request, waits on a connection from the pool, joins the chunk back to its page and reads the text out. We quote the 35, because that is the number a person waiting for an answer actually pays, and it is what the 28 for Turbopuffer is measured the same way against.

The graph added nothing

We layer a knowledge graph on top of retrieval, pulling in entities and relationships connected to the retrieved chunks. Turning it on moved the average scores by about a hundredth of a point, well inside the noise, on both pgvector and Turbopuffer. That matches an earlier finding of ours, but this time we went further and sliced the scores by category to check the building-permit and process questions, the exact place an earlier test had seen the graph help. It did not help there either. Building-permit questions scored 0.979 with and without the graph.

The reason is encouraging: our plain retrieval has improved enough since that earlier test that there is no gap left for the graph to fill. When the baseline was weaker, structured graph context helped patch the holes. Now the holes are mostly gone, and the graph is carrying cost (one extraction pass per chunk) and latency (a graph lookup per query) for no measurable gain.

Hybrid search made citations worse

Turbopuffer also does hybrid search, combining vector similarity with keyword (BM25) matching. On our content it was a net loss. Citation quality fell from about 0.955 to 0.86, groundedness slipped, and it overlapped the pure-vector results by only 36 percent. The keyword half kept pulling in chunks that matched the words in the question but did not cite an official source cleanly. For a system whose whole promise is that every answer points back to Toronto.ca, that is the wrong trade. We stayed with pure vector.

Cost, and where the data lives

At our scale, tens of thousands of chunks, Turbopuffer usage is small enough that the launch-tier minimum of about 16 US dollars a month is effectively the whole bill. pgvector has no separate invoice, but it costs memory and disk on a VPS that was already running out of both. So the honest framing is not dollars against dollars. It is roughly 16 dollars a month to move vectors off a memory-starved box and get worst-case latency about four times tighter. When this article was written we thought we were also buying a 20x speedup. We were not, and the money is still defensible without it, which is worth saying plainly rather than quietly restating the price.

There was one more reason we were comfortable making the move. Turbopuffer runs a Toronto region, so the vectors and their text stay on Canadian soil. Canadian data residency is a long-term goal for this project, and it is unusual to get that from a hosted vector database.

How we shipped it

We did not flip a switch in production. We staged it. New content now writes to both stores at once, so pgvector stays current as a fallback. Reads are gated by a config flag that defaults to pgvector, and the Turbopuffer path falls back to pgvector automatically if anything goes wrong. Once we flip the flag and watch the latency drop with no fallback errors, Turbopuffer is live, and pgvector is still there for an instant rollback. Only later, once we trust it, do we drop the pgvector index and actually reclaim the memory. That last step is where the server pressure finally eases.

We have since abandoned that last step. The plan to drop the pgvector index was written when we believed pgvector was 20 times slower; it was really an unindexed table. A working pgvector index costs 38 MB, and it is the only path that keeps this capability on hardware we control. Deleting it would have removed the sovereign option at the moment it became viable.

Methodology

Six retrieval configurations ran against the same 100 benchmark questions. Every configuration used the same Nomic query embedding, the same top-5 chunk limit, the same generation model (Qwen2.5-7B), and the same judge (Llama-3.3-70B, from a different family than any model under test, to avoid self-scoring bias). We mirrored the exact pgvector chunks into Turbopuffer and reused their existing embeddings, so nothing about the content or the vectors changed between backends. Retrieval was the only variable.

Limitations

The quality differences between configurations are small, often a hundredth or two, which is inside the noise of a single judge. We would not read the exact ranking as gospel. One finding is robust and has survived everything we have measured since: none of the alternatives beat pure vector on quality. The speed finding did not survive, for the reason given in the correction above, and at the median the two backends are close to a wash. One more signal worth naming: many categories scored a perfect 1.000 on both backends, which means the benchmark is getting close to too easy for these models. That caps what any retrieval change can demonstrate, and it tells us the next useful step is a harder set of questions.

Frequently asked questions