We Benchmarked 17 Models. The Problem Was Our Benchmark.

Zeever answers questions about City of Toronto services using Toronto.ca content. Every answer has to come from retrieved evidence, and the system is supposed to say so plainly when the evidence runs out. That second rule is the hard one, and it is the one this work turned out to be about.

Two GPUs, one lesson about bandwidth

The local side of the comparison runs on two machines. A Dell XPS with an NVIDIA RTX 3070, 8GB, and an HP Z640 workstation with an AMD Radeon RX 7900 XT, 20GB, running ROCm. Both reachable over Tailscale from the VPS that serves the site.

We expected the AMD card to win comfortably. It has 800 GB/s of memory bandwidth against the 3070’s 448, and token generation is bandwidth bound: the model weights are read once per token, so throughput is bytes moved divided by bytes per token. On paper that is a 1.8x advantage. We measured 1.12x.

Same model, same quantization, same prompt, cold cache on both. Efficiency is measured throughput against the card’s theoretical bandwidth ceiling.

The AMD card also loses on prompt processing, which is compute bound and where it should dominate. That points at software rather than silicon. The HIP backend in llama.cpp reuses kernels written and tuned for NVIDIA hardware, and RDNA3 does not get the same attention. We tried the obvious remedies. Flash attention, quantized KV cache, and pinning models resident moved decode from 56 to 53 tokens per second on one model, which is to say they did nothing. We checked the card was not throttling by sampling its clocks during a generation, and it was not: memory pinned at 1,249 MHz, 250 watts, 63 degrees.

What did work was changing the model. An MoE model activates a fraction of its parameters per token, so it moves fewer bytes, so it decodes faster even when it is much larger.

The 20 billion parameter model is the fastest of the three, and it beats a 7 billion parameter dense model on the same card. The 20GB of VRAM on the AMD workstation is what makes that possible. So the card earns its place, just not for the reason we bought it. Its value is capacity, not speed.

Seventeen models, nine points apart

With the local side sorted we ran the full comparison: four Together models, four from Cohere, three from Canadian provider Augure, and six local models across the two Ollama boxes. One hundred questions each, 1,700 generations, judged by Llama 3.3 70B. Total cost of the paid arms was $3.80.

Selected rows from a seventeen model run. Augure prices are converted from CAD. Costs are per hundred questions.

Nine points separate the best model from the worst. Drop the single weakest and it is five points across sixteen. Every model was reading identical retrieved chunks, because we precompute retrieval once and share it across the run, so this is a clean comparison of what each model did with the same evidence. They all did about the same thing.

That is a useful result even though it is a boring one. If model choice moves the answer by five points and costs range from free to $1.15 per hundred questions, the interesting decisions are elsewhere. Cohere’s command-a-plus is the clearest illustration: the most expensive model in the run, and the weakest of every paid model on judged relevance.

The Canadian provider held up. Augure’s tofino-3 landed at 0.950 relevance and 0.985 groundedness for sixteen cents per hundred questions, and rosedale-1 posted the highest groundedness of anything we tested at 0.995. Both are slower than the hosted American options, and ossington-5 was the weakest model in the field, so the lineup is uneven. But sovereignty did not cost quality, which is the thing we wanted to know.

The judge was scoring honesty at zero

One question in the set asks how to set up a new water account in Toronto. Two models said the retrieved evidence did not cover it. A third produced a confident five step procedure. The judge scored the invention at 1.00 relevance and the honest answers at 0.50.

The rubric was doing what we told it to. It ended with the line 0.0 = does not answer the question or says it cannot help, which on a question the corpus cannot answer scores fabrication at one and honesty at zero. For a product whose entire premise is that answers come from cited Toronto.ca content, that is the wrong way round.

We fixed the ranking first, by scoring relevance multiplied by groundedness and counting a quadrant we started calling confident plus: answers given confidently from sources that do not support them. That reordered the table immediately. Together’s gpt-oss-20b, top of the relevance ranking, had seven such answers, the most of any model, and fell to tenth. Augure’s rosedale-1 had none.

Reading those answers rather than counting them showed the pattern. The models were answering from what they knew about Toronto instead of from the retrieved text. One told us potholes are repaired within four days on busy roads and thirty days on side streets, which is a real City service standard, correctly recalled, and absent from every chunk it was given. Correct by luck is indistinguishable from wrong by luck until it is wrong.

So we rewrote the system prompt to name that failure. The evidence rule moved from one bullet among twelve to its own section near the top, it now tells the model that recognising an answer from training does not make it evidence, and it lists what has to be checked: phone numbers, fees, deadlines, form names, numbered steps. It also says plainly that a well supported question should be answered in full, because the obvious way to lose is trading invention for hedging.

The bottom row is the one that mattered. Buying grounding with refusals would have shown up as more answers with neither usefulness nor support, and that number went down. Five of the seven answers that left the risky quadrant kept full relevance while their groundedness went from 0.50 to 1.00. The models did not start declining. They kept answering and started doing it from the evidence.

Then we checked the benchmark

Along the way we had been quoting a retrieval accuracy of 89 percent, meaning that for 89 of 100 questions the page we expected appeared in the five chunks the app reads. The check is a substring: does any returned URL contain the expected fragment.

We finally counted how many pages each of those fragments actually matches. One question expected the word water, which matches 431 pages in our corpus. Another expected community-people, which matches 1,645. Nineteen of our hundred questions were scored against something specific enough to mean anything. The other 81 were being graded on whether retrieval stayed in the right neighbourhood.

Retrieval did not get worse between those three rows. We stopped grading ourselves generously. And the loose version was not merely flattering, it was actively concealing. The question about reporting a water main break scored a retrieval hit on every single run, because its expected fragment was water, while the page that answers it has never been in our index at all.

The section of Toronto.ca we had never crawled

Three questions kept failing in a way that looked like the City simply does not publish an answer: opening a water account, reporting a water main break, and when street cleaning happens. We had gone as far as marking them in the benchmark as questions where declining is the correct response.

That was wrong, and the way we found out is worth recording. Every diagnostic we had ran inside our own index, so it could only ever tell us what our index contained. Searching Toronto.ca directly found all three answers immediately, in the 311 knowledge base, including the street sweeping schedule we had concluded did not exist: arterial roads twice a month, collector roads once a month, local roads every two months.

Toronto.ca publishes a sitemap. It lists eight WordPress page sitemaps and nothing else, and robots.txt declares no other. The 311 knowledge base is a separate application, so a sitemap driven crawler cannot see it. Our URL filter was not excluding it. We were simply never told it was there. And it is precisely where the City puts its how do I report this content, which is the most common shape of question a service assistant gets.

A benchmark too loose to fail let a missing content section read as healthy retrieval, and we then treated our own silence as evidence about the world. That is the mistake we would most like not to repeat.

What we changed, and why

The rule we settled on partway through, and which we would now apply from the start: a retrieval change ships when a mechanism explains it, not because a benchmark moved. Every row below has a mechanism.

Hybrid search is the instructive rejection. It was already written and had never been called, so wiring it up felt like free improvement. At five chunks it scored 86 against dense search’s 89. At ten chunks it scored 93, the highest raw recall of anything we tested, which was tempting. But it damaged the same category of questions in every single run, and no mechanism explained why it should help. We left it behind a flag, turned off, with the evidence attached.

What is still wrong

The corpus wide sample gave us something the curated questions never could. Recall by section:

HTML retrieval is fine. The 56 percent headline is dragged down by PDFs, which are 55 percent of the sample and the majority of our corpus: 10,800 parsed PDFs against 7,700 HTML pages. Some of them are genuinely useful and retrieve badly, with titles like 8Ccb Monthlyutilitybillingapplication derived from filenames. Others are engineering specifications, council background files and municipal code chapters that no resident would ever ask about, where the eval invents a question and then correctly fails to find it.

Our own gap analysis has been listing 13,695 uncrawled PDFs as outstanding work. On this evidence, fetching them would roughly double the worst performing content class while the section people actually ask about stays missing. That decision now needs numbers behind it rather than a completeness instinct.

Three things are queued. Crawling the 311 knowledge base, which needs link following the crawler does not have. Narrowing the 81 loose expectations so the benchmark can see improvement at all. And working out which PDFs belong in a resident facing corpus. We will keep the corpus wide sample running alongside the hundred questions, because a hundred precise targets is exactly the kind of thing a system can be tuned to hit while getting worse everywhere else.

The models were never the problem. Seventeen of them, from three continents and two price brackets, agreed with each other to within a few points. What separated a good answer from a fabricated one was whether the right page was retrieved and whether the prompt told the model that remembering is not the same as knowing. We spent a week learning to measure those two things, and we are still not finished.