How we build, evaluate, and improve Zeever.ca. We publish our methodology, results, and what we learned along the way.
September 8, 2026
Our Vector Index Had Never Been Used Once
For months every search read all 58,000 rows. The answers were correct the whole time, which is why nobody noticed, and it meant a hosted vector database had been compared against a database with no working index.
September 8, 2026
My GPU Benchmark Was Measuring a Cold Model
We published a comparison of two GPUs running local language models. One finding was wrong, because the first request after a model loads under-reports prompt processing by about 25 percent and nothing warns you.
September 3, 2026
Canada Now Has Two Data Centre Rulebooks. Only One Can Say No.
In three weeks Ontario proposed a screening playbook and Ottawa launched a national framework. They agree on almost everything and can do almost nothing to each other. One gates access to the grid. The other is a set of expectations with no metric, no auditor and no consequence.
August 31, 2026
My Toronto.ca Corpus Contained Other People’s Websites
Toronto.ca redirects some of its own pages to other organisations, and my crawler stored what came back under the toronto.ca address. Fixing that deleted 57 real pages and uncovered 13,188 ghost vectors in the search index. Cleaning up the corpus then moved retrieval accuracy eight points, which is more than seventeen models moved between them.
August 30, 2026
I Benchmarked 17 Models. The Problem Was My Benchmark.
Seventeen models scored within nine points of each other on my Toronto question set. Then I checked the benchmark itself and watched retrieval accuracy fall from 89 percent to 68 to 56. Here is what the measurement was hiding, including an entire section of Toronto.ca I had never crawled.
August 23, 2026
Toronto Is 315 MW: What CBRE’s Data Changed in Our Data-Centre Research
CBRE now reports Toronto and Montreal on the same basis as the U.S. markets. We checked our 11-part series against it, corrected six things, and learned that most data-centre arguments are really arguments about definitions.
August 12, 2026
AI Isn’t the Next 2008. But Its Financial System Is Starting to Rhyme.
Data-centre securitization went from $4 billion in 2020 to roughly $61 billion today, and SEC staff have now said qualifying versions are not legally asset-backed securities. This is not subprime. The risk is opacity.
July 31, 2026
Turbopuffer vs pgvector: Equal Quality, and a Speed Claim We Got Wrong
We benchmarked Turbopuffer against our pgvector setup on 100 Toronto questions. Answer quality was a tie, and still is. The 20x speed advantage was not: our pgvector index had never been used, so the comparison measured an unindexed table.
July 28, 2026
Does Proprietary Beat Open? Testing Cohere Command on City RAG
We put Cohere's three Command models against our open Qwen2.5-7B baseline on 100 Toronto questions, scored by a neutral judge. The open model held its ground.
Series · 11 chapters · July 23, 2026
Canada's Data Centre Race
An 11-part research series on the physical foundations of the AI buildout in Canada: data centres, power, fibre, water, sovereignty, jobs, capital, and the alternatives. Grounded in primary sources, with every figure cited.
May 2, 2026
Mapping Canadian AI Compute: Why We Built the Zeever Compute Index
A verified inventory of 39 Canadian GPU providers, normalized on H100 USD/GPU·hr against a $7.50 ceiling, with a sovereignty taxonomy that makes the procurement trade-off legible.
April 23, 2026
The Shift to Agent-First AI: What Model Deprecations Tell Us
AI inference platforms are deprecating chat-first models and replacing them with agent-first MoE architectures. What this shift means for production AI.
April 17, 2026
Running Ollama on an Old Dell XPS With an NVIDIA 3070
We tested local inference with Ollama on a Dell XPS with an RTX 3070 against Fireworks, Together.ai, and OVHcloud. The old desktop held its own.
April 3, 2026
Switching Inference Providers: A 24-Hour Latency Test
We ran a 24-hour latency test across Fireworks, Together, and OVHcloud. Together.ai was 1.7x faster. Here is why we switched.
April 2, 2026
The Missing Layer: AI Inference in Canada
Canada has GPU hosting but no easy way to test and prototype open-source models. That gap is pushing Canadian companies toward US-hosted black boxes.
March 31, 2026
100 Questions Across Toronto.ca: Building a Comprehensive Benchmark
We expanded our evaluation from 12 building permit questions to 100 prompts across 16 categories covering all of Toronto.ca.
March 30, 2026
Scaling a RAG Pipeline from 174 Pages to 35,000 Documents
Our database hit 25GB, the server crashed from OOM, and the web admin went dark during crawls. Here is how we fixed all three.
March 28, 2026
How We Built an LLM-as-Judge to Replace Keyword Scoring
Keyword matching scored our best answer at 0.00. The LLM judge scored it 1.00. Here is how we built a semantic evaluation system.
March 28, 2026
Comparing 7 Open-Source Models for RAG on City Data
We tested 7 open-source LLMs on Toronto city services questions. The cheapest model won.
March 27, 2026
Fixing Vector Search: Probes, Chunking, and Classification
Three retrieval fixes that improved citation accuracy by 35% and fixed queries that returned zero results.
March 27, 2026
Vector RAG vs GraphRAG on Toronto City Data
We compared plain vector retrieval against graph-enhanced retrieval. Graph mode helped on some queries and hurt on others.
Methodology notes
- All evaluations use temperature=0 for deterministic output.
- The judge model (GPT-oss 120B via Fireworks.ai) is the same model used for answer generation. We acknowledge the potential for self-evaluation bias.
- The 100-prompt benchmark suite covers 16 categories across all sections of Toronto.ca.
- Groundedness scoring evaluates whether claims are supported by retrieved evidence. It does not verify that the evidence itself is current on Toronto.ca.
Code and data
The evaluation framework, benchmark prompts, scoring code, and model comparison scripts are available on request. Contact us at infozeever [dot] ca.