The Smaller Card Won Again

The call that runs before everything else

Most descriptions of a retrieval-augmented site put the language model at the centre, because that is the part that writes sentences. The embedding step gets a clause. It converts text into a list of numbers so that passages can be compared by meaning rather than by keyword, and then the interesting part begins.

That description is accurate and it badly understates the dependency. On this site the embedding call is the first thing that happens to a question and nothing can proceed without it. The question has to become a vector before the database can be asked which passages are relevant. It also has to become a vector before we can check whether we have answered this question before, because our cache matches on the meaning of a question rather than its exact wording, and the key it matches on is the vector.

So the cache cannot save you from an embedding outage. An answer can be sitting in the database, complete and correct, and still be unreachable, because the lookup that would find it needs the call that just failed.

We had a fallback chain four providers deep for generation. For embeddings we had one US provider and no second option. The most load-bearing call in the system was the only one with nowhere to go.

Moving it in-house without rebuilding anything

We run two machines with consumer GPUs, already serving the language models behind the site. The embedding model we pay for, nomic-embed-text-v1.5, is open-weight and about 274 megabytes. There was no technical reason it could not run beside them.

The obstacle was the 57,400 vectors already in our database. Vectors produced by different embedding models cannot be compared to each other, so switching models normally means re-embedding the entire corpus before anything works. We were not switching models, only deployments of the same model, but that is exactly the kind of assumption worth checking rather than believing.

So we embedded the same passages both ways and compared:

The third row is the control. Without it, 0.9999 is a number with no scale attached to it.

Same weights, same vector space. The stored vectors stayed valid, no reindex was needed, and the two sources can even be mixed within a single crawl without splitting the corpus into two coordinate systems that cannot see each other.

Then the benchmark, which went the wrong way again

We have two cards. The sensible expectation was that the larger one would win and the question was only by how much.

Bulk is 200 distinct passages from our own corpus in batches of 50, which is how the crawler sends them. Both cards warmed first, after learning the hard way that a cold model under-reports.

The smaller card won both measurements. It has 40 percent of the memory and 56 percent of the bandwidth of the card it beat.

This is the third time we have measured these two against each other and the third time the result has been counter to the specifications, so at this point the surprise is not that it happened but that we keep expecting otherwise.

The explanation is the same one each time, and it is not that the AMD card is bad. Generating text is memory-bound: every token requires reading the whole model, so a card that moves data faster produces tokens faster, and the 7900 XT duly wins that contest. Our earlier work found it converts about 75 percent of its theoretical bandwidth into tokens where the 3070 manages 89 percent, but 75 percent of 800 still beats 89 percent of 448.

Embedding is not that shape of problem. The model is 274 megabytes. It fits comfortably in cache, there is no token-by-token loop, and each call is a single forward pass. Nothing waits on memory, so the bandwidth advantage never appears and the efficiency gap decides the outcome by itself.

Which is worth stating plainly, because “bigger card, better performance” is a reasonable heuristic that happens to be wrong for this workload: the specification that decides one job on a GPU can be irrelevant to the next one.

Giving the small card a job

The larger card runs the model that writes answers and it will keep doing so — 20GB of memory is exactly what a 13GB language model needs, and the 3070 cannot hold one. The smaller card had been sitting as a fallback that had served no production traffic in months.

Embeddings now go to it first. It is the faster of the two at this particular job, it is otherwise idle, and the model needs 274MB of the 8GB it has.

We expected the stronger argument to be contention — that a full recrawl hammering the same card the site answers from would make the site slow. We measured that too, and it is true but small:

A hundred milliseconds. Real, worth having, and not the reason. We had assumed it would be the headline and the measurement demoted it to a footnote, which is the ordinary outcome of measuring something you were about to assert.

The actual reason is that two cards each doing one job can each cover for the other. Embeddings try the NVIDIA card, then the AMD card, then the hosted provider we used to depend on. The call that had no fallback now has two.

Sixteen seconds of patience in the wrong place

The first live test of that chain worked and took sixteen seconds to do it.

The retry logic was written for a different failure. In September a crawl died forty minutes in because a provider dropped a connection, so retries back off and wait — one second, two, four, eight — on the reasonable theory that something briefly overloaded will recover if given a moment.

A machine that is switched off refuses in a millisecond and will not recover in sixteen seconds. Spending that patience in front of a fallback that is sitting right there, on the one call that blocks every question on the site, is close to the worst place to spend it.

Now patience goes where it pays: two quick attempts when there is somewhere else to go, the full sequence on the last option, where giving up early means giving up. Failover went from sixteen seconds to four.

The bug we would not have found otherwise

Self-hosting has a constraint the hosted provider does not: Ollama caps embedding input at 2,048 tokens, where the provider accepts 8,192.

It does not warn you. Send it 3,000 tokens and it returns a well-formed 768-dimension vector, indistinguishable from any other, built from the first 2,048. That is not a slightly worse match for your document. It is a vector for a different document. We measured a truncated passage against its untruncated version at 0.53 cosine — the same score two unrelated pages get.

Checking how much of our corpus this would affect turned up something we were not looking for. Our chunker is configured to produce passages of about 500 tokens. The corpus contained chunks of 7,239.

The cause was a gap in the splitting logic. Text is divided at blank lines, and any resulting piece larger than the limit was supposed to be subdivided — but a single paragraph with no blank line in it was passed through whole, never split. Toronto.ca has plenty of those: fee tables, eligibility lists, schedules that render as one continuous block.

This had been harmless for as long as we sent those chunks somewhere that accepted 8,192 tokens. It was one configuration change away from silently corrupting a portion of the index, in the specific way that produces no error and no visible symptom — the passages would still be there, still returned by searches, describing content they no longer contained.

Both halves are now fixed: oversized paragraphs are split at sentence boundaries, and the embedding call refuses text over the limit instead of sending it. Notably it refuses rather than quietly using the provider with the larger window, which would work and would hide a chunking bug behind a bill.

We have written before about how the worst bugs here are the ones that produce correct-looking output. An index that had never been used still returned right answers. A GPU benchmark measuring a cold model still produced plausible numbers. A truncated embedding is the same family: everything downstream continues to work, and the only sign is that retrieval gets quietly worse in a way no test asserts against.

What it adds up to

Embeddings were the last inference this project paid for and the last call that left the country. Both are now gone, which matters more to us than the money, since the money was a few dollars across the life of the project.

What we gained that is measurable: embedding is about twice as fast as the provider it replaced and far more consistent — 81 to 87 milliseconds against 172 to 331. Rebuilding the whole corpus takes 17 minutes instead of 38. And the single most load-bearing call in the system finally has a fallback, tested by switching a host off and watching the next one take over.

What we gained that is harder to count: an old bug found, because a new constraint made visible something a generous limit had been hiding. That keeps happening. Constraints are not only costs.

Frequently asked questions

The earlier measurements of these two cards are at Benchmarking Local Inference and My GPU Benchmark Was Measuring a Cold Model. The other bug that produced correct-looking output for months is at Our Vector Index Had Never Been Used Once.