HTTP 200, one megabyte, and a casino

I run a crawler that reads public company registries in 93 countries. Once a month I sweep all of them and look at what comes back. The sweep is boring by design: 92 countries measured, a keyword and a city per country, count the rows.

This time three countries came back with nothing at all: Nigeria, Kenya, Ghana. Zero rows, no exception, no timeout, nothing in the log. So I went and looked at what was actually behind the zero.

1. The registry that became a casino

Ghana's company registry endpoint, orc.gov.gh, was wired into my Ghana adapter as the official source. I requested it by hand. It answered HTTP 200 with 1.0 MB of content.

The content was an Indonesian gambling site. The title was a slot machine promotion. The domain had been hijacked, and the registry role was simply gone.

My code had been fetching it happily for weeks. The parser looked for table rows, found none, and returned an empty list. The layer above recorded the fetch as successful and attached a confidence of 90, because the confidence was computed from the transport result and not from the content. A 200 arrived, a megabyte arrived, so the source was healthy.

The only reason no casino spam ever reached a customer is that the hijacked page happens to be structured differently from a registry table. If the page had contained anything my parser recognised as a row, I would have shipped it as a Ghanaian company. That is a coin flip, not a safeguard.

2. Empty is not the same as broken, except to the customer

The shared entry point every country goes through looked roughly like this:

try:
    return enforce_location(await wait_for(primary(), timeout=budget))
except asyncio.TimeoutError:  ...fallback...
except Exception:             ...fallback...

The fallback chain ran on a timeout and on an exception. It did not run on a successful empty result, because a successful empty result is not an error, so it was returned as-is.

To the person using the product, "the source did not respond" and "the source responded with nothing" are the same event: zero companies. Measured on the same queries, the country's own chain returned 0 rows while the shared fallback returned 4, 7 and 10 rows in about 1.5 seconds. The data was one branch away the whole time.

3. The empty page with no error at all

This is the one I would not have found by reading code.

aiohttp advertises br in Accept-Encoding as soon as a brotli package is importable in the process. If the response then comes back brotli-encoded and decoding fails, you get ClientPayloadError: Can not decode content-encoding: br inside the body read. Depending on how the call site handles it, the page arrives empty, with nothing useful in the log.

Measured against one directory page, same URL, same process, aiohttp 3.13.5:

The nasty part is the blast radius. I had written a compatibility shim for this months earlier, but it was wired into the search stack only. Any process that did not import the search stack, meaning my own sweep, my tools and every probe I used to diagnose the problem, was silently reading empty pages. My diagnostic instrument had the bug I was diagnosing.

If a source of yours "does not respond" but curl gets the page, check this before you touch a timeout.

4. The bug whose fix would have been the real disaster

Fourteen adapters called a European business directory with URLs that all returned 404. An obvious cleanup: fix the URLs.

Before doing that I measured what the working URL actually returns. I requested the country filter for Nigeria and then for Kenya. The two responses were byte for byte identical: 30 companies, of which 17 German, 5 Italian, 3 Austrian, 2 Spanish, and one each Bulgarian, Swiss and Turkish. Zero from the country I asked for.

The filter did nothing. Those 404s were the only thing standing between my pipeline and German companies being served as Nigerian leads, with a plausible name, a real address and a working phone number. The tidy fix would have converted a visible zero into an invisible lie.

A visible zero is a bug report. A confident wrong answer is a customer telling their prospect they read something about them that was never true.

What I changed

A shared hijack gate that runs before parsing, not after. It looks for spam markers in the title or at least two distinct markers in the body, deliberately strict so a genuine directory that happens to mention the word casino is not flagged. A domain that trips it is marked failed for the rest of the run, using the same memory as a 403 or a 429.

Beyond that: Accept-Encoding pinned to gzip, deflate across all 93 countries. An empty result now falls through to the fallback chain instead of being returned. The country filter is enforced on my side by reading the country flag off each row, because the upstream filter cannot be trusted. Each of these has a test that fails against the old code, which is the only version of "fixed" I believe.

The one remaining empty is Malta, and it is empty on purpose: I asked for Valletta, the source returned Maltese companies from other cities, and the geo gate dropped them. An honest zero beats a nearby city.

The measurement was measuring the wrong thing

One more, because it is the one that generalises furthest. My sweep had been calling a private method rather than the public entry point the product calls. It therefore skipped the time budget, the fallback chain and the geo gate.

So the number I trusted was not describing what a customer receives. It was describing a code path no customer ever takes. A metric that does not run the product path is not a weaker metric, it is a metric about something else.

The part that is not about my code

Every one of these failures was silent. No exception, no red flag, no dip that looked like anything. A 200 arrived. Bytes arrived. In one case a confidence score of 90 arrived. The countries just quietly returned less, and "that market is thin" is an explanation that is always available and always plausible.

I found them because I own the fetch. I can put a gate before the parser, log the byte count, and go read the page myself. That is the actual argument for building the source rather than buying the list, and it has nothing to do with price.

If you buy company data: which source did a given row come from, and when was that source last verified?

Because when a source rots, no exclamation mark appears on your list. The rows keep arriving, on schedule, in the same format as always. And they look completely fine.