I Was Asking for Painters
A second tool did not catch the mistake. A second question did.
My own tool returned zero again for fine art pet portraits in Arizona. Painters still held most of the top positions, and my business did not appear anywhere.
That was supposed to answer the question I left open in the last issue. The July 17 changes had gone live, the second run had arrived, and I expected the score to tell me whether the web had learned anything.
It did not.
The problem was not that the score stayed flat. The problem was that the question had never been measuring my market closely enough for the score to mean what I thought it meant. I still do not know whether the July 17 fixes changed anything. The instrument I planned to use for that answer was surveying painters.
Another monitor, running during the same period, reported fine art pet photography as one of my strongest positions. It queried the same answer engines and evaluated the same business during the same week, yet one score said I did not exist while the other placed me near the top of everything it tracked.
The questions were different. The commercial monitor included the word “photographer” and asked about Tucson. Mine asked who creates fine art pet portraits in Arizona. The engines answered my question correctly. The artist taking most of the top positions is a real fine art pet portrait artist working in Arizona.
She paints.
The geography was doing more work than I had noticed too. Ask someone from Phoenix about Arizona and there is a decent chance the answer will concern Phoenix, Scottsdale, Tempe, Mesa, Chandler, or someplace else in the Valley. Tucson may eventually enter the conversation, along with Flagstaff and whatever part of Sedona they visited last winter, but Phoenix has enough people to mistake its part of the state for the state itself. It seems the models made the same assumption.
I had spent two months reading that output as a verdict on my photography in Pima County.
The plumbing was fine
I built the tool, so I knew where to look when a result seemed wrong. I checked whether the search fired, whether sources came back, whether the provider stayed grounded, and whether the output parsed correctly. I also checked for the failures I have learned to expect: malformed JSON, missing citations, silently degraded calls, and information that sounded convincing until I tried to verify it.
None of those things had happened. The engines received a clear question and returned an accurate answer, which meant the tool had done exactly what I asked it to do. The problem sat one layer earlier. I had audited the plumbing and never examined the premise.
A tool cannot flag a well-formed question about the wrong thing. There is no error state for that. The output looks exactly like a valid measurement because it is one, and the mistake occurs before the search begins. That makes it harder to catch than the failures that announce themselves with missing fields or broken responses.
I apply more skepticism to other people’s numbers. When a vendor publishes a statistic about AI search, I treat it as a hypothesis until I understand what was measured, how the sample was built, and what the company has to gain by presenting the result that way. My own tool’s output entered my files as a finding. I did not trust the underlying engines more because I had built the interface around them. I trusted my own question without noticing that I was doing it.
The second tool is not what helped
The obvious response is to add another monitor, but that is not what caught the mistake. Had I copied my prompt from one system into the other, both would have surveyed painters. I would have received the same zero twice, from two different tools, and treated the agreement as confirmation.
Two tools agreeing on a bad premise is one measurement billed twice.
The disagreement existed because the questions had been composed separately. I instructed Claude to write the prompt inside my tool while thinking about category banks, geographic variables, and keeping weekly runs comparable. I wrote the commercial monitor’s prompt by hand in a different sitting, thinking about how someone looking for my work might phrase the request. Same person, two working postures, two questions that were not actually about the same market.
That is useful, but it does not solve the problem by itself. Another person can still share my assumptions. An AI system drafting blind will probably rebuild a lot of the same category language from the same business context. Even I, a week later, am still bringing the same model of my own market to the page.
I already had a source farther upstream.
Earlier this year I did two rounds of prospect research, one for pet photography and one for headshots. I asked six AI systems to map the questions people ask while they are still deciding whether professional photography makes sense for them. I was looking for the language on the other side of the transaction, before someone has read my site long enough to adopt my vocabulary.
Those questions do not sound much like my category labels.
Pet owners ask whether a photographer can work with an anxious dog, whether the whole thing is worth the money when they already have thousands of phone photos, or whether it is too late to make good photographs of an old dog. Headshot clients ask what happens if they are not photogenic, whether they will look awkward, and how to find someone who is good with people who hate being photographed.
The important part is not that AI generated those questions. It is that I generated them months ago for a different purpose, before this measurement problem existed and without any reason to shape the wording around what I would later want a visibility tool to prove. Working with these tools are like that; it’s not always a straight line but the work, when done right, always builds on itself.
That research was sitting in my files while my monitoring tool asked about “fine art pet portraits in Arizona.”
It is still mediated evidence, and my inquiry history gives me another source that began outside my own category system entirely. But both are more useful here than asking myself to invent one supposedly neutral prompt after I already know what result I am trying to test.
