Most business owners think they’ve got a handle on their AI SEO the moment they get a good answer from ChatGPT. A remodeling business owner sat down at a marketing conference breakfast and proudly pulled out his phone. He’d asked ChatGPT for the best remodelers in his city, and there he was, right at the top. His marketing consultant, sitting across the table, was asked to see it. He then pulled out his own phone and typed the exact same question. The business wasn’t there at all. Two phones, one identical question, two completely different answers. Only one of them was good news.
TL;DR
Two people asking ChatGPT the exact same question about your business can get two completely different answers, because ChatGPT tailors its responses based on saved memory, conversation history, and account-specific context. That means checking your own AI visibility from your own logged-in account can hand you false confidence. Your result might just reflect your own chat history, not what a real prospective customer would actually see. Here’s why that happens and how to test it properly.
Why this isn’t a one-off story
It’s easy to write off the remodeler’s experience as a fluke or bad luck. It isn’t. A handful of independent write-ups on ChatGPT’s behavior this year point to the same underlying mechanism: the thing doesn’t retrieve one fixed answer the way a search engine returns a ranked list. It generates a response shaped by whatever context exists for that specific account, in that specific moment.
What stuck with us about consultant’s story wasn’t the technical explanation. It was the thirty seconds of whiplash, pride to concern, over the exact same question, asked from two phones on the same table. That wasn’t a bug. It was the system doing exactly what it’s built to do, which is precisely what makes it dangerous if you’re relying on a single self-check to feel confident about anything.
And confidence is really the whole issue here. The entire point of checking your own visibility is that the check should reflect reality. If it doesn’t, every decision built on top of it, whether to invest, how much, what to prioritize, is resting on something shakier than it looks.
The four reasons two people never see the same answer
Saved memory is the big one. If you’ve ever mentioned your business, your industry, or even something loosely adjacent in a past ChatGPT conversation, that context can quietly color how later questions get answered, even ones that sound completely unrelated. OpenAI built memory specifically to carry context across sessions. Genuinely useful for most things people use it for. Genuinely distorting if you’re trying to run an objective test.
Custom instructions do something similar. Anyone who’s told ChatGPT their job, their preferences, or how they like answers formatted has changed how every future response gets generated, recommendations included. Someone with “I’m a marketing consultant evaluating vendors” in their custom instructions is probably going to get a different kind of answer than someone with nothing set at all.
Then there’s plain conversation history within a single chat. Ask a question at the start of a fresh session and you might get a different answer than asking it ten messages deep into an existing one, simply because the model has more to work with by that point. Talk about dietary restrictions, then ask about restaurants, and watch the restaurant answer shift, even though the current question never mentioned food allergies at all.
What this means if you’ve already checked
If you’ve run any kind of self-audit, asked ChatGPT point blank whether it knows your business, checked what it says about your services, that answer came filtered through your own account’s accumulated history the whole time. A good result doesn’t mean a stranger, asking from a fresh account with zero history tied to your business, would see the same thing.
It’s a real gap in a lot of otherwise decent self-testing advice, including the shape of the AI visibility checks we’ve written about before, like the AI Trust Stack framework. The four layers in that piece still hold up fine. What needs adjusting is the assumption that one check, from one logged-in account, tells the whole story.
How to actually test this properly
Run it logged out, or in an incognito window, so saved memory and custom instructions aren’t quietly nudging the result. That alone removes the biggest source of distortion.
Ask someone else to run the exact same question from their own account, ideally someone who’s never talked to ChatGPT about your industry before. Compare notes. If there’s a real gap between what you got and what they got, that’s worth digging into, not shrugging off as noise.
And run it more than once. Sampling variation alone can produce different answers under otherwise identical conditions, so a business that shows up consistently across several fresh, logged-out attempts is in a meaningfully stronger spot than one that showed up exactly once.
Self-check versus a real audit
| Factor | Typical self-check | Reliable audit |
|---|---|---|
| Account state | Logged in, own account with history | Logged out or freshly incognito |
| Number of runs | One | Several, across separate sessions |
| Who tests | Only the business owner | Owner plus at least one independent account |
| Confidence in result | May reflect personal chat history | Reflects a genuine prospective customer |
Why this matters more for small businesses specifically
A bigger company running paid AI monitoring tools already has some protection built in, since those tools generally query from clean, unauthenticated sessions rather than someone’s personal ChatGPT account. A small business owner checking from the phone they use for everything, every day, doesn’t have that built-in safety net.
It gets worse in one specific, common case. A business owner who lives in ChatGPT all day for their own work, drafting emails, researching suppliers, brainstorming ad copy, has built up months of history directly tied to their own business and industry. That’s the account where a self-check is least trustworthy, because it’s the account with the richest, most business-specific context of anyone who might ever ask that question.
This is the exact gap ongoing monitoring is meant to close. Not a one-time test you run once and either relax or panic. Testing done properly, on a schedule, under conditions that actually resemble a stranger’s, treating the result as a real signal instead of a mirror of your own chat history.
Key takeaways for your AI SEO strategy
- ChatGPT tailors answers based on saved memory, custom instructions, and conversation history, so identical questions can return different results for different people.
- There’s a real, publicly told case of a business owner seeing a favorable AI recommendation on his own phone that vanished entirely when the same question came from someone else’s account.
- Checking your own visibility from your own logged-in account risks a false positive, since your account’s context may be quietly shaping what you see.
- Testing logged out, roping in someone else to test independently, and running it more than once all cut down this specific source of distortion.
- This blind spot hits small businesses harder than larger ones, since bigger companies are more likely to already be running dedicated AI monitoring tools that test clean by default.
Frequently asked questions
Does this mean my ChatGPT check was worthless?
Not worthless. Incomplete. A favorable result from your own account is a data point, not proof, since it might be shaped by your own history in ways that don’t hold up for a real prospective customer.
Is this a ChatGPT-only thing, or does it happen everywhere?
The specifics vary by platform, but the underlying idea, that account context, conversation history, and probabilistic generation all shape what comes out, applies broadly across the major AI assistants, not just one.
How often should a business retest this?
Monthly, run under clean, logged-out conditions, is a reasonable baseline for most small businesses. AI answers and the facts feeding them can shift faster than most owners expect.
Can I just turn off personalization for a cleaner test?
Most platforms let you disable memory and conversation references in settings, and that helps. But the safest route for a genuinely clean test is still a logged-out or incognito session rather than tweaking settings on an account with existing history.
Is this the kind of thing a proper agency actually watches for?
Yes. Testing consistently, under conditions that reflect a real prospective customer rather than one favorable result from one account, is a core part of doing this work properly rather than checking a box once.
Does hiring someone for this eliminate the problem completely?
It cuts it down a lot, since real monitoring is built around repeated, controlled testing rather than an occasional ad hoc check. But no amount of monitoring makes any single AI answer perfectly predictable. The goal is a reliable trend across many clean tests, not a guarantee on any one of them.
Conclusion
The uncomfortable part of the remodeler’s story isn’t that ChatGPT got something wrong. It’s that his test was never actually testing what he thought it was testing. A confident result from your own phone tells you something about how ChatGPT treats your account. It doesn’t reliably tell you what a stranger asking the same question would see.
If you want a genuinely reliable read on where your business actually stands, our AI SEO agency team can run the test properly, or let us run it for you.
