Why does ChatGPT give wrong links? It guesses the URL, and my first three fixes each failed for a different reason
I asked ChatGPT whether Circleboom has an X shadowban test.
The answer was right. It said yes, named all five checks the tool runs, and explained that each check gets its own result. Under the answer was a link labelled “Circleboom X Shadowban Test.”
The link opened our 404 page.
An assistant had described my product correctly and then sent me to an address that does not exist on my own site. I wanted to know why, how often it happens, and how to fix it.
ChatGPT gives wrong links because it often writes the URL instead of looking it up. The model has learned what addresses on a website tend to look like, and it produces one that fits the pattern. In our case the page was real and the address was made up.
Finding the cause took me hours. Counting the damage meant pulling nine and a half months of analytics, and it was larger than I expected. Fixing it took four attempts, and each of the first three failed for a different reason.
ChatGPT wrote a shorter Twitter shadowban test URL than the real one
Here are the two addresses, without the domain.
What ChatGPT linked: /twitter-shadowban-test
Where the page lives: /twitter-management-tool/twitter-search-tool/twitter-shadowban-test
The real Twitter shadowban test sits two folders down. ChatGPT’s version sits at the root, with a last segment that is identical to ours, character for character. Everything the model dropped was our folder structure.
The short address is not in our sitemap, and the server answers it with a 404.
A row of small source chips beside the answer said Circleboom. The model had read our pages before it replied, which is why the five check names were correct. What a model reads first on a page is its title, a subject I covered in how to write a title tag for AI search.
So the facts came from our pages. The address, as far as I can tell from outside, came from the model.
I can’t open ChatGPT and prove that. But the wrong URL looks like a guess. It is clean, short, built from the page’s own name, and missing only the part that nobody outside our team would know.
A Cloudflare post explained ChatGPT’s wrong link in one paragraph
The explanation, when I found it, was the first paragraph of a Cloudflare post about something else. The post announces its Web Search API, and it opens like this:
“Fun fact: when you use an agent and it needs to fetch a live web page, the agent usually just guesses the URL of the page and then makes a tool call to curl it. This is why you’ll sometimes see web fetches come back with a 404 Not Found, which happens if the agent incorrectly guesses the URL of that information.”
Cloudflare was describing my 404 from the agent’s side.
A language model produces text one piece at a time, each piece chosen because it is likely to follow the pieces before it. A URL is text too, and most of the web names a page with the domain, a slash, and the thing’s name in hyphens. That makes a short address near the root the most likely string for almost any page on almost any site.
The further your real structure sits from that short pattern, the more often the likely string and the true string differ.
Ours sits a long way from it. Circleboom has 335 tool and landing pages, and most of them live one to three folders down, under the product and the menu they belong to. That layout mirrors how we arranged the product. No visitor thinks of a shadowban test as part of “search tool,” and no model does either.
Cloudflare’s answer on the agent side is retrieval: let the agent run a search and work from addresses that came back from an index. It launched with three search partners, Ceramic.ai, Exa and Linkup.
I tried Linkup while I was reading about them. You send a question and get back an answer with its sources attached, and the source addresses come from its own index. An assistant answering a question often works without a full index of your site. Companies like this one supply it.
I have no relationship with the company and got nothing for mentioning it. It did what I asked of it, and I like a startup that picks one plain job and does it.
Retrieval narrows the problem. An assistant may run many searches behind one question, which Google calls query fan-out.
In my case retrieval did not remove the problem: ChatGPT had searched, and it still wrote a wrong address.
I stopped waiting for the model side to be fixed and looked at the side I control.
ChatGPT tags its own wrong links, and your analytics can count them
The full link ChatGPT gave me ended in ?utm_source=chatgpt.com.
OpenAI documents the tag in its publishers and developers FAQ. ChatGPT adds that parameter to the links it sends people through, so a site owner can see the visits in analytics.
Most teams read the parameter in one report, the one that counts sessions from ChatGPT. It is more useful next to your 404s.
Filter your 404s for visits that carry utm_source=chatgpt.com, and what comes back is a list of the addresses ChatGPT sent people to on your site that do not exist. Some will be old pages you moved. The rest are addresses the model made up.
The model labels its own mistakes and delivers them to the one place where you can count them. So I counted.
Our AI visitors act twice as often as the average, and ChatGPT sends most of them
I pulled every Google Analytics 4 session from January 1 to October 13 this year whose source was an AI assistant: ChatGPT, Gemini, Perplexity, Claude, Copilot, Meta AI and Mistral, plus a few others that sent nothing.
AI assistants sent us 21,999 sessions in that period, 1.6% of all our site sessions.
What those visitors do matters more than the count. A key event is an action we have marked as important in Google Analytics, and AI sessions produce key events at twice the site rate: 1.6 per 100 sessions, against 0.8. Only paid search does better.
My guess at why: someone who arrives from an AI answer has already asked a specific question and been told that we answer it. They come to do one thing.
The volume is growing too, from about 34 AI sessions a day in April to about 117 in August.
ChatGPT sent 88.8% of the sessions we could tie to a named assistant. I compared these visitors with our Google visitors in a separate piece.
Two caveats on all of these numbers. Visits from AI apps that pass no referrer are filed by Google as Direct, so every figure here is a floor. And Google only began filing these visits under its own AI Assistant channel in the week of May 31; before that, the same visits sat under Referral or Unassigned, and I counted them from the source names.
So the visitors an assistant sends are some of the best we get. Then I looked at where they landed.
AI assistants sent more people to the fake shadowban test than to the real one
I checked where AI visitors landed, page by page, for the top 60 landing pages. Nine of the 60 do not exist.
Those nine dead addresses took 1,609 AI sessions. That is 7.3% of all AI sessions, from the top 60 pages alone, so the real share is at least that.
The worst one was a near copy of the address ChatGPT gave me: /twitter-management-tool/twitter-shadowban-test. That guess kept our product folder and dropped only the middle one.
It took 1,040 sessions. The real shadowban page took 656.
dd the three invented versions together and 1,132 AI sessions came for our shadowban test and found nothing. Only 656 found the page.
For that one tool, about 63% of the visitors an assistant sent us arrived at an address that does not exist.
Sessions on the main invented address lasted 5 seconds on average, against 38 seconds on the real page.
The main invented address peaked in May, with 585 sessions that month. It fell to 273 in June, 129 in July and 35 in August. I can’t tell you what changed to make it fade.
The other dead addresses fall into two kinds.
The first kind is a tool guess: a sensible name that is not our path. /twitter-management took 79 sessions; the real page is /twitter-management-tool. /twitter-management-tool/delete-tweets took 57; the real tool is /twitter-management-tool/delete-all-tweets. With the shadowban guesses, tool guesses account for 1,268 of the 1,609 dead sessions.
The second kind is a blog post with its /blog/ folder removed. /best-investor-twitter-accounts-to-follow-on-x took 163 sessions, while the real post, the same address under /blog/, took 139. Three more articles lost their folder the same way, for 341 dead sessions in all.
Not one of the 1,609 sessions completed a key event. Neither did the 656 on the real shadowban page, so for that tool the loss is the visit itself, and the chance to be read.
A 404 also costs you the AI crawl
The sessions are the part you can see. There is a second cost.
When an agent guesses an address and fetches it, the 404 is the whole of what it gets from you. It goes back to its answer without your page in it.
The person never visits, and the model never reads you.
Our site sits behind Cloudflare, so I looked at how AI crawlers fare there. Around 10 to 15 percent of their first attempts to fetch a page fail.
I can’t tell you that every one of those failures is an invented link. Some will be ordinary errors.
But after reading the guessed addresses in our analytics, I strongly believe that a good share of them are the same kind of guess that sent me to a dead shadowban test.
My first fix was to redirect every 404 to the homepage
Every one of those 1,609 dead sessions was a visitor we lost. My first fix went straight at them: send every unknown address to the homepage, and no visitor sees an error again.
Get Kevin O. Frank’s stories in your inbox
Join Medium for free to get updates from this writer.
The logic held up: a visitor on the homepage can still find the tool, and a visitor on an error page leaves. The only open question seemed to be which single page would suit the most people: the homepage, the login page, a demo.
Then I played it through as one of those 1,040 sessions. Someone asked an assistant a precise question and got a precise answer. They clicked the link to the shadowban test, and they arrived at a page about everything we make.
Nothing would tell them the link had been wrong. They would have to work out alone that they had been sent to the homepage, find the menu, find the tool, and start again.
Before building it, I checked the idea against what Google has published. I have disagreed with Google about AI search, and I said why in are AEO, GEO and AI search optimization just SEO. On status codes I trust it: the guidance is old, tested and specific.
A server answers every request with a status code, and three of them matter here: 200 means the page exists, 404 means it does not, and 301 means it has moved for good.
Google has argued against the plainest version of my fix since 2008: answer a missing address with the homepage and a 200. It calls that a soft 404, and its own example reads, “This site returns a 200 response code and the site’s home page for URLs that don’t exist.”
A redirect to the homepage is the same idea one step removed. Google’s guidance on site moves says so directly: “Don’t redirect many old URLs to one irrelevant single URL destination, such as the home page of the new site. This can confuse users and might be treated as a soft 404 error.”
A 404 is a gauge reading. Wire every gauge to show green and the engine is still broken. You have also removed the one instrument that told you where to look.
The question I had been weighing, which single page suits the most people, has no good answer, because no single page suits them all.
A wrong URL usually says what the visitor wanted
If no single page fits, the fix has to send each visitor somewhere different.
The wrong address already says where.
/twitter-shadowban-test states what the person wanted as clearly as a search query would. Treat the wrong URL as a query, and find the real page it describes. If you can't tell which page it means, say so.
I wrote a strict rule and tested it against our sitemap, which holds 2,164 addresses: 335 tool and landing pages and 1,829 blog posts. Accept a page when its last segment is identical to the guess, or when the words overlap by at least 0.6 on a scale of 0 to 1 and beat the runner-up by 0.15. Otherwise answer 404.
First I fed it twelve addresses a model or a bot could ask us for. Six found their page and six did not. /pricing matched three pricing pages equally, and a machine should not choose between those. /wp-login.php, the address a bot asks for when it probes for WordPress, matched nothing.
Then I fed it the nine dead addresses from the analytics. It found a page for all nine, and for eight of them it was the page the visitor wanted. It even caught /twitter-management-tool/twitter-search-ban-shadowban-test, the strangest of the shadowban guesses.
Measured in sessions, those eight cover 1,552 of the 1,609 visits that hit a dead address.
The ninth went wrong, and in the direction that matters. For /twitter-management-tool/delete-tweets it found no clear winner among our tools, looked through the blog, and picked an article on how to delete tweets. The visitor wanted the tool. The rule would have handed them a post about it.
That mistake changed the rule. A guess that sits inside a product folder has to be matched against that product’s pages, never the blog.
The rule failed in a second way too. Our post planner lives at an address that ends in x-post-planner. I asked for /x-post-plan, one syllable short, and three pages tied at 0.50.
To a person those are the same page. The real version has to match on meaning, and keep the same refusal to guess when two pages are close.
So a match exists for almost every dead address our visitors hit. I thought delivering it would be the easy part.
My second fix was a 301 redirect, and my developers stopped it
Once there is a match, the obvious delivery is a permanent redirect. The visitor asks for the wrong address, the server answers 301 with the right one, and the browser goes there.
Google’s own list of honest outcomes for a missing address supports it. When it added soft 404 reporting to its webmaster tools in 2010, it said such a URL either “Should 301 redirect to a more accurate URL,” or “Doesn’t exist and should return a 404 or 410 response.”
I took the 301 to our developers, and they raised a problem I had not considered: files.
Missing files raise 404s as well as missing pages: a missing .gif or .jpg, a renamed script, an old font file. A site our size produces them all day. A rule that sits at that level and redirects on a guess would answer a request for a picture with a web page.
A permanent redirect has a second weakness. Browsers keep a 301 and reuse it, so one wrong match keeps sending people to the wrong place after we have corrected it on the server. The delete-tweets mistake had just shown me that wrong matches happen.
Both problems can be engineered around. But I had brought them a simple rule, and they had shown me a rule that would answer our own missing pictures with web pages.
I dropped the 301 and kept the match.
My third fix was a 404 page that redirects in two seconds
We moved the redirect off the server and onto the page. The address would still answer 404. We would log the request, find the closest real page, and show the visitor this:
“We’re redirecting you to the correct webpage. The URL you’re trying to reach is not valid or mistaken. You probably meant this page.”
A progress bar under the message filled in two seconds, and then the browser moved to the matched page.
For a person it read well. You click ChatGPT’s link, you are told plainly that the address was wrong, you see the name of the page you meant, and two seconds later you are on it.
I liked it because the status code stayed honest, the visitor saw what was happening, and nothing was cached anywhere. We had the copy written.
The problem had been in the Cloudflare paragraph since the first time I read it: “a tool call to curl.”
Curl is a command-line program that asks a server for an address and gets text back. It has no screen, and it runs no scripts.
The visitor who guessed the address is often an agent fetching text, with no browser behind it. A progress bar means nothing to that agent, and the agent never reaches the next page, because moving is something a browser does. The agent would get a 404 and one sentence, “We’re redirecting you to the correct webpage,” and nothing about the shadowban test.
That is the same kind of failed first attempt we see in Cloudflare, with a polite sentence attached.
We had designed a careful answer for the human and left the machine with one sentence of text. And the machine is the one that wrote the link.
We cancelled the two-second page.
That was three fixes, and ChatGPT’s link still opened our 404 page.
The fourth fix answers the wrong address with the right page
Each fix would have failed someone different. Sending everyone to the homepage would have failed the visitor, and the 301 would have put our own site at risk. The two-second page would have failed the machine that guessed the address and will guess it again.
All three tried to move the visitor somewhere else, so the fourth serves the page at the address the visitor already asked for.
This is the one we are building, and it has not shipped.
When a request arrives for an address that does not exist, the router, the part of our server that decides which page answers an address, will compare it with the real pages by what the words mean. If one page wins clearly, it returns that real page at the invented address.
The page it returns carries a canonical link, a tag that tells search engines and crawlers which address is the real one. In plain terms, the invented URL says: my canonical is the real URL.
Play it through once more as the visitor. You click ChatGPT’s link and the shadowban test opens, with nothing shown in between. You never learn the link was wrong, because for you it was not.
The agent fetches the invented address with curl and gets the full shadowban test page as text, which is the only form it can read. A 301 would have needed a second request, and curl does not make one unless it is told to follow redirects.
A search engine sees the same page at two addresses, and the canonical link tells it which one to keep.
I have been sure three times already, so here is the cost. It is the one Google named in 2008: many addresses answering 200.
The difference from my first fix is what comes back. The homepage answered every question with the same page. The fourth fix answers one question with the page it asked for, and only when the match is clear.
Google’s guidance on canonicalization says some duplicate content is normal, and that “having the same content accessible through many different URLs can be a bad user experience.” The same page calls a canonical preference “a hint, not a rule.”
Google’s page on consolidating duplicate URLs lists the canonical link as a “strong signal that the specified URL should become canonical.”
Google’s page lists a redirect above the canonical link. I am trading some of that strength for a page an agent can read in one request.
These are the conditions I have set for the fourth fix:
- Answer a guessed address only when one page wins clearly, and match a guess inside a product folder only against that product’s pages. Everything else gets a 404 status and a page that shows the nearest matches.
- Send a canonical link from the invented address to the real address with every matched response.
- Leave file requests alone. A missing
.gifor.jpgstays a 404. - Log every match with the address asked for, the page returned, the score and the referrer. A matched guess no longer shows up as a 404, so this log has to do the 404’s job.
- Count how often each guess repeats.
The first three keep the fix from repeating the earlier mistakes or adding a new one. The last two are how I will find out if this one has to go too.
The repeat count matters most to me. When the same wrong address keeps arriving, I read it as a request. The next time an assistant sends someone to /twitter-management-tool/twitter-shadowban-test, the router will handle it.
By the fiftieth, I should ask why the page does not live there. That address passed five hundred in a single month this spring.
ChatGPT’s short URL is the one a stranger would pick
I began by treating that link as ChatGPT’s error. It is an error: the page is not at that address, and 1,132 AI sessions that trusted an answer like it got a 404.
But put the two addresses side by side once more. One has the tool’s name and nothing else. The other has the tool’s name behind two folders that describe how we arranged our menus.
If I showed both to a stranger and asked which one is real, they would pick the short one. The model picked it too, and for the same reason.
That can look like the reverse of what I argued about how to choose a domain name for AI search, where the name that spells out your function is the wrong buy. The two fit together, because a domain has to name you and a path only has to name the page.
My first fix would have hidden that from me. My third would have hidden the real page from the machine asking for it. The fourth is decided and not built, and I hope it does not start a problem of its own.
ChatGPT was wrong about where our page lives. It was right about where it belongs.
