Unsupported Directives and Typos in Your Robots.txt File
Summary: Google reads only four directives in a robots.txt file: User-agent, Disallow, Allow and Sitemap. Crawl-delay, a noindex written inside the file and Host lines are ignored. A misspelled line is skipped silently, so you think you have blocked something when in fact nothing is blocked.
What Google recognises in robots.txt
Robots.txt is a plain text file at the root of your site, for example https://yoursite.az/robots.txt. Its job is to tell bots which paths they may crawl. The list of directives Google's documentation recognises is short:
User-agentsays which bot a rule applies toDisallowblocks crawling of the given pathAllowreopens a specific path inside a blocked folderSitemapgives the full address of the sitemap and does not depend on which bot it sits under
In paths, the asterisk and the $ sign that marks the end of a line both work, and paths are case-sensitive. To Google, /Blog and /blog are two different paths.
Unrecognised lines and how Google really behaves
Every line outside this list is an unrecognised directive. Google returns no error for it; it simply skips the line without reading it. That is exactly the problem: the file looks as if it works.
| Line | What Google does | What to do |
|---|---|---|
Crawl-delay | Not recognised, has no effect on crawl speed | Delete it. If the server is overloaded, solve that on the server side |
noindex inside the file | Not recognised, the page stays in the index | Delete it. Move it to the page's own meta robots tag |
Host | A Yandex directive, Google does not read it | Delete it. Settle the main domain with a redirect |
Some sources write that these directives will become official sooner or later. There is no such promise. Google's current documentation does not recognise them, and no change has been announced. Build the file for today's documentation, not for what you expect in the future.
One typo cancels the whole rule
Spellings like Dissalow, Diallow and Disalow turn up often. Google does not correct them. Because the line is not recognised, it is ignored, and the section you believe is blocked stays fully open. A block of rules written without a User-agent line has the same result, because it is not clear whom the rule applies to.
The costliest mistake: confusing crawling with indexing
Robots.txt blocks crawling, not indexing. If other sites link to a blocked page, Google can put that address in the index without seeing its content. The result is a bare line with no description in the search results.
It gets worse. If you block a page with Disallow, Google cannot read the noindex tag inside it, because it never opens the page at all. So if you want a page out of the index, do the opposite: allow crawling and put the noindex tag on the page itself.
Checking the file in ten minutes
- Open the address in your browser. If the file is not in the root folder, Google does not find it.
- Mark and delete every line outside the four recognised directives.
- Check the spelling of
Disallowletter by letter. - Add the full address of your sitemap with a
Sitemapline. - Keep the test and live environments apart. The most widespread disaster is the
Disallow: /line from the test file being copied to the live site on launch day. - Open the indexing report in Search Console and look at the list of addresses blocked by robots.txt.
After a change, the result does not show at once. Google keeps a copy of the file in its cache for a while, so follow the effect of the fix over a few days.
Who should do this
Robots.txt is not a strategy on its own. It is the smallest, cheapest and most easily broken part of technical SEO. If your site was built by a freelancer, or nobody has touched the file since launch day, having it checked once costs little.
SaytPro is a team of developers working in Baku since 2023 and has delivered more than 120 projects. Our monthly SEO service costs 700 ₼, and a technical audit is part of it. New websites come with 2 months of formal warranty and technical support after handover. Write to us for a free consultation about the state of your file.
