comparison guide
Index Follow vs Noindex Nofollow
Understand index, noindex, follow and nofollow as separate crawler directives, including when robots.txt can prevent a noindex directive from being seen.
Published and reviewed · Version 2
The four combinations in plain language
Index or noindex controls whether a page may appear in search results. Follow or nofollow controls whether supported crawlers may follow links found on the page. A page can therefore be noindex, follow: excluded itself while still allowing its links to be discovered.
These values are crawler directives, not authorization. Anyone who knows a public URL may still request it. Sensitive material requires authentication or removal from public hosting.
- index, follow: ordinary public page
- noindex, follow: exclude the page but allow link discovery
- index, nofollow: allow indexing but ask crawlers not to follow page links
- noindex, nofollow: request both restrictions
Why robots.txt and noindex can conflict
A crawler must fetch a page to read a robots meta tag or X-Robots-Tag header. If robots.txt blocks the fetch, the crawler may never see noindex. A blocked URL can sometimes remain known through external links or earlier crawl data.
For a page that must leave search results, keep it crawlable long enough for noindex to be processed, or remove it with an appropriate HTTP status when the content is genuinely gone. Exact handling depends on the publishing goal.
Meta tag versus HTTP header
HTML documents commonly use a robots meta tag. Non-HTML resources and site-wide server rules can use X-Robots-Tag response headers. Check the rendered response rather than assuming a CMS setting reached production.
Conflicting directives are risky. Review page markup, framework metadata, CDN headers and server configuration together.
<meta name="robots" content="noindex, follow">Pre-publish checklist
Confirm the URL's purpose, canonical target, response status and sitemap membership before changing directives. Remove noindex pages from the sitemap, and do not use noindex as a substitute for consolidating duplicate content.
- Inspect the live HTML and response headers
- Confirm robots.txt permits the required crawl
- Keep only canonical indexable URLs in the sitemap
- Recheck after deployment and wait for recrawl evidence
Sources and methodology
Sources support standards or platform behavior; examples and workflow guidance are original ToolNovaX editorial material.
- Introduction to robots.txt — Google Search Central
Editorial attribution
ToolNovaX Editorial Team
The internal publishing workflow responsible for tool verification, examples, accessibility review and source checks. This is an organizational attribution, not a claim of individual professional credentials.
Frequently asked questions
What does noindex, follow mean?
It asks a crawler not to index the page while still allowing links on it to be followed.
Does nofollow hide a URL?
No. It is a link-following directive, not access control.
Can robots.txt remove a page from Google?
Robots.txt controls crawling. It does not reliably remove a known URL from search results.
Should a noindex URL be in a sitemap?
No. A sitemap should list canonical URLs intended for indexing.
Is noindex immediate?
No. A crawler generally needs to revisit and process the directive.
Can I use X-Robots-Tag instead?
Yes. It is especially useful for non-HTML responses, provided the crawler can fetch the response.
Related guides
Related tools
Change history
- Version 1: reviewed publication in Batch 2.
- Version 2: expanded from verified search-intent signals with original examples, clearer boundaries and updated FAQs.