Two small text files sit at the root of almost every website, and both are misunderstood constantly. robots.txt and the XML sitemap look simple, which is exactly why they get misconfigured. The two most damaging mistakes are opposite in nature: using robots.txt to try to hide a page (it does not do that), and blocking the CSS and JavaScript that Google needs to render your pages. Get either wrong and you can hide content you wanted indexed or break how Google sees your site.
This guide explains what each file controls, where people go wrong, and how to keep both tidy. It is a companion to our complete technical SEO guide, which covers crawlability, indexability, and site structure end to end.
Key takeaways
- robots.txt controls crawling, not indexing. Google is explicit that it is not a mechanism for keeping a web page out of Google. To keep a page out of search, use noindex or password protection.
- Never block CSS and JavaScript in robots.txt. If Google cannot fetch the resources needed to render a page, it cannot analyze it properly.
- A sitemap helps discovery, it does not force indexing. Submitting a sitemap is, in Google’s words, merely a hint.
- Every sitemap is limited to 50MB uncompressed or 50,000 URLs, whichever comes first. Beyond that, split it and use a sitemap index.
- A clean sitemap lists only canonical URLs you want indexed. Filling it with redirects, 404s, and noindexed pages sends Google mixed signals.
What robots.txt actually controls: crawling, not indexing
A robots.txt file tells search engine crawlers which URLs they can request on your site. That is the whole job. According to Google’s robots.txt introduction, the file is used mainly to avoid overloading your site with requests, and it is “not a mechanism for keeping a web page out of Google.”
This is the single most common misconception in technical SEO. People add a Disallow rule expecting a page to disappear from search, but Disallow only stops crawling. Google states plainly that a page disallowed in robots.txt can still be indexed if it is linked to from other sites. When that happens, the URL can appear in results, though the listing usually shows no description because Google was never allowed to read the page. Blocking crawling can actually lock in that ugly, descriptionless result, because Google cannot reach the page to see a noindex tag.
To genuinely keep a page out of Google, you need a method that permits crawling but forbids indexing: a noindex tag or X-Robots-Tag header, or password protection for anything sensitive. The effect of robots.txt also depends on the file type, as Google’s own documentation lays out.
| File type | What a robots.txt Disallow does |
|---|---|
| Web page (HTML, PDF) | Manages crawl traffic, but the URL can still be indexed if linked from elsewhere. Not a reliable way to hide a page. |
| Media file (image, video, audio) | Can prevent the file from appearing in Google results, but does not stop others from linking to it. |
| Resource file (CSS, JS, unimportant images) | Blocks the resource, but if the page depends on it to render, Google will not analyze the page well. Usually a mistake. |
A basic robots.txt is easy to read once you know the parts. User-agent names the crawler a rule applies to, Disallow lists paths to keep crawlers out of, and the Sitemap line points to your sitemap. Here is a simple, honest example:
User-agent: *
Disallow: /cart/
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xmlThe Sitemap directive is independent of the user-agent blocks and can sit anywhere in the file. You can list several sitemap lines if you have more than one, and Google will pick them up the next time it crawls the file, as described in the build and submit a sitemap documentation.
Never block CSS and JavaScript
This deserves its own section because it is a quiet, expensive error. Modern Google does not just read raw HTML, it renders pages the way a browser does, which means it downloads the CSS and JavaScript a page references. If robots.txt blocks those resources, Google sees a degraded version of your page.
Google’s guidance on resource files is direct: you can block unimportant script or style files only if pages loaded without them are not significantly affected. If the absence of those resources makes the page harder to understand, do not block them, “or else Google won’t do a good job of analyzing pages that depend on those resources.” Search Engine Journal makes the same practical point: blocking your own CSS and JS can degrade rendering and indexing, even though blocking a few external third-party resources is usually fine.
Common mistake. The rule of thumb is simple. Never disallow the directories that hold the CSS and JavaScript your templates depend on. A blanket Disallow on a folder like /wp-includes/ or /assets/ is one of the most common ways sites accidentally break their own rendering.
What a sitemap does, and what it does not do
An XML sitemap is a list of the URLs on your site that you want search engines to know about. It helps Google discover pages, especially on large sites or sites where pages are not well linked internally. What it does not do is force anything. Google is explicit that submitting a sitemap is merely a hint and does not guarantee that Google will download it or use it for crawling, and a sitemap never guarantees indexing. Google still judges each URL against its quality criteria.
That distinction matters. A sitemap helps with the discovery and crawling step. Whether a page gets indexed still comes down to its content and value. If your pages are not being indexed, adding them to a sitemap again will not change the verdict.
You may not even need one. Google notes that if you have a small site of around 500 pages or fewer and every page is reachable by following links from your homepage, you probably do not need a sitemap at all. Sitemaps earn their keep on larger or more complex sites.
When you do build one, respect the hard limits. Every sitemap format is capped at 50MB uncompressed or 50,000 URLs, whichever you hit first. Larger sites break the list into multiple sitemaps and submit a single sitemap index file that points to them all. A basic XML sitemap looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/guide.html</loc>
<lastmod>2026-06-04</lastmod>
</url>
</urlset>One detail worth knowing: Google ignores the priority and changefreq values some tools still add, and it only trusts lastmod when the date is accurate and verifiable. A lastmod that updates every time the copyright year changes is worse than none, because it teaches Google not to trust the signal.
Keeping the sitemap clean: good vs bloated
The most useful sitemap is a curated one. Google’s guidance is to include the URLs you want to see in search results, and to point to canonical URLs rather than every variant that leads to the same content. A bloated sitemap that dumps in redirects, dead URLs, and pages you have told Google not to index does the opposite of helping, it hands Google a list of contradictions.
A practical, sourced standard for a clean sitemap: as SE Ranking summarizes the best practice, every URL you list should return a 200 status, carry no noindex tag, and be the canonical version of the page. Anything else belongs out of the file. Here is the difference in practice.
| Clean sitemap (include) | Bloated sitemap (leave out) |
|---|---|
| Canonical URLs that return 200 OK | URLs that redirect (301 or 302) to another page |
| Pages you actually want indexed | Pages carrying a noindex tag |
| The single preferred version of each page | Non-canonical duplicates and parameter variants |
| Live, reachable pages | 404s and other error pages |
| Absolute, fully qualified URLs | Relative URLs or pages blocked in robots.txt |
Standard drawn from Google’s sitemap best practices and SE Ranking’s sitemap error guide.
If your CMS generates the sitemap automatically, which most do, the work is mostly checking that it excludes noindexed and redirected URLs rather than building the file by hand. After you submit it, use the Sitemaps report in Search Console to confirm Google read it and to see any processing errors. A sitemap that Google reports as read with warnings usually means it contains URLs that do not belong there.
How the two files work together
robots.txt and the sitemap are two sides of the same coin. One tells crawlers where not to go, the other suggests where to look. The clean setup is consistent: do not list a URL in your sitemap that you have blocked in robots.txt, and do not block resources your pages need to render. Point to the sitemap from robots.txt with a Sitemap line, keep the sitemap to canonical 200-status URLs, and reserve indexing control for noindex rather than crawl blocking. Get that alignment right and these two files quietly do their job instead of working against you.
FAQ
Will blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it, usually without a description. To remove a page from search, use a noindex tag or password protection, and make sure the page is not also blocked from crawling, or Google will never see the noindex.
Do I need an XML sitemap for a small website?
Probably not. Google says that if your site has around 500 pages or fewer and everything is reachable from your homepage, a sitemap is optional. It becomes valuable on larger sites, sites with poor internal linking, or sites with lots of new or media content.
Does adding a page to my sitemap guarantee it gets indexed?
No. A sitemap is a hint that helps discovery and crawling, not a command. Google still evaluates each page on its quality and value before deciding to index it.
What is the maximum size of an XML sitemap?
Each sitemap file is limited to 50MB uncompressed or 50,000 URLs, whichever comes first. If you exceed either limit, split the URLs across multiple sitemaps and reference them from a single sitemap index file.
Should I put my sitemap in robots.txt?
It is a good idea. Adding a Sitemap: line to robots.txt lets any crawler find your sitemap without you submitting it manually. You can list multiple sitemap lines, and the directive works regardless of where it sits in the file. Submitting through Search Console as well gives you error reporting that robots.txt alone does not.
Written by Matthis Duarte, a senior SEO and organic growth expert with 10+ years of experience driving organic growth for international brands across highly competitive verticals. He is the founder of Knownful, an independent publication on SEO and organic growth featuring in-depth guides, best practices, playbooks and original analyses, including a free monthly study of which brands AI actually recommends across 10 industries.
