Sitemaps & robots.txt explained

Two files that tell search engines where to go on your site — what they are, why they matter, and who manages them.

Your website uses two special files to communicate with search engines: a sitemap and a robots.txt file. Most business owners never need to edit these directly — but it helps to understand what they do and why they matter.

Quick summary

A sitemap is a file listing the pages on your website, which you can submit to Google to help it discover your content. A robots.txt file tells crawlers which parts of your site they may visit — it is not a way to keep a page out of Google. Both are handled by your site setup, but a misconfigured robots.txt can accidentally block Google from crawling your entire site, so it's important to know they exist.

What is a sitemap?

A sitemap is a file (usually in XML format) where you list the pages, images and videos on your site that you want search engines to know about, plus some information about them. Think of it as a table of contents you hand directly to Google.

A sitemap lets Google:

Discover pages it might not have found by following links alone
See when a page last changed, as long as the date in the sitemap is accurate and kept up to date
See which pages you think are important on your site

Two things a sitemap does not do. It doesn't guarantee indexing: in Google's words, a sitemap "helps search engines discover URLs on your site, but it doesn't guarantee that all the items in your sitemap will be crawled and indexed." And it doesn't let you set crawl order — Google ignores the priority and changefreq values some sitemap tools used to include. What it does do is help Google work more efficiently, which matters most on large or new sites.

Does your site have a sitemap?

Most modern content management systems can generate a sitemap for you:

WordPress with Rank Math: Rank Math builds a sitemap index at yoursite.com/sitemap_index.xml
WordPress with Yoast SEO: switch XML sitemaps on under Yoast SEO > Settings > Site features > Technical SEO, then click View the XML sitemap to open yours
Webflow: can generate one — check that Auto-generate sitemap is set to "Yes" under Site settings > SEO tab > Sitemap section, then save changes and publish the site
Squarespace: generates one automatically at yoursite.com/sitemap.xml (not on trial sites or parking pages)

We submit your sitemap to Google Search Console as part of every site launch.

What is a robots.txt file?

The robots.txt file is a text file at the root of your website (at yoursite.com/robots.txt) that tells search engine crawlers which URLs they may access on your site.

It's used to:

Manage crawler traffic, so a busy crawler doesn't overload your server
Keep crawlers off unimportant or near-identical pages, such as admin screens or filtered list views

robots.txt is not a way to hide a page from Google

Google is explicit about this: robots.txt "is not a mechanism for keeping a web page out of Google." If another site links to a page you've disallowed, the URL can still turn up in search results, just without a description. To keep a page out of Google, use a "noindex" rule or password-protect it instead.

The robots.txt file uses a simple format:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xml

This example tells all bots ("User-agent: *") not to crawl the WordPress admin area, but allows one specific file needed for site functionality. It also points bots to the sitemap.

The critical danger with robots.txt

A wrong robots.txt can block your entire site from Google

A single incorrect line in your robots.txt file can tell Google not to crawl your entire site. This is a serious SEO error. If you suspect this has happened — or if you suddenly see a major drop in Google traffic — contact us immediately. We'll check your robots.txt and Search Console for issues.

This is one of the most common accidental technical SEO mistakes. It can happen when a site is launched from staging to live and a "noindex" or "disallow all" setting from the staging environment is accidentally carried over.

What you don't need to do

For most business owners, both files are set up and managed for you:

We create and submit your sitemap during site setup
We configure your robots.txt to allow Google access to your important content
We check these files as part of any SEO audit or site migration

You don't need to edit these files yourself. If something seems wrong with your search visibility, let us know and we'll check.

How to verify your sitemap is submitted

If you have access to Google Search Console:

Open Google Search Console and select your site property.

Open the Sitemaps report from the left-hand menu.

Look for your sitemap URL in the list. A status of "Success" means Google fetched and read it without errors. "Has errors" or "Couldn't fetch" means something needs fixing — let us know. Note that the report only lists sitemaps submitted through it or through Google's API, so a sitemap Google found some other way (for example, from your robots.txt) won't appear here even though Google is using it.

Common questions

Do I need a sitemap if my site is small?

Possibly not. Google says you might not need one if your site is "small" — it puts a number on that: about 500 pages or fewer — and every page you care about can be reached by following links from your homepage. There's no harm in having one, and we submit one anyway because the Search Console report is a handy way to spot problems. Just don't expect the sitemap itself to improve your rankings.

Your navigation menu is for human visitors — it's the visible links at the top of your site. A sitemap is a machine-readable file for search engines. They serve different purposes and one doesn't replace the other.

Can I block specific pages from appearing in Google?

Yes — but with a "noindex" rule on the page, not with robots.txt. That's the method Google names for keeping a page out of search results, alongside password-protecting it. You might do this for thank-you pages or login pages. We handle this as part of technical SEO setup.

My new pages aren't showing up in Google. Could the sitemap be the issue?

Possibly. Check whether your new pages are included in the sitemap, and whether the sitemap has been submitted to Search Console. Also check that your robots.txt isn't blocking those pages. We can diagnose this for you.

Need a hand?

If you're stuck, email support@chykalophia.com and we'll help. Include your website address and a screenshot if you can.