Sitemaps & robots.txt explained
Two files that tell search engines where to go on your site — what they are, why they matter, and who manages them.
Your website uses two special files to communicate with search engines: a sitemap and a robots.txt file. Most business owners never need to edit these directly — but it helps to understand what they do and why they matter.
Quick summary
A sitemap is a list of all your website's pages, submitted to Google to help it discover your content faster. A robots.txt file tells search engines which pages to ignore. Both are handled by your site setup — but a misconfigured robots.txt can accidentally block Google from your entire site, so it's important to know they exist.
What is a sitemap?
A sitemap is a file (usually in XML format) that lists all the pages on your website. Think of it as a table of contents you hand directly to Google.
When you submit your sitemap to Google Search Console, Google can:
- Discover pages it might not have found through link-following alone
- See when pages were last updated
- Prioritize which pages to crawl
A sitemap doesn't guarantee that Google will index every page — it's a guide, not a command. But it does help Google do its job more efficiently, especially on large or new sites.
Does your site have a sitemap?
Most modern content management systems generate sitemaps automatically:
- WordPress with Yoast or Rank Math: Your sitemap is typically at
yoursite.com/sitemap_index.xml - Webflow: Automatically generates a sitemap
- Squarespace: Automatically generates a sitemap at
yoursite.com/sitemap.xml
We submit your sitemap to Google Search Console as part of every site launch.
What is a robots.txt file?
The robots.txt file is a text file at the root of your website (at yoursite.com/robots.txt) that tells search engine bots which pages or sections of your site they should or shouldn't crawl.
It's used to:
- Prevent Google from wasting crawl time on pages that shouldn't be indexed (like admin pages, login pages, or duplicate content)
- Block bots from accessing certain site sections for legitimate reasons
The robots.txt file uses a simple format:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xmlThis example tells all bots ("User-agent: *") not to crawl the WordPress admin area, but allows one specific file needed for site functionality. It also points bots to the sitemap.
The critical danger with robots.txt
A wrong robots.txt can block your entire site from Google
A single incorrect line in your robots.txt file can tell Google not to crawl your entire site. This is a serious SEO error. If you suspect this has happened — or if you suddenly see a major drop in Google traffic — contact us immediately. We'll check your robots.txt and Search Console for issues.
This is one of the most common accidental technical SEO mistakes. It can happen when a site is launched from staging to live and a "noindex" or "disallow all" setting from the staging environment is accidentally carried over.
What you don't need to do
For most business owners, both files are set up and managed for you:
- We create and submit your sitemap during site setup
- We configure your robots.txt to allow Google access to your important content
- We check these files as part of any SEO audit or site migration
You don't need to edit these files yourself. If something seems wrong with your search visibility, let us know and we'll check.
How to verify your sitemap is submitted
If you have access to Google Search Console:
Open Google Search Console and select your site property.
Click "Sitemaps" in the left-hand menu under Indexing.
Look for your sitemap URL in the list. If it shows a green "Success" status, it's been submitted and Google can read it. If it shows an error, let us know.
Common questions
Related guides
Need a hand?