How to create a sitemap and robots.txt for a small site
robots.txt tells crawlers which parts of a site they may fetch; sitemap.xml lists the pages you want found. A small site needs both to be simple: a robots.txt that allows everything and points to the sitemap, and a sitemap that lists every real page at its final address. Most problems come from one wrong line.
robots.txt, line by line
The file lives at the root of the host, such as https://www.example.com/robots.txt; crawlers don't look for it anywhere else. Its format is standardised in RFC 9309. A sensible file for a small business site is four lines:
User-agent: *- The rules that follow apply to every crawler. You can add separate groups for named crawlers, such as
User-agent: Googlebot. Allow: /- Everything may be fetched. An empty
Disallow:line means the same thing. - (blank line)
- Separates groups; it's there for readability.
Sitemap: https://www.example.com/sitemap.xml- The full address of your sitemap. It isn't tied to a user-agent group and can appear anywhere in the file.
The one line that hides a whole site
Disallow: / under User-agent: * tells every compliant crawler not to fetch any page. It's common on staging sites and sometimes survives the move to production. Search Console will then report pages as blocked, and new pages won't be crawled.
Two details are often misunderstood. First, robots.txt controls crawling, not indexing. Google's robots.txt introduction notes that a blocked URL can still be indexed if other pages link to it, just without a description. To keep a page out of results, let it be crawled and use a noindex meta tag instead. Second, robots.txt is a request. Well-behaved crawlers follow it; it isn't a security control, so never use it to hide private pages, because the file itself lists them for anyone to read.
sitemap.xml, line by line
A sitemap is an XML file following the sitemaps.org protocol. For a five-page site it's short:
<?xml version="1.0" encoding="UTF-8"?>: the XML declaration.<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">: the wrapper, with the protocol's namespace.<url><loc>https://www.example.com/services/</loc><lastmod>2026-10-20</lastmod></url>: one entry per page.locis required and must be the full, final address;lastmodis optional and should be the date the content really changed.</urlset>: closes the file.
The protocol also defines changefreq and priority, but Google says it ignores them, so there's little point maintaining them by hand. A sitemap can hold up to 50,000 URLs; a small site will never come close.
Create and check them, step by step
- List your real pages at the exact addresses visitors use: same protocol (
https), same host (wwwor not), same trailing slash convention. - Write the sitemap with one
<url>per page, or use what your platform generates. - Write robots.txt with the four lines above, using your sitemap's full address.
- Upload both to the root of the host the site is served from.
- Open each in a browser.
/robots.txtshould show plain text;/sitemap.xmlshould show XML without errors. - Compare the sitemap to the site. Every page listed should load with a 200 status, and no live page you care about should be missing.
- Submit the sitemap in Google Search Console and Bing Webmaster Tools, then check the report a few days later for errors.
Mistakes to look for
- Sitemap addresses on the old domain or the
http://version after a move. - Pages in the sitemap that redirect or return 404.
- A
lastmodthat changes on every request, which teaches crawlers to ignore it. - Blocking CSS or image folders in robots.txt, which stops search engines rendering the page as visitors see it.
- A robots.txt that returns an HTML error page instead of plain text.
What Anticode serves for you
Every published Anticode site serves its own robots.txt and sitemap.xml at its root, on its anticode.dev address and on a connected domain, so there's nothing to write or upload. The sitemap includes the site's separate pages and its CMS collection pages. Open both on your live address and read them with the guide above; checking takes a minute. Each site also serves an llms.txt file for AI assistants, explained in what is llms.txt.
Limits
Anticode generates these files; it doesn't let you hand-edit them, and it doesn't submit the sitemap to search engines on your behalf. The generated robots.txt allows everything, so a published site can't be hidden from search engines, and you can't block a specific crawler or keep a single page out of results with it. For submitting and monitoring, see submitting a site to Google Search Console.
Questions people ask
Does a small site need a sitemap?
Not strictly; a well-linked small site gets crawled anyway. But it's cheap, helps new pages get found sooner, and Search Console's sitemap report is a useful check.
Should I block AI crawlers in robots.txt?
That's a business decision. Several AI companies document crawler names you can disallow, but blocking them may also keep your business out of AI assistants' answers.
Can robots.txt remove a page from Google?
No. Use a noindex tag on a crawlable page, or remove the page and let it return 404 or 410.