sitemap.xml and robots.txt, written for every published site
Every site published on Anticode answers at /robots.txt and /sitemap.xml without you creating either. Both are built when a crawler asks for them, from the pages you published and the CMS entries in your project, so a new entry is listed the next time a search engine looks, with no republish and no plugin.
What the two files are for
A sitemap is a list of the addresses on a site that you'd like search engines to know about, with the date each last changed. robots.txt is the first file a well-behaved crawler reads, saying what it may fetch and where the sitemap is. Neither makes a page rank, but a missing or stale sitemap is a common reason new pages take longer to be found, and a wrong robots.txt can hide a whole site. A website builder automatic sitemap takes both off your list, provided it really is current. If you want the background first, the guide to sitemaps and robots.txt explains them from scratch.
robots.txt
Each site's robots.txt is four lines: every user agent, allow everything, a blank line, and the full address of the sitemap. The address uses whichever host the request arrived on, so on your own domain it points at your own domain's sitemap, and on <name>.anticode.dev at that one. Nothing on a published site is meant to be private, which is why nothing is disallowed: a published site is public at its address.
sitemap.xml
| Entry | Address | Last modified |
|---|---|---|
| Home page | / | The day the site was last published |
| Each separate page (Work, About, Pricing, Contact…) | /about/ and so on | The same day, since they're published together |
| Each CMS collection's index | /<collection>/ | Not given |
| Each CMS entry | /<collection>/<slug> | The day that entry was last edited |
The file follows the standard sitemaps.org format and can hold up to 20,000 entries. It's cached for a few minutes, so an entry you add in the dashboard appears in it shortly afterwards. Entry pages carry their own canonical link, and their title and meta description come from the entry's SEO title and meta description fields when you fill them in, or from its first text field and an excerpt when you don't.
Why it stays current
Many site tools write a sitemap at publish time and leave it there, so it drifts the moment content changes elsewhere. Anticode's isn't stored at all. Each request reads the list of published pages and the project's collections at that moment. Add a post to a CMS collection and it's in the sitemap within minutes; delete one and it's gone. Republish the site with a new About page and that page is listed with the new date. A collection address typed without its trailing slash, such as /journal, moves permanently to /journal/, so search engines settle on one address per page rather than two.
What you still do
- Submit the sitemap address once in Google Search Console and in Bing Webmaster Tools, so both know where to look.
- Give important CMS entries a proper SEO title and meta description in the dashboard instead of relying on the excerpt.
- If you move to your own domain, submit the sitemap on that domain too.
Limits
- Not editable. You can't add rules to robots.txt or hand-edit the sitemap. There's no way to block a particular crawler, AI crawlers included, or to hide one page from search.
- Not every address is listed. The generated Privacy and Terms pages, llms.txt and the security report aren't in the sitemap, though crawlers can still follow links to them.
- One language. Sites are in English only, so there are no hreflang alternates.
- Dates are page-level. Every design-engine page shares the last publish date, even if you only changed one of them.
Questions people ask
Do I have to update the sitemap when I add a blog post?
No. If the post is an entry in a CMS collection, it's listed the next time the sitemap is requested, a few minutes at most after you add it.
Does the sitemap use my own domain?
Yes, when it's requested on your own domain. The addresses inside it follow the host the request came in on.
Can I stop AI crawlers from reading my site?
Not through Anticode's robots.txt, which allows every crawler. Your dashboard does count AI and search crawlers separately, so you can see how often they visit.