bramdeansleepers.co.uk

robots.txt configuration pasted into DNS

$ dig TXT bramdeansleepers.co.uk

;; QUESTION SECTION:

;bramdeansleepers.co.uk. IN TXT

;; ANSWER SECTION:

bramdeansleepers.co.uk. 3600 IN TXT "Sitemap: https://www.bramdeansleepers.co.uk/sitemap.xml Sitemap: https://www.bramdeansleepers.co.uk/sitemap_pages.xml User-agent: * Disallow: Crawl-Delay: 30"

What Happened

The admin pasted their entire robots.txt configuration directly into the DNS TXT record. "Sitemap: https://www.bramdeansleepers.co.uk/sitemap.xml | User-agent: * | Disallow: | Crawl-Delay: 30". Search engine crawlers look for robots.txt at the web root (yourdomain.com/robots.txt), not in a DNS text record. Googlebot will never see this.

Full TXT Record Value

Sitemap: https://www.bramdeansleepers.co.uk/sitemap.xml Sitemap: https://www.bramdeansleepers.co.uk/sitemap_pages.xml  User-agent: * Disallow:   Crawl-Delay: 30