User-agent: Scrapy Allow: / User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Sitemap: https://www.bespoketextiles.co.uk/sitemap_index.xml