TXT It!

TXT It! allows the direct upload of text files that end in the ".txt" extension to the root level of your site. From here on, we will refer to this type of file as a "TXT" file.

Use TXT It! if you want to create and upload a custom robots.txt file.

Upload Your Own robots.txt File

Be careful, this is dangerous. We do not recommend it, and we do not support it at SiteSell Support if you have problems or questions. This is geeky optional stuff, so you're on your own.

Solo Build It! automatically creates a properly formatted robots.txt file containing the following information for search engines (the sitemap file below is an example only)...

Sitemap: https://www.anguilla-beaches.com/sitemap.xml

User-agent: *
Disallow: /cgi-bin/
Disallow: /bin/
Disallow: /dyn/
Disallow: /objects/
Disallow: /help/
Disallow: /no-such-url.html
Disallow: /fb/channel.html

User-agent: Googlebot
Disallow: /cgi-bin/
Disallow: /dyn/
Disallow: /help/
Disallow: /no-such-url.html
Disallow: /fb/channel.html

User-Agent: Googlebot-Mobile
Disallow: /cgi-bin/
Disallow: /bin/
Disallow: /dyn/
Disallow: /help/
Disallow: /no-such-url.html
Disallow: /fb/channel.html

User-Agent: Googlebot-Image
Disallow: /cgi-bin/
Disallow: /bin/
Disallow: /dyn/
Disallow: /help/
Disallow: /no-such-url.html
Disallow: /fb/channel.html

User-agent: Mediapartners-Google
Disallow: /cgi-bin/
Disallow: /bin/
Disallow: /dyn/
Disallow: /help/
Disallow: /fb/channel.html
Disallow: /no-such-url.html

User-agent: bingbot
Disallow: /cgi-bin/
Disallow: /bin/
Disallow: /dyn/
Disallow: /objects/
Disallow: /help/
Disallow: /no-such-url.html
Disallow: /fb/channel.html
Crawl-delay: 30

User-agent: msnbot
Disallow: /cgi-bin/
Disallow: /bin/
Disallow: /dyn/
Disallow: /objects/
Disallow: /help/
Disallow: /no-such-url.html
Disallow: /fb/channel.html
Crawl-delay: 30

User-agent: Slurp
Disallow: /cgi-bin/
Disallow: /bin/
Disallow: /dyn/
Disallow: /objects/
Disallow: /help/
Disallow: /no-such-url.html
Disallow: /fb/channel.html
Crawl-delay: 30

If you upload your own robots.txt file, you will overwrite the SBI! file. Please note -- if you do create your own robots.txt file, it must start with the above text.

You may add to it, at your own risk. But your uploaded robots.txt file must start with the above text, especially if you want the search engines to have quick access to your sitemap (we ping the engines with your sitemap every time you change your site content, but a reference to it in the robots.txt file is also helpful).

It's dangerous to play with robots.txt. However, if you have a special reason to do so, please make sure you know what you're doing. Click here for Robotstxt.org's definitive information about dealing with robots.

Also, if you uploaded your own version, any updates that we make to the automatic version will not be added to your uploaded file.

Important Note About Google

If you've verified ownership of your site with Google Search Console, you have access to a lot of information there. One thing you'll find on your dashboard is URLs "Restricted by robots.txt."

Check through them. Any link with SBI!'s referral code (used for Click-Through Data in Site Central's TrafficCenter) contains "/cgi-bin/." This is the "cause" of the problem with Google.

You can ignore all of the links listed here with the referral code that contains "/cgi-bin/."

You'll also see other restricted URLs for the "/dyn/" folder. Google does not have access to that folder, and accurately reports it as restricted.