About Setoozbot

Setoozbot obeys Robots Exclusion Protocol (robots.txt). If you do not want Setoozbot to crawl your website please add the appropriate Disallow entries in robots.txt file with User-Agent name as "Setoozbot". For more help on adding entries to robots.txt, please look here.

We try our best to conduct polite crawls. However, you face any problem with our crawler, please send an e-mail to bot at] setooz.com

FAQ

What is Setoozbot? Why is it crawling my website?
Setoozbot is a web crawler looking to index non-English web pages on the internet for it's search engine SETOOZ (Setu in sanskrit means a "Bridge". Setooz pronounced as "say-th-uuz", means bridges) . In the process Setoozbot may also crawl English web-pages or web-pages which are linked through non-English web-pages.

How do I stop Setoozbot from crawling my website or a portion of my website?
For example to disallow crawling on portions of website, add the following lines to your robots.txt. Use a '*' for User-agent to block all crawlers including Setoozbot. In order to block all folders of your website except 'robots.txt' add '/' for Disallow .

User-agent: OOZBOT/SETOOZBOT/oozbot/setoozbot
Disallow: [folder-names]

I added the lines in my robots.txt but Setoozbot seems to be still crawling my website.
Once the changes are made to the file, the crawler would not crawl the website for all further crawls. If a crawl is currently in progress, the crawler has only seen the older version of your robots.txt. Please allow some time for the crawler to load your changes.

I am unhappy with your crawler. What do I do?
Immediately contact us at [bot] (at) setooz.com and let us know. Your feedback is very valuable to us.