|

|
About Setoozbot
Setoozbot obeys Robots Exclusion Protocol (robots.txt). If you do not want Setoozbot to crawl your website please add the appropriate Disallow entries in robots.txt file with User-Agent name as "Setoozbot". For more help on adding entries to robots.txt, please look here.
We try our best to conduct polite crawls. However, you face any problem with our crawler, please send an e-mail to bot at] setooz.com
FAQ
What is Setoozbot? Why is it crawling my website?
Setoozbot is a web crawler looking to index non-English web pages on the internet for it's search engine SETOOZ (Setu in sanskrit means a "Bridge". Setooz pronounced as "say-th-uuz", means bridges) . In the process Setoozbot may also crawl English web-pages or web-pages which are linked through non-English web-pages.
How do I stop Setoozbot from crawling my website or a portion of my website?
For example to disallow crawling on portions of website, add the
following lines to your robots.txt. Use a '*' for User-agent to block
all crawlers including Setoozbot. In order to block all folders of your website except 'robots.txt' add '/' for Disallow .
User-agent: OOZBOT/SETOOZBOT/oozbot/setoozbot
Disallow: [folder-names]
I added the lines in my robots.txt but Setoozbot seems to be still crawling my website.
Once the changes are made to the file, the crawler would not crawl the
website for all further crawls. If a crawl is currently in progress, the
crawler has only seen the older version of your robots.txt. Please
allow some time for the crawler to load your changes.
I am unhappy with your crawler. What do I do?
Immediately contact us at [bot] (at) setooz.com and let us know. Your feedback is very valuable to us.
Submit Your Site |
About Us |
Setoozbot |
Feedback
©2010 SETU Software Systems - Privacy
| |