Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: Cloud Spider 3.02
-
Fix Version/s: Cloud Spider 3.02
-
Component/s: None
-
Labels:None
Description
Site: http://www.sportchalet.com
The spider is looking into the "crawl-delay" directive in the robots.txt, even when honor robots is disabled. As the crawl-delay is too high, it decides not to crawl it.
