Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: Sustaining
-
Component/s: Cloud Spider
-
Labels:None
-
Sprint:Sprint 2
Description
S-4152
They are having an issue with spiders failing after crawling only 3-4k pages on www.Loeb.com.
This was from Abhi: “We don’t know the exact cause of this behavior. Initial attempts to reproduce this on local box failed so this might need more investigation. One thing we can try is launch a crawl with more than 100,000 pages so it will run on bigger node and terminate it after 20,000 pages or so.”
Jimmy was able to do this and get a report for 3k pages, but there were still almost 30k pages that weren’t spidered.
