Spider: Decrease time spent on a single URL

Note: Desired behavior is not yet completely defined - Ted

Ted Ives
Sent: Wednesday, March 24, 2010 9:13 AM
To:
Mark Fillmore
Cc:
Mark Podhajsky
Attachments:
Ideally the spider ought to maintain a list, and then at the end of its run,
give those pages one more shot or something.

Either way, doing something about this makes a ton of sense, will add it to the
list.

Ted
________________________________________
From: Mark Fillmore
Sent: Tuesday, March 23, 2010 7:31 PM
To: Ted Ives
Cc: Mark Podhajsky
Subject: FW: Slow spidering

Hey Ted,

I need to add an item to the ADR list.

We need to decrease the wait time we’ll spend on a single URL. After 30 seconds
we need to flag the URL as non-responsive, and move on. It would be great if we
could add these URLs to their own tab.

Please add this to our list - if we can make this one of the five items we
tackle over the next month, I would place this in spot #2.

Thanks,