Cloud Spider: Ignore Robots.txt Crawl-Delay Always

New SEO team request to ignore crawl-delay in robots.txt every time, regardless of honorRobots is true or false.

---Mail attached----

From: Mark Fillmore

Sent: Friday, September 16, 2011 10:35 AM

To: Antony Rajiv; Abhiram Bhagwat; David 'Otto' Rahmel

Cc: Autumn Francesca

Subject: RE: Still no Sport Chalet Love

But, for next week's patch, can we ignore the crawl delay command?

From: Antony Rajiv

Sent: Friday, September 16, 2011 10:29 AM

To: Abhiram Bhagwat; David 'Otto' Rahmel; Mark Fillmore

Cc: Autumn Francesca

Subject: RE: Still no Sport Chalet Love

We will add it to the ice box as a new feature (possibly with a UI checkbox option) and can discuss it for a future release.

Thanks,

-Antony.