Spider: Issues in crawls with "Honor NoIndex" and "Honor No Follow" options

Test Env: client-pc.ri.thesearchagency.com

Prerequisites:
Create four links in page
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/indexfollowlink.htm"

------------------------
1.
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/noindex_nofollow.htm"
with the meta tag
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
and their link in their page to
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S1.htm"

2.
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/noindex_follow.htm"
with the meta tag
<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
and their link in their page to
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S2.htm"

3.
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/index_nofollow.htm"
with the meta tag
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
and their link in their page to
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S3.htm"

4.
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/index_follow.htm"
with the meta tag
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
and their link in their page to
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S4.htm"
------------------------

Steps:
1. Log into AdMax application.
2. Navigate to SEO section, click on "Spider"
3. Enter a valid URL to spider say
"http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/indexfollowlink.htm"

Case:1 Ensure "Honor NoIndex" and "Honor No Follow" check box is checked and
spider it

Observation:

• In "ADR - Spidered Urls_1", the link
  "http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S3.htm" which is
  in page
  "http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/index_nofollow.htm"
  is listed

Case:2 Ensure "Honor NoIndex" is unchecked and "Honor No Follow" is checked and
spider it

Observation:

• In "ADR - Spidered Urls_1", the link
  "http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S3.htm" which is
  in page
  "http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/index_nofollow.htm"
  is listed
• The "Spider Crawl Details inofrmation" dialog box shows "URLs Not Allowed to
  Spider: 0", but the url
  "http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S1.htm" is not
  listed in "ADR - Spidered Urls_1" as expected

Case:3 Ensure "Honor NoIndex" is checked and "Honor No Follow" is unchecked and
spider it

Observation:

• The "Spider Crawl Details inofrmation" dialog box shows "URLs Not Allowed to
  Spider: 2" as expected, but in "ADR - Spidered Urls_1", the link
  "http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/Spider/S1.htm" which is
  in page
  "http://pvwb-of1pvd0010.ri.thesearchagency.com/gurpreet/noSpider/index_nofollow.htm"
  is not listed

Case:4 Ensure "Honor NoIndex" is unchecked and "Honor No Follow" is unchecked
and spider it

Observation:

• In "ADR - Spidered Urls_1", all the pages are listed which is expected