ADR v3 Spider - Some robots denied urls are misflagged as Missing Title Tags

All these URLs are "robots denied" they are not fetched, and do not have any response codes.

However, they should not be flagged as "missing title tags" in the ADR, which is a bug.

-Antony.

----------

The ADR is failing to flag URLs as 302s:

S-1645

S-1245

Examples under Missing Title Tags:

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8xLTUtOC1laWdodGgtaW5jaC14LTctOC1laWdodGgtaW5jaC1saWdodC1kdXR5LXN3aXZlbC1jYXN0ZXItNDE1MTkuaHRtbA,,/product/3653/3653

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8xMC1pbmNoLXBuZXVtYXRpYy1yaWdpZC1jYXN0ZXItMzg5NDMuaHRtbA,,/product/1157/1157

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8xMC1pbmNoLXBuZXVtYXRpYy1yaWdpZC1jYXN0ZXItMzg5NDMuaHRtbA,,/product/402/402

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8xMC1pbmNoLXBuZXVtYXRpYy1yaWdpZC1jYXN0ZXItMzg5NDMuaHRtbA,,/product/843/843

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8xMC1pbmNoLXBuZXVtYXRpYy1zd2l2ZWwtY2FzdGVyLTM4OTQ0Lmh0bWw,/product/402/402

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8xMC1pbmNoLXBuZXVtYXRpYy1zd2l2ZWwtY2FzdGVyLTM4OTQ0Lmh0bWw,/product/4419/4419

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8xMC1pbmNoLXBuZXVtYXRpYy1zd2l2ZWwtY2FzdGVyLTM4OTQ0Lmh0bWw,/product/842/842

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8yLWluY2gtY2xlYXItcG9seXVyZXRoYW5lLXN3aXZlbC1jYXN0ZXItNjUxODQuaHRtbA,,/product/1141/1141

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8yLWluY2gtY2xlYXItcG9seXVyZXRoYW5lLXN3aXZlbC1jYXN0ZXItNjUxODQuaHRtbA,,/product/2265/2265

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8yLWluY2gteC03LTgtZWlnaHRoLWluY2gtbGlnaHQtZHV0eS1zd2l2ZWwtY2FzdGVyLTQxNTE4Lmh0bWw,/product/2672/2672

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8yLWluY2gteC03LTgtZWlnaHRoLWluY2gtbGlnaHQtZHV0eS1zd2l2ZWwtY2FzdGVyLTQxNTE4Lmh0bWw,/product/3246/3246

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8yLWluY2gteC03LTgtZWlnaHRoLWluY2gtbGlnaHQtZHV0eS1zd2l2ZWwtY2FzdGVyLTQxNTE4Lmh0bWw,/product/3356/3356

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLTEtMi1oYWxmLWluY2gteC0xLTEtNC1xdWFydGVyLWluY2gtbWVkaXVtLWR1dHktc3dpdmVsLWNhc3Rlci00MTUyNC5odG1s/product/1460/1460

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2FzdC1pcm9uLW1vbGRlZC1ydWJiZXItc3dpdmVsLWNhc3Rlci0zODcwOS5odG1s/product/3653/3653

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2FzdC1pcm9uLW1vbGRlZC1ydWJiZXItc3dpdmVsLWNhc3Rlci0zODcwOS5odG1s/product/818/818

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2FzdC1pcm9uLW1vbGRlZC1ydWJiZXItc3dpdmVsLWNhc3Rlci0zODcwOS5odG1s/product/822/822

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2xlYXItc3dpdmVsLWNhc3Rlci13aXRoLWJyYWtlLTY1MTk0Lmh0bWw,/product/1598/1598

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2xlYXItc3dpdmVsLWNhc3Rlci13aXRoLWJyYWtlLTY1MTk0Lmh0bWw,/product/4419/4419

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2xlYXItc3dpdmVsLWNhc3Rlci13aXRoLWJyYWtlLTY1MTk0Lmh0bWw,/product/821/821

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2xlYXItcG9seXVyZXRoYW5lLXN3aXZlbC1jYXN0ZXItNjUxOTAuaHRtbA,,/product/1262/1262

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2xlYXItcG9seXVyZXRoYW5lLXN3aXZlbC1jYXN0ZXItNjUxOTAuaHRtbA,,/product/1435/1435

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtY2xlYXItcG9seXVyZXRoYW5lLXN3aXZlbC1jYXN0ZXItNjUxOTAuaHRtbA,,/product/1599/1599

http://www.harborfreight.com/checkout/cart/add/uenc/aHR0cDovL3d3dy5oYXJib3JmcmVpZ2h0LmNvbS9tYXRlcmlhbC1oYW5kbGluZy9jYXN0ZXJzLXN3aXZlbC8zLWluY2gtbnlsb24tc3dpdmVsLWNhc3Rlci05NDUyMS5odG1s/product/2053/2053