Google sepull3 failures - more retries and limitations on # of current threads needed in v201109 upgrade

Early failures in sepull3.sh daily pulls which succeed after ITC retries. This happened almost every day.

Caitlyn states:

This will be resolved in the Google v201109 upgrade by adding retry logic to googlemcc and limiting the # of concurrent threads. For now, data-monitoring is taking care of it manually.

Exceptions in sepull3.sh_d_3_T_2012_02_02__timezoneid_1,13,8_03:57:05.log

2012-02-03 03:59:20.580 (3) [P3T1]: Caught other/unrecoverable API Exception: "com.google.api.adwords.v201101.cm.ApiException"

2012-02-03 03:59:33.432 (3) [P3T4]: Caught other/unrecoverable API Exception: "com.google.api.adwords.v201101.cm.ApiException"

2012-02-03 04:00:42.534 (3) [P3T3]: Caught other/unrecoverable API Exception: "com.google.api.adwords.v201101.cm.ApiException"

2012-02-03 04:00:49.017 (3) [P3T2]: Caught other/unrecoverable API Exception: "com.google.api.adwords.v201101.cm.ApiException"

2012-02-03 04:01:01.752 (3) [P3T6]: Caught other/unrecoverable API Exception: "com.google.api.adwords.v201101.cm.ApiException"

2012-02-03 04:03:13.464 (1) [P3T1]: Exception [ Error parsing /var/local/tsa/sedata/google/tsa-mcc-report-2012-02-02-A591-U1774.xml.gz]:com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry '2012-02-02-6877200-241198110-25105521-299478870-unknown-Search N' for key 'PRIMARY'

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)

2012-02-03 03:59:17.248 (3) [P3T1]: Downloading report attempt 1: name=/var/local/tsa/sedata/google/tsa-mcc-placement-report-2012-02-02-A454-U1391.xml.gz wi

th id=477639493 from url https://adwords.google.com/api/adwords/reportdownload?__rd=477639493

2012-02-03 04:03:13.464 (1) [P3T1]: Exception [ Error parsing /var/local/tsa/sedata/google/tsa-mcc-report-2012-02-02-A591-U1774.xml.gz]:com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry '2012-02-02-6877200-241198110-25105521-299478870-unknown-Search N' for key 'PRIMARY'

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)

2012-02-03 03:59:19.551 (3) [P3T4]: Downloading report attempt 1: name=/var/local/tsa/sedata/google/tsa-mcc-keyword-report-2012-02-02-A458-U1391.xml.gz with

id=477639613 from url https://adwords.google.com/api/adwords/reportdownload?__rd=477639613

2012-02-03 04:00:40.052 (3) [P3T2]: Downloading report attempt 1: name=/var/local/tsa/sedata/google/tsa-mcc-placement-report-2012-02-02-A434-U1391.xml.gz wi

th id=477641533 from url https://adwords.google.com/api/adwords/reportdownload?__rd=477641533

2012-02-03 04:00:58.588 (3) [P3T6]: Downloading report attempt 1: name=/var/local/tsa/sedata/google/tsa-mcc-placement-report-2012-02-02-A435-U1391.xml.gz wi

th id=477642253 from url https://adwords.google.com/api/adwords/reportdownload?__rd=477642253

2012-02-03 04:00:39.292 (3) [P3T3]: Downloading report attempt 1: name=/var/local/tsa/sedata/google/tsa-mcc-keyword-report-2012-02-02-A431-U1391.xml.gz with

id=477641413 from url https://adwords.google.com/api/adwords/reportdownload?__rd=477641413

Reran

nohup ./sepull3.sh -d3 -T 2012-02-02 -a 454 -U 1391 --force 2> /tmp/2012-02-03/sepull3_a454_U1391_rerun.log &

=== Completed Account #454, Spark - LDSMingle – SUCCESS ===

nohup ./sepull3.sh -d3 -T 2012-02-02 -a 591 -U 1774 --force 2> /tmp/2012-02-03/sepull3_a591_U1774_rerun.log &

=== Completed Account #591, Guthy-Renker - Music Video Revolution – SUCCESS ===

nohup ./sepull3.sh -d3 -T 2012-02-02 -a 458 -U 1391 --force 2> /tmp/2012-02-03/sepull3_a458_U1391_rerun.log &

=== Completed Account #458, Spark - SingleParentsMingle – SUCCESS ===

nohup ./sepull3.sh -d3 -T 2012-02-02 -a 434 -U 1391 --force 2> /tmp/2012-02-03/sepull3_a434_U1391_rerun.log &

=== Completed Account #434, Spark - BBWPersonalsPlus – SUCCESS ===

nohup ./sepull3.sh -d3 -T 2012-02-02 -a 435 -U 1391 --force 2> /tmp/2012-02-03/sepull3_a435_U1391_rerun.log &

=== Completed Account #435, Spark - BlackSingles – SUCCESS ===

nohup ./sepull3.sh -d3 -T 2012-02-02 -a 431 -U 1391 --force 2> /tmp/2012-02-03/sepull3_a431_U1391_rerun.log &

=== Completed Account #431, Spark - AdventistSinglesConnection – SUCCESS ===