CLONE -sepull: automatically retry if parsing report fails

There is support for automatic retries if downloading the report from Google fails, but not for retrying if parsing the report fails.

Retries need to be redundant like they were in v13 report downloads. If the report download fails (due to network issue / google being down / corrupt download file) retry 3 times. If parsing the downloaded file fails (out of memory error, temporary database connection issue) retry. If either part exceeds the retry limit (due to Google generating a corrupt file, for example) request a new report and retry until successful.

Pseudo code illustrating this:

function main():

while (retry)

downloadAndParseReport()

function downloadAndParseReport():

while (retry)

downloadReport()

while (retry)

parseReport()

These duplicate key exceptions are causing the SEU to fail, so data isn't getting summarized

2012-02-08 03:58:12.155 (3) [P1T6]: Deleting existing records where ((`startTime`="2012-02-07 00:00:00"))

2012-02-08 03:58:13.710 (1) [P1T6]: Exception [ Error parsing /var/local/tsa/sedata/google/tsa-mcc-report-2012-02-07-A456-U1391.xml.gz]:com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Du

plicate entry '2012-02-07-698302-243034693-19911182-333481333-unknown-Search Ne' for key 'PRIMARY'

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)

at com.mysql.jdbc.Connection.execSQL(Connection.java:3170)

at com.mysql.jdbc.Statement.executeUpdate(Statement.java:1316)

at com.mysql.jdbc.Statement.executeUpdate(Statement.java:1235)

at com.mysql.jdbc.Statement.executeUpdate(Statement.java:1388)

at com.carldunham.jst.db.Database.executeUpdate(Database.java:401)

at com.carldunham.jst.db.BaseDatabaseTable.insert(BaseDatabaseTable.java:366)

at com.carldunham.jst.db.BaseDatabaseTable.insert(BaseDatabaseTable.java:354)

at com.thesearchagency.searchengines.google.GoogleAPISearchEngine.parseReport(GoogleAPISearchEngine.java:1856)

at com.thesearchagency.searchengines.google.GoogleAPISearchEngine.parseReport(GoogleAPISearchEngine.java:1615)

at com.thesearchagency.searchengines.google.GoogleMCCReport$GoogleMCCReportAccountWorker.stageSearchEngineReport(GoogleMCCReport.java:198)

at com.thesearchagency.searchengines.common.SearchEngineDataStager$AccountWorker.run(SearchEngineDataStager.java:314)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

2012-02-08 03:58:13.711 (3) [P1T6]: ...done

mysql> select * from googleMccV2011 where startTime='2012-02-07' and campaignid=698302 and adgroupid=243034693 and keywordid=19911182 and creativeid=333481333;

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

1 row in set (0.02 sec)

I gunziped the report in my home dir on xml06 and I only see one row that matches that key, so I'm wondering why it's getting a duplicate key exception at all.. maybe it's not clearing that campaignid for some reason. Though I don't understand why it'd already have a row for 2012-02-07. Thoughts?

<row campaignID="698302" adGroupID="243034693" adID="333481333" keywordID="19911182" avgPosition="3.79" clicks="0" impressions="28" cost="0" conv1PerClick="0" network="Search Network"/>

On a manual rerun, no exceptions were thrown.

Code should handle this and automatically retry for x number of times.