[ar/local/tsa/leads/transformed] ExecProducer INFO Executing ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input-new7.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false] [ar/local/tsa/leads/transformed] ExecProducer INFO The command ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input-new7.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false] had exit value 0 [ar/local/tsa/leads/transformed] leads-data-loader-exec INFO STDOUT: 2015-02-10 11:13:55 INFO [main] org.springframework.context.support.ClassPathXmlApplicationContext:512 - Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@236acdd1: startup date [Tue Feb 10 11:13:55 UTC 2015]; root of context hierarchy 2015-02-10 11:13:55 INFO [main] org.springframework.beans.factory.xml.XmlBeanDefinitionReader:315 - Loading XML bean definitions from class path resource [merchant-main-spring-config.xml] 2015-02-10 11:13:55 INFO [main] org.springframework.beans.factory.xml.XmlBeanDefinitionReader:315 - Loading XML bean definitions from URL [jar:file:/usr/local/tsa/bidmgr-distribution-2.8.0-SNAPSHOT-dev-t20150130085958/libs/deployment-bidmgr-shaded-jar-2.8.0-SNAPSHOT.jar!/merchant/datasource-configuration.xml] 2015-02-10 11:13:56 INFO [main] org.springframework.beans.factory.xml.XmlBeanDefinitionReader:315 - Loading XML bean definitions from URL [jar:file:/usr/local/tsa/bidmgr-distribution-2.8.0-SNAPSHOT-dev-t20150130085958/libs/deployment-bidmgr-shaded-jar-2.8.0-SNAPSHOT.jar!/merchant/summarizer-configuration.xml] 2015-02-10 11:13:56 INFO [main] org.springframework.beans.factory.xml.XmlBeanDefinitionReader:315 - Loading XML bean definitions from URL [jar:file:/usr/local/tsa/bidmgr-distribution-2.8.0-SNAPSHOT-dev-t20150130085958/libs/deployment-bidmgr-shaded-jar-2.8.0-SNAPSHOT.jar!/dao/merchant-performance-dao-configuration.xml] 2015-02-10 11:13:56 INFO [main] org.springframework.beans.factory.config.PropertyPlaceholderConfigurer:172 - Loading properties file from class path resource [merchant/datasource.properties] 2015-02-10 11:13:56 INFO [main] org.springframework.beans.factory.config.PropertyPlaceholderConfigurer:172 - Loading properties file from class path resource [merchant/datasource-overrides.properties] 2015-02-10 11:13:56 WARN [main] org.springframework.beans.factory.config.PropertyPlaceholderConfigurer:181 - Could not load properties from class path resource [merchant/datasource-overrides.properties]: class path resource [merchant/datasource-overrides.properties] cannot be opened because it does not exist 2015-02-10 11:13:56 INFO [main] org.springframework.beans.factory.support.DefaultListableBeanFactory:603 - Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@293b9fae: defining beans [merchant-datasource-property-configurer,merchant-target-datasource,merchant-datasource,merchant-transaction-manager,org.springframework.aop.config.internalAutoProxyCreator,org.springframework.transaction.annotation.AnnotationTransactionAttributeSource#0,org.springframework.transaction.interceptor.TransactionInterceptor#0,org.springframework.transaction.config.internalTransactionAdvisor,merchant-performance-dao,merchant-summarizer]; root of factory hierarchy [ar/local/tsa/leads/transformed] leads-data-loader-exec INFO STDERR: 2015-02-10 11:13:54.635 (3) [main]: Debug Level set to 4 2015-02-10 11:13:54.636 (3) [main]: Starting Data Loader. args [[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input-new7.txt, 1]] 2015-02-10 11:13:54.639 (3) [main]: ======== Starting Call Data Load ======== 2015-02-10 11:13:54.639 (3) [main]: Loading filename: /var/local/tsa/leads/transformed/call-leads-test-input-new7.txt 2015-02-10 11:13:54.721 (4) [main]: registered MBean [thesearchagency.db.util:type=MultiplexDatabasePoolb659bb21-05c0-41f3-a72c-3444b1109eb8] for class com.thesearchagency.db.util.MultiplexDatabasePool 2015-02-10 11:13:54.730 (4) [main]: no account, or instance for databaseInstance, skipping lookup... 2015-02-10 11:13:54.734 (4) [main]: registered MBean [com.carldunham.jst.pooling:type=ObjectPool31007df8-e51f-42ed-ad94-b027362521df] for class com.carldunham.jst.pooling.ObjectPool 2015-02-10 11:13:54.734 (3) [main]: DatabasePool with a limit of 80 created 2015-02-10 11:13:54.735 (4) [main]: registered MBean [com.carldunham.jst.db:type=DatabasePoolc6597aeb-b37b-4bc2-aabd-c38baf241baf] for class com.carldunham.jst.db.DatabasePool 2015-02-10 11:13:54.736 (4) [ConnectionCloser]: Starting DatabasePool ConnectionCloserThread 2015-02-10 11:13:55.010 (3) [main]: JDBC Driver Version: 5.1 (com.mysql.jdbc.Driver) 2015-02-10 11:13:55.011 (4) [main]: opened [jdbc:mysql://tsacommon-01-write/tsacommon?useUnicode=true&zeroDateTimeBehavior=convertToNull&dontTrackOpenResources=true&jdbcCompliantTruncation=false&useServerPrepStmts=false&rewriteBatchedStatements=true] 2015-02-10 11:13:55.011 (4) [main]: Created new Object 2015-02-10 11:13:55.013 (4) [main]: spike: SQL->[select @@version] 2015-02-10 11:13:55.018 (4) [main]: time=0.0050 2015-02-10 11:13:55.018 (4) [main]: Query executed in 0.005s 2015-02-10 11:13:55.019 (4) [main]: test query succeeded, returned "5.5.33" 2015-02-10 11:13:55.019 (4) [main]: spike: SQL->[select id, description, databaseInstance from accounts where isActive = "true"] 2015-02-10 11:13:55.036 (4) [main]: time=0.017 2015-02-10 11:13:55.037 (4) [main]: Query executed in 0.017s 2015-02-10 11:13:55.096 (4) [main]: Returning Object to Pool 2015-02-10 11:13:55.098 (4) [main]: no account, or instance for databaseInstance, skipping lookup... 2015-02-10 11:13:55.098 (4) [main]: Took Existing Object from Pool 2015-02-10 11:13:55.099 (4) [main]: spike: SQL->[select @@version] 2015-02-10 11:13:55.101 (4) [main]: time=0.0020 2015-02-10 11:13:55.101 (4) [main]: Query executed in 0.002s 2015-02-10 11:13:55.101 (4) [main]: test query succeeded, returned "5.5.33" 2015-02-10 11:13:55.101 (4) [main]: preparing SQL statement [insert into callFeedExecution (fileName, runTimestamp, status) values (?, ?, ?) on duplicate key update runTimestamp=?, status=?, firstDate=null, lastDate=null, numDataLines=0, numDataLinesWithNegatives=0, numInvalidDataLines=0], genflags=yes 2015-02-10 11:13:55.115 (4) [main]: Returning Object to Pool 2015-02-10 11:13:55.120 (4) [main]: Account: 655eb91d-a388-412e-af18-2b1417cddbdf, date: 2014-01-12, calls: 22 2015-02-10 11:13:55.121 (1) [main]: Exception [invalid line #2]:com.thesearchagency.datafeeds.calldata.exception.InvalidLineException: invalid call value [23.5] at com.thesearchagency.datafeeds.calldata.DataLoader.validateCalls(DataLoader.java:283) at com.thesearchagency.datafeeds.calldata.DataLoader.handleDataRow(DataLoader.java:236) at com.thesearchagency.datafeeds.calldata.DataLoader.parseFile(DataLoader.java:221) at com.thesearchagency.datafeeds.calldata.DataLoader.run(DataLoader.java:117) at com.thesearchagency.datafeeds.calldata.DataLoader.main(DataLoader.java:60) Caused by: java.lang.NumberFormatException: For input string: "23.5" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:458) at java.lang.Integer.parseInt(Integer.java:499) at com.thesearchagency.datafeeds.calldata.DataLoader.validateCalls(DataLoader.java:280) ... 4 more 2015-02-10 11:13:55.132 (4) [main]: Account: 40e7409e-ee8a-48bf-96f9-a869d34bfd7e, date: 2014-12-02, calls: 24 2015-02-10 11:13:55.133 (4) [main]: no account, or instance for databaseInstance, skipping lookup... 2015-02-10 11:13:55.163 (3) [main]: JDBC Driver Version: 5.1 (com.mysql.jdbc.Driver) 2015-02-10 11:13:55.163 (4) [main]: opened [jdbc:mysql://tsacommon-01-write/tsacommon?useUnicode=true&zeroDateTimeBehavior=convertToNull&dontTrackOpenResources=true&jdbcCompliantTruncation=false&useServerPrepStmts=false&rewriteBatchedStatements=true] 2015-02-10 11:13:55.174 (4) [main]: spike: SQL->[select * from `tsacommon`.`accounts` where (`id`=1) limit 1] 2015-02-10 11:13:55.176 (4) [main]: time=0.0020 2015-02-10 11:13:55.176 (4) [main]: Query executed in 0.002s 2015-02-10 11:13:55.182 (4) [main]: Record Loaded 2015-02-10 11:13:55.183 (4) [main]: DataCache: put key acct-st-tracker 1 2015-02-10 11:13:55.185 (4) [main]: spike: SQL->[select * from `tsacommon`.`databaseInstances` where ((`name`="acct-st-tracker-1") and (`type`="mysql")) limit 1] 2015-02-10 11:13:55.188 (4) [main]: time=0.0030 2015-02-10 11:13:55.188 (4) [main]: Query executed in 0.003s 2015-02-10 11:13:55.189 (4) [main]: Record Loaded 2015-02-10 11:13:55.189 (4) [main]: DataCache: put key acct-st-tracker-1 2015-02-10 11:13:55.190 (4) [main]: closed [jdbc:mysql://tsacommon-01-write/tsacommon?useUnicode=true&zeroDateTimeBehavior=convertToNull&dontTrackOpenResources=true&jdbcCompliantTruncation=false&useServerPrepStmts=false&rewriteBatchedStatements=true] 2015-02-10 11:13:55.191 (4) [main]: registered MBean [com.carldunham.jst.pooling:type=ObjectPool10fd4e74-b238-4d13-9726-761a74b4d53b] for class com.carldunham.jst.pooling.ObjectPool 2015-02-10 11:13:55.191 (3) [main]: DatabasePool with a limit of 80 created 2015-02-10 11:13:55.191 (4) [main]: registered MBean [com.carldunham.jst.db:type=DatabasePoolff00e115-6b0a-4ab7-afe0-a20d2f602463] for class com.carldunham.jst.db.DatabasePool 2015-02-10 11:13:55.217 (3) [main]: JDBC Driver Version: 5.1 (com.mysql.jdbc.Driver) 2015-02-10 11:13:55.217 (4) [main]: opened [jdbc:mysql://acctdb-01-write:3306/st-tracker?useUnicode=true&zeroDateTimeBehavior=convertToNull&dontTrackOpenResources=true&jdbcCompliantTruncation=false&useServerPrepStmts=false&rewriteBatchedStatements=true] 2015-02-10 11:13:55.218 (4) [main]: Created new Object 2015-02-10 11:13:55.218 (4) [main]: spike: SQL->[select @@version] 2015-02-10 11:13:55.219 (4) [main]: time=0.0010 2015-02-10 11:13:55.219 (4) [main]: Query executed in 0.001s 2015-02-10 11:13:55.219 (4) [main]: test query succeeded, returned "5.5.33" 2015-02-10 11:13:55.220 (4) [main]: DB AutoCommit -> false 2015-02-10 11:13:55.220 (4) [main]: spike: SQL->[start transaction] 2015-02-10 11:13:55.221 (4) [main]: time=0.0010 2015-02-10 11:13:55.222 (4) [main]: Query executed in 0.001s 2015-02-10 11:13:55.222 (4) [main]: preparing SQL statement [delete from callFeedData where callFeedExecutionID =?] 2015-02-10 11:13:55.224 (3) [main]: [Shard 1] Deleted 2 rows of existing call data for call-leads-test-input-new7.txt (execution id=83), assuming rerun 2015-02-10 11:13:55.256 (4) [main]: preparing SQL statement [INSERT INTO callFeedData(callFeedExecutionID,accountID,accountDescription,date,numCalls) VALUES (?,?,?,?,?)] 2015-02-10 11:13:55.261 (4) [main]: DB AutoCommit -> false 2015-02-10 11:13:55.262 (4) [main]: spike: SQL->[start transaction] 2015-02-10 11:13:55.264 (4) [main]: time=0.0020 2015-02-10 11:13:55.264 (4) [main]: Query executed in 0.002s 2015-02-10 11:13:55.268 (4) [main]: DB AutoCommit -> true 2015-02-10 11:13:55.268 (4) [main]: DB AutoCommit -> false 2015-02-10 11:13:55.269 (3) [main]: [Shard 1] Loaded data with file /var/local/tsa/tmp/calldata_shard_1_208224067739975353.tsv containing 2 rows of data 2015-02-10 11:13:55.270 (4) [main]: preparing SQL statement [insert into `products` (productCode, accountID, description, countConversions, columnLabel, showConv) select ?, accounts.id, ?, 'false', ?, 'true' from `tsacommon`.`accounts` left join callFeedProductSourceMap on (accountID=accounts.id) where accountID is null and databaseInstance=?] 2015-02-10 11:13:55.277 (4) [main]: spike: SQL->[update `clientProductSummary` sum join callFeedProductSourceMap map on map.accountid=sum.accountid and map.sourceID=sum.sourceID and map.productID=sum.productID set count=0, cost=0, value=0 where startTime >= '2014-01-12' and startTime <= '2014-12-02' and resolution=86400], genflags=no 2015-02-10 11:13:55.280 (4) [main]: time=0.0030 2015-02-10 11:13:55.280 (4) [main]: Query executed in 0.003s, 4 row(s) affected 2015-02-10 11:13:55.280 (4) [main]: spike: SQL->[insert into `clientProductSummary` (sourceID, searchEngineListingID, productID, accountID, startTime, resolution, count) select sourceID, 0, productID, data.accountID, data.date, 86400, @sumOfCalls := sum(numCalls) from callFeedData data join callFeedProductSourceMap map on (data.accountID = map.accountID) where date >= '2014-01-12' and date <= '2014-12-02' group by data.accountid, data.date on duplicate key update count=@sumOfCalls], genflags=no 2015-02-10 11:13:55.282 (4) [main]: time=0.0020 2015-02-10 11:13:55.282 (4) [main]: Query executed in 0.002s, 4 row(s) affected 2015-02-10 11:13:55.282 (4) [main]: [Shard 1] Inserted or updated 4 rows into `st-tracker`.`clientProductSummary` table (cleared 4 existing rows) 2015-02-10 11:13:57,038 [main] INFO jdbc.sqlonly - UPDATE `merchant_performance` SET `call_leads` = 0 WHERE `start_date` >= '01/12/2014 00:00:00.000' AND `start_date` < '12/03/2014 00:00:00.000' 2015-02-10 11:13:57,045 [main] INFO c.t.p.m.dao.MerchantPerformanceDAO - clearDailyCallData ran, affected 10 rows 2015-02-10 11:13:57,047 [main] INFO jdbc.sqlonly - INSERT INTO `merchant_performance`(`merchant_uuid`, `start_date`, `call_leads`) SELECT `data`.`accountDescription`, `data`.`date`, SUM(`numCalls`) FROM `st-tracker`.`callFeedData` as `data` WHERE `data`.`date` >= '01/12/2014 00:00:00.000' AND `data`.`date` < '12/03/2014 00:00:00.000' GROUP BY `data`.`accountID`, `data`.`date` ON DUPLICATE KEY UPDATE `call_leads` = values(`call_leads`) 2015-02-10 11:13:57,049 [main] INFO c.t.p.m.dao.MerchantPerformanceDAO - addDailyCallData ran, affected 4 rows 2015-02-10 11:13:57.052 (4) [main]: spike: SQL->[commit] 2015-02-10 11:13:57.053 (4) [main]: time=0.0010 2015-02-10 11:13:57.053 (4) [main]: Query executed in 0.001s 2015-02-10 11:13:57.053 (4) [main]: DB AutoCommit -> true 2015-02-10 11:13:57.054 (4) [main]: Returning Object to Pool 2015-02-10 11:13:57.054 (4) [main]: no account, or instance for databaseInstance, skipping lookup... 2015-02-10 11:13:57.055 (4) [main]: Took Existing Object from Pool 2015-02-10 11:13:57.055 (4) [main]: spike: SQL->[select @@version] 2015-02-10 11:13:57.056 (4) [main]: time=0.0010 2015-02-10 11:13:57.056 (4) [main]: Query executed in 0.001s 2015-02-10 11:13:57.056 (4) [main]: test query succeeded, returned "5.5.33" 2015-02-10 11:13:57.056 (4) [main]: preparing SQL statement [update callFeedExecution set firstDate=?, lastDate=?, numDataLines=?, numDataLinesWithNegatives=?, numInvalidDataLines=?, status=? where id=?] 2015-02-10 11:13:57.059 (4) [main]: Returning Object to Pool 2015-02-10 11:13:57.059 (3) [main]: Lines processed: 2 2015-02-10 11:13:57.059 (3) [main]: Invalid lines: 1 2015-02-10 11:13:57.059 (3) [main]: ======== Finished Call Data Load (COMPLETED) ======== [ar/local/tsa/leads/transformed] leads-data-loader-exec INFO The leads data loading command succeeded. result is: 0