DA status not being set in WarehouseSummarizer for STAT WH when an account does not have Position data

On Data Availability page in production, it's been observed recently that STAT WH is left as 'not started' for some accounts, but not all accounts.

After researching it seems to be related to whether the account has position data or not. Here is the offending code in WarehouseSummarizer.java:

if (theDoOrganicPositionFlag && theAliveFlag && haveOrganicPositionData()) {

// By default, limit to organic distributions processed by the summarizer:

if (theDistributions == null)

{ theDistributions = "(" + ORGANIC_SE_DISTS + ")"; Debug.msg(Debug.INFORMATION, "Processing limited to organic SE distributions: " + theDistributions); }

type = DataSourceTable.TypeValue.WH_POSITION;

setDAInProgress(type);

if (clearPosition() > 0) datachange = true;

boolean success = processOrganicPositionData();

if (!success) setDAError(type);

else datachange = true;

}

If haveOrganicPositionData() is false it skips all this code. In setDAInProgress(type), it finds the data sources for that account and type. If this code is skipped, the data sources are not found. Yet the code will still make changes to the warehouse tables, so DA should be updated and therefore it seems setDAInProgress should be called whether or not an account has position data. Also, I believe this holds up running WH LOW.

I've attached log files for two accounts: 652 (Move.com - Moving.com), which DA status stays 'not started' and 147 (Move.com - Move Brand), which DA status gets updated to 'success' for 2011-08-31.

Here are the commands that got run:

nohup $DIR/whsumm.sh -d5 --position --highgran --date 2011-08-31 -a 652 2> /tmp/2011-09-01/whsumm.sh-statpull_a_652_rerun.log &

nohup $DIR/whsumm.sh -d5 --position --highgran --date 2011-08-31 -a 147 2> /tmp/2011-09-01/whsumm.sh-statpull_a_147_rerun.log &

In each, search for 'select * from `sourcePositionSummary`' and compare the output from there. Here is the query that checks if position data is available:

select * from `sourcePositionSummary` where (((`sourcePositionSummary`.`startTime`="2011-08-31 00:00:00"))

and (`sourcePositionSummary`.`accountID`=652)) limit 1

If it is available, you will see 'Processing limited to organic SE distributions: (86,101,122)' and then queries to get dataSources and update dataAvailability, like so:

select * from `dataSources` where ((`accountID`=147) and (`type`="wh_position")) limit 1

replace into `dataAvailability` (`oldUpdated`,`needsRerun`,`statusText`,`override`,`dataSourceID`,`status`

,`date`,`updated`,`id`) values ("00:00:00","false",null,"false",51852,"in progress","2011-08-31","2011-09-01 08:16:09",60513572)

If it is not available it skips that, finishes Daily and goes straight to updating the warehouse tables like so:

2011-09-01 08:12:10.129 (3) [P1T1]: Finished daily for Account #652 Move.com - Moving.com on Wed Aug 31 00:00:00 PDT 2011

2011-09-01 08:12:10.130 (3) [P1T1]: Starting Account #652 Move.com - Moving.com daily group aggregates for Wed Aug 31 00:00:00 PDT 2011 to <Thu Sep 01 00:00:

00 PDT 2011

2011-09-01 08:12:10.180 (4) [P1T1]: perfsum: SQL->[delete from `warehouse_7`.`whDailyGroupConv` where ((`accountID`=652) and ((((`startTime`>="2011-08-31 00:

00:00")) and ((`startTime`<"2011-09-01 00:00:00")))))], genflags=no