Low Gran WH (EST) timing out for large periods of time

/tmp/2012-05-09/lowgran_whsumm.sh_d_3_T_2012_05_08_a_477,687,690,698,719,759,774,798,808,826,846_01:39:59.log

2012-05-09 03:41:50.547 (3) [P1T10]: Finished Account #477 Daily Candy aggregates for 2012-05-08 00:00:00 EDT to <2012-05-09 00:00:00 EDT

2012-05-09 03:41:50.579 (3) [P1T10]: Sending 'account done' JMS message for 477 for 2012-05-08 00:00:00 EDT

2012-05-09 11:40:56.630 (2) [main]: Timeout exceeded. Exiting...

2012-05-09 11:40:56.631 (2) [main]: Remaining account IDs: [759, 690, 798, 687, 774]

2012-05-09 11:40:56.650 (2) [main]: setDate should be implemented by children of AccountDispatcher.

2012-05-09 11:40:56.680 (2) [main]: setDate should be implemented by children of AccountDispatcher.

/tmp/2012-05-10/lowgran_whsumm.sh_d_3_T_2012_05_09_a_477,687,690,698,719,759,774,798,808,826,846_01:48:00.log

2012-05-10 02:02:10.963 (3) [P1T8]: Sending 'account done' JMS message for 846 for 2012-05-09 00:00:00 EDT

2012-05-10 02:02:11.195 (3) [P1T9]: ... 597 Monthly Group Conversion Records Inserted

2012-05-10 02:02:12.777 (3) [P1T9]: ... 7487 Monthly Group Attribute Records Inserted

2012-05-10 02:02:13.283 (3) [P1T9]: Finished Account #826 ACMG aggregates for 2012-05-09 00:00:00 EDT to <2012-05-10 00:00:00 EDT

2012-05-10 02:02:13.290 (3) [P1T9]: Sending 'account done' JMS message for 826 for 2012-05-09 00:00:00 EDT

2012-05-10 11:48:23.887 (2) [main]: Timeout exceeded. Exiting...

2012-05-10 11:48:23.888 (2) [main]: Remaining account IDs: [759, 690, 477, 798, 687, 774, 719]

2012-05-10 11:48:23.894 (2) [main]: dispatchAccount retry attempt: 1. Sleeping for 10 seconds.

2012-05-10 11:48:33.927 (2) [main]: setDate should be implemented by children of AccountDispatcher.

2012-05-10 11:48:33.942 (2) [main]: dispatchAccount retry attempt: 1. Sleeping for 10 seconds.

/tmp/2012-05-11/lowgran_whsumm.sh_d_3_T_2012_05_10_a_477,687,690,698,719,759,774,798,808,826,846_01:45:26.log

Today it is stalled at a retry from 5:48, but it started to stall at 2.

2012-05-11 02:01:42.380 (3) [P1T8]: Sending 'account done' JMS message for 846 for 2012-05-10 00:00:00 EDT

May 11, 2012 5:29:27 AM org.apache.cxf.phase.PhaseInterceptorChain doIntercept

INFO: Interceptor has thrown exception, unwinding now

org.apache.cxf.interceptor.Fault: Connection refused

at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:96)

at org.apache.cxf.interceptor.BareOutInterceptor.handleMessage(BareOutInterceptor.java:68)

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)

at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:469)

at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:299)

at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:251)

at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)

at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:124)

at $Proxy58.findDataAvailability(Unknown Source)

at com.thesearchagency.perf.AccountLowGranDispatcher$DataAvailabilityServiceWrapper.findDataAvailability(AccountLowGranDispatcher.java:790)

at com.thesearchagency.perf.AccountLowGranDispatcher.findAccountsWithStatusSuccess(AccountLowGranDispatcher.java:609)

at com.thesearchagency.perf.AccountLowGranDispatcher.runForKnownDate(AccountLowGranDispatcher.java:250)

at com.thesearchagency.perf.AccountLowGranDispatcher.run(AccountLowGranDispatcher.java:103)

at com.thesearchagency.perf.WarehouseSummarizer.run(WarehouseSummarizer.java:366)

at com.thesearchagency.perf.WarehouseSummarizer.main(WarehouseSummarizer.java:423)

Caused by: com.ctc.wstx.exc.WstxIOException: Connection refused

at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:313)

at org.apache.cxf.interceptor.AbstractOutDatabindingInterceptor.writeParts(AbstractOutDatabindingInterceptor.java:94)

... 14 more

Caused by: java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(Unknown Source)

at java.net.PlainSocketImpl.connectToAddress(Unknown Source)

at java.net.PlainSocketImpl.connect(Unknown Source)

at java.net.Socket.connect(Unknown Source)

at sun.net.NetworkClient.doConnect(Unknown Source)

at sun.net.www.http.HttpClient.openServer(Unknown Source)

at sun.net.www.http.HttpClient.openServer(Unknown Source)

at sun.net.www.http.HttpClient.<init>(Unknown Source)

at sun.net.www.http.HttpClient.New(Unknown Source)

at sun.net.www.http.HttpClient.New(Unknown Source)

at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)

at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)

at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)

at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1881)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1839)

at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:42)

at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)

at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:96)

at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)

at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:311)

... 15 more

2012-05-11 05:29:27.758 (3) [main]: findDataAvailability retry 1