BES not retrying internal Google errors

Log from aq-demo-core1. I recommend fixing this in Newport or Diu because it it causes estimates to fail and that error is visible in the UI.

Unfortnately this is an intermittent error so it will be hard to reproduce. I filed a request with Mark Bowyer to see if Google is aware of this issue.

Stack trace:

2014-03-31 19:50:59,465 INFO [com.thesearchagency.core.smb.budgetestimate.impl.searchengine.api.trafficestimator.GoogleTrafficEstimator] (Thread-35990) Finish BES thread: 129128
2014-03-31 19:50:59,466 INFO [com.thesearchagency.core.smb.budgetestimate.impl.searchengine.api.trafficestimator.GoogleTrafficEstimator] (Thread-35991) Start BES thread: 129129
2014-03-31 19:51:00,907 INFO [com.google.api.ads.adwords.lib.client.AdWordsServiceClient.requestInfoLogger] (Thread-35991) Request made: Service: "TrafficEstimatorServiceInterfacePort" Method: "get" URL: "https:
//adwords.google.com/api/adwords/o/v201309/TrafficEstimatorService"
2014-03-31 19:51:00,948 WARN [com.google.api.ads.adwords.lib.client.AdWordsServiceClient.soapXmlLogger] (Thread-35991) SOAP Request:

2014-03-31 19:51:00,951 DEBUG [com.thesearchagency.core.smb.budgetestimate.impl.searchengine.api.trafficestimator.GoogleTrafficEstimator] (Thread-35991) Exception found on TrafficEstimatorThread 2
AxisFault
faultCode:

{http://schemas.xmlsoap.org/soap/envelope/}

Server
faultSubcode:
faultString: [InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.<init>(InternalApiErro]
faultActor:
faultNode:
faultDetail:

{https://adwords.google.com/api/adwords/o/v201309}

ApiExceptionFault:<ns2:message>[InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.<init>(In
ternalApiErro]</ns2:message><ns2:ApplicationException.Type>ApiException</ns2:ApplicationException.Type><ns2:errors xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:InternalApiError"><ns2:fieldP
ath/><ns2:trigger/><ns2:errorString>InternalApiError.UNEXPECTED_INTERNAL_API_ERROR</ns2:errorString><ns2:ApiError.Type>InternalApiError</ns2:ApiError.Type><ns2:reason>UNEXPECTED_INTERNAL_API_ERROR</ns2:reason></n
s2:errors>
[InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.<init>(InternalApiErro]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:104)
at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:90)
at com.google.api.ads.adwords.axis.v201309.cm.ApiException.getDeserializer(ApiException.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.axis.encoding.ser.BaseDeserializerFactory.getSpecialized(BaseDeserializerFactory.java:154)
at org.apache.axis.encoding.ser.BaseDeserializerFactory.getDeserializerAs(BaseDeserializerFactory.java:84)
at org.apache.axis.encoding.DeserializationContext.getDeserializer(DeserializationContext.java:464)
at org.apache.axis.encoding.DeserializationContext.getDeserializerForType(DeserializationContext.java:547)
at org.apache.axis.message.SOAPFaultDetailsBuilder.onStartChild(SOAPFaultDetailsBuilder.java:157)
at org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1035)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)
at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.google.api.ads.adwords.axis.v201309.o.TrafficEstimatorServiceSoapBindingStub.get(TrafficEstimatorServiceSoapBindingStub.java:925)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.api.ads.common.lib.soap.SoapClientHandler.invoke(SoapClientHandler.java:109)
at com.google.api.ads.common.lib.soap.axis.AxisHandler.invokeSoapCall(AxisHandler.java:175)
at com.google.api.ads.common.lib.soap.SoapServiceClient.callSoapClient(SoapServiceClient.java:64)
at com.google.api.ads.common.lib.soap.SoapServiceClient.invoke(SoapServiceClient.java:93)
at $Proxy992.get(Unknown Source)
at com.thesearchagency.core.smb.budgetestimate.impl.searchengine.api.trafficestimator.GoogleTrafficEstimator$TrafficEstimatorThread.run(GoogleTrafficEstimator.java:600)
2014-03-31 19:51:00,959 INFO [com.thesearchagency.core.smb.budgetestimate.impl.searchengine.api.trafficestimator.GoogleTrafficEstimator] (Thread-35991) Finish BES thread: 129129
2014-03-31 19:51:00,959 WARN [com.thesearchagency.core.smb.utilities.AlertUtils] (http-0.0.0.0-8080-1) sendAlert: executing command: [/usr/local/tsa/send_alarm.sh, WARNING, APSD, AdMax_BudgetEstimateService: Com
munication - Could not retrieve budget estimates]
2014-03-31 19:51:01,015 WARN [com.thesearchagency.core.smb.budgetestimate.impl.searchengine.api.trafficestimator.GoogleTrafficEstimator] (http-0.0.0.0-8080-1) Budget estimate retrieval failed:
AxisFault
faultCode:

{http://schemas.xmlsoap.org/soap/envelope/}

Server
faultSubcode:
faultString: [InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.<init>(InternalApiErro]
faultActor:
faultNode:
faultDetail:

{https://adwords.google.com/api/adwords/o/v201309}

ApiExceptionFault:<ns2:message>[InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.<init>(In
ternalApiErro]</ns2:message><ns2:ApplicationException.Type>ApiException</ns2:ApplicationException.Type><ns2:errors xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:InternalApiError"><ns2:fieldP
ath/><ns2:trigger/><ns2:errorString>InternalApiError.UNEXPECTED_INTERNAL_API_ERROR</ns2:errorString><ns2:ApiError.Type>InternalApiError</ns2:ApiError.Type><ns2:reason>UNEXPECTED_INTERNAL_API_ERROR</ns2:reason></n
s2:errors>

[InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.<init>(InternalApiErro]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:104)
at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:90)
at com.google.api.ads.adwords.axis.v201309.cm.ApiException.getDeserializer(ApiException.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.axis.encoding.ser.BaseDeserializerFactory.getSpecialized(BaseDeserializerFactory.java:154)
at org.apache.axis.encoding.ser.BaseDeserializerFactory.getDeserializerAs(BaseDeserializerFactory.java:84)
at org.apache.axis.encoding.DeserializationContext.getDeserializer(DeserializationContext.java:464)
at org.apache.axis.encoding.DeserializationContext.getDeserializerForType(DeserializationContext.java:547)
at org.apache.axis.message.SOAPFaultDetailsBuilder.onStartChild(SOAPFaultDetailsBuilder.java:157)
at org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1035)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)
at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.google.api.ads.adwords.axis.v201309.o.TrafficEstimatorServiceSoapBindingStub.get(TrafficEstimatorServiceSoapBindingStub.java:925)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.api.ads.common.lib.soap.SoapClientHandler.invoke(SoapClientHandler.java:109)
at com.google.api.ads.common.lib.soap.axis.AxisHandler.invokeSoapCall(AxisHandler.java:175)
at com.google.api.ads.common.lib.soap.SoapServiceClient.callSoapClient(SoapServiceClient.java:64)
at com.google.api.ads.common.lib.soap.SoapServiceClient.invoke(SoapServiceClient.java:93)
at $Proxy992.get(Unknown Source)
at com.thesearchagency.core.smb.budgetestimate.impl.searchengine.api.trafficestimator.GoogleTrafficEstimator$TrafficEstimatorThread.run(GoogleTrafficEstimator.java:600)