ADS stuck in a loop on deployCampaigns

First time I ran ADS deployCampaigns this morning, with master build, it worked fine. Then I tried a redeploy and every deployCampaigns request since then has been seemingly stuck in a loop. Keeps getting to this error and running indefinitely.

Got transaction issues just now on my local machine when trying to deployCampaigns (ADS) for a customer with 12 campaigns. The request didn't fail though.. it seemed to put the request in some sort of loop. I manually killed jboss to stop it and remove the message from JBM_MSG and JBM_MSG_REF in the schema referenced in $JBOSS_HOME/server/<node>/farm/smb-jboss-ds.xml. I'm going to look into this more and see if there was some change between 1.3 and 1.5 that would cause this behavior.

02:45:04,552 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX -3f57e17f:aa46:50b8e702:961c in state RUN
02:45:04,560 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57e17f:aa46:50b8e702:961c invoked while multiple threads active within it.
02:45:04,561 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57e17f:aa46:50b8e702:961c aborting with 1 threads active!
02:45:04,627 WARN [SQLErrorCodesFactory] Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is org.jboss.util.NestedSQLException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:296)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:320)
at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:212)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:141)
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:104)
at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:636)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:665)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:673)
at com.thesearchagency.core.smb.dao.contentmodel.component.ContentFieldDAOSpringJDBC.readCustomizableEntityFields(ContentFieldDAOSpringJDBC.java:100)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.setAllFields(DOMPersistanceManager.java:409)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.addChildrenToEntity(DOMPersistanceManager.java:400)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.constructEntityGraph(DOMPersistanceManager.java:325)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.getContentModelLayer(DOMPersistanceManager.java:263)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.getContentModelLayer(DOMPersistanceManager.java:275)
at sun.reflect.GeneratedMethodAccessor677.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
at $Proxy1091.getContentModelLayer(Unknown Source)
at com.thesearchagency.core.smb.service.content_model.component.CustomerCampaignServiceBD.readAllWithAdPermutations(CustomerCampaignServiceBD.java:120)
at sun.reflect.GeneratedMethodAccessor687.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy1097.readAllWithAdPermutations(Unknown Source)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.getAdCopies(AdMaxDeploymentServiceImpl.java:1619)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaign(AdMaxDeploymentServiceImpl.java:1152)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaigns(AdMaxDeploymentServiceImpl.java:1022)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.convertDeploymentRequest(AdMaxDeploymentServiceImpl.java:441)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.handleDeployCampaignRequest(AdMaxDeploymentRequestMDB.java:181)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.onMessage(AdMaxDeploymentRequestMDB.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:73)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:59)
at sun.reflect.GeneratedMethodAccessor558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_30178960.invoke(InvocationContextInterceptor_z_fillMethod_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_30178960.invoke(InvocationContextInterceptor_z_setup_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:270)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140)
at $Proxy539.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_30178960.invoke(SessionAspect_z_handleRun_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:234)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.jboss.util.NestedSQLException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.checkTransactionActive(WrapperDataSource.java:165)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransactionActive(WrappedConnection.java:833)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkStatus(WrappedConnection.java:848)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransaction(WrappedConnection.java:825)
at org.jboss.resource.adapter.jdbc.WrappedConnection.getMetaData(WrappedConnection.java:527)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:285)
... 99 more
Caused by: javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK
at org.jboss.resource.connectionmanager.TxConnectionManager.checkTransactionActive(TxConnectionManager.java:332)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.checkTransactionActive(BaseConnectionManager2.java:972)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.checkTransactionActive(WrapperDataSource.java:161)
... 104 more
02:45:04,632 WARN [AdMaxDeploymentServiceImpl] Error while processing Campaigns... 817-19882_2001047_10005_817-19882-11
com.thesearchagency.core.smb.dao.customization.impl.DAOException: ERROR Could not read CustomizableContentField :: PreparedStatementCallback; uncategorized SQLException for SQL [SELECT * FROM contentField WHERE customizable_entity_id = ?]; SQL state [null]; error code [0]; Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK); nested exception is org.jboss.util.NestedSQLException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK)
at com.thesearchagency.core.smb.dao.contentmodel.component.ContentFieldDAOSpringJDBC.readCustomizableEntityFields(ContentFieldDAOSpringJDBC.java:104)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.setAllFields(DOMPersistanceManager.java:409)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.addChildrenToEntity(DOMPersistanceManager.java:400)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.constructEntityGraph(DOMPersistanceManager.java:325)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.getContentModelLayer(DOMPersistanceManager.java:263)
at com.thesearchagency.core.smb.dao.contentmodel.DOMPersistanceManager.getContentModelLayer(DOMPersistanceManager.java:275)
at sun.reflect.GeneratedMethodAccessor677.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
at $Proxy1091.getContentModelLayer(Unknown Source)
at com.thesearchagency.core.smb.service.content_model.component.CustomerCampaignServiceBD.readAllWithAdPermutations(CustomerCampaignServiceBD.java:120)
at sun.reflect.GeneratedMethodAccessor687.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy1097.readAllWithAdPermutations(Unknown Source)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.getAdCopies(AdMaxDeploymentServiceImpl.java:1619)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaign(AdMaxDeploymentServiceImpl.java:1152)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaigns(AdMaxDeploymentServiceImpl.java:1022)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.convertDeploymentRequest(AdMaxDeploymentServiceImpl.java:441)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.handleDeployCampaignRequest(AdMaxDeploymentRequestMDB.java:181)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.onMessage(AdMaxDeploymentRequestMDB.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:73)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:59)
at sun.reflect.GeneratedMethodAccessor558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_30178960.invoke(InvocationContextInterceptor_z_fillMethod_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_30178960.invoke(InvocationContextInterceptor_z_setup_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:270)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140)
at $Proxy539.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_30178960.invoke(SessionAspect_z_handleRun_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:234)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [SELECT * FROM contentField WHERE customizable_entity_id = ?]; SQL state [null]; error code [0]; Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK); nested exception is org.jboss.util.NestedSQLException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:636)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:665)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:673)
at com.thesearchagency.core.smb.dao.contentmodel.component.ContentFieldDAOSpringJDBC.readCustomizableEntityFields(ContentFieldDAOSpringJDBC.java:100)
... 89 more
Caused by: org.jboss.util.NestedSQLException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK; - nested throwable: (javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.checkTransactionActive(WrapperDataSource.java:165)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransactionActive(WrappedConnection.java:833)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkStatus(WrappedConnection.java:848)
at org.jboss.resource.adapter.jdbc.WrappedConnection.checkTransaction(WrappedConnection.java:825)
at org.jboss.resource.adapter.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:240)
at org.springframework.jdbc.core.JdbcTemplate$SimplePreparedStatementCreator.createPreparedStatement(JdbcTemplate.java:1362)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:580)
... 93 more
Caused by: javax.transaction.RollbackException: Transaction TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING > cannot proceed STATUS_ROLLING_BACK
at org.jboss.resource.connectionmanager.TxConnectionManager.checkTransactionActive(TxConnectionManager.java:332)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.checkTransactionActive(BaseConnectionManager2.java:972)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.checkTransactionActive(WrapperDataSource.java:161)
... 99 more
02:45:04,633 INFO [AdMaxDeploymentServiceImpl] Saving deployment status to 'Failure' for campaignID: 817-19882-11
02:45:04,633 WARN [AdMaxDeploymentServiceImpl] error updating deployment status
com.thesearchagency.core.smb.service.admaxdeployment.AdMaxDeploymentException: Campaign-level deployment status update failed for campaignID 817-19882-11
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.insertOrUpdateDeployStatus(AdMaxDeploymentServiceImpl.java:2306)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaigns(AdMaxDeploymentServiceImpl.java:1048)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.convertDeploymentRequest(AdMaxDeploymentServiceImpl.java:441)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.handleDeployCampaignRequest(AdMaxDeploymentRequestMDB.java:181)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.onMessage(AdMaxDeploymentRequestMDB.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:73)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:59)
at sun.reflect.GeneratedMethodAccessor558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_30178960.invoke(InvocationContextInterceptor_z_fillMethod_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_30178960.invoke(InvocationContextInterceptor_z_setup_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:270)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140)
at $Proxy539.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_30178960.invoke(SessionAspect_z_handleRun_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:234)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
02:45:04,665 ERROR [TxPolicy] javax.ejb.EJBTransactionRolledbackException: Could not get JDBC Connection; nested exception is org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >)
02:45:04,679 WARN [AdMaxDeploymentServiceImpl] Error while processing Campaigns... 817-19882_2000084_10005_817-19882-12
javax.ejb.EJBTransactionRolledbackException: Could not get JDBC Connection; nested exception is org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >)
at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:115)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:186)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:176)
at org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:216)
at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)
at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
at $Proxy435.getGoogleLocationTarget(Unknown Source)
at sun.reflect.GeneratedMethodAccessor776.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.springframework.ejb.access.LocalSlsbInvokerInterceptor.invokeInContext(LocalSlsbInvokerInterceptor.java:71)
at org.springframework.ejb.access.AbstractSlsbInvokerInterceptor.invoke(AbstractSlsbInvokerInterceptor.java:189)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy1076.getGoogleLocationTarget(Unknown Source)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaign(AdMaxDeploymentServiceImpl.java:1116)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaigns(AdMaxDeploymentServiceImpl.java:1022)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.convertDeploymentRequest(AdMaxDeploymentServiceImpl.java:441)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.handleDeployCampaignRequest(AdMaxDeploymentRequestMDB.java:181)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.onMessage(AdMaxDeploymentRequestMDB.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:73)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:59)
at sun.reflect.GeneratedMethodAccessor558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_30178960.invoke(InvocationContextInterceptor_z_fillMethod_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_30178960.invoke(InvocationContextInterceptor_z_setup_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:270)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140)
at $Proxy539.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_30178960.invoke(SessionAspect_z_handleRun_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:234)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:572)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:636)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:665)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:673)
at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:728)
at com.thesearchagency.core.smb.dao.businesslocation.BusinessLocationSpringJDBCDAO.getBusinessLocation(BusinessLocationSpringJDBCDAO.java:56)
at com.thesearchagency.smb.service.businesslocation.component.BusinessLocationServiceBD.getBusinessLocation(BusinessLocationServiceBD.java:70)
at com.thesearchagency.smb.service.businesslocation.component.BusinessLocationServiceBD.getGoogleLocationTarget(BusinessLocationServiceBD.java:187)
at com.thesearchagency.smb.service.businesslocation.binding.BusinessLocationServiceEJB.getGoogleLocationTarget(BusinessLocationServiceEJB.java:88)
at sun.reflect.GeneratedMethodAccessor777.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:73)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:59)
at sun.reflect.GeneratedMethodAccessor558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_30178960.invoke(InvocationContextInterceptor_z_fillMethod_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_30178960.invoke(InvocationContextInterceptor_z_setup_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
... 93 more
Caused by: org.jboss.util.NestedSQLException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >; - nested throwable: (javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:95)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 132 more
Caused by: javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: -3f57e17f:aa46:50b8e702:961c status: ActionStatus.ABORTING >
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:370)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
... 134 more
02:45:04,680 INFO [AdMaxDeploymentServiceImpl] Saving deployment status to 'Failure' for campaignID: 817-19882-12
02:45:04,680 WARN [AdMaxDeploymentServiceImpl] error updating deployment status
com.thesearchagency.core.smb.service.admaxdeployment.AdMaxDeploymentException: Campaign-level deployment status update failed for campaignID 817-19882-12
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.insertOrUpdateDeployStatus(AdMaxDeploymentServiceImpl.java:2306)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.processCampaigns(AdMaxDeploymentServiceImpl.java:1048)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentServiceImpl.convertDeploymentRequest(AdMaxDeploymentServiceImpl.java:441)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.handleDeployCampaignRequest(AdMaxDeploymentRequestMDB.java:181)
at com.thesearchagency.core.smb.service.admaxdeployment.component.AdMaxDeploymentRequestMDB.onMessage(AdMaxDeploymentRequestMDB.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:73)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:59)
at sun.reflect.GeneratedMethodAccessor558.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_30178960.invoke(InvocationContextInterceptor_z_fillMethod_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_30178960.invoke(InvocationContextInterceptor_z_setup_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:282)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:270)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140)
at $Proxy539.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_30178960.invoke(SessionAspect_z_handleRun_30178960.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:234)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
02:45:04,681 INFO [AdMaxDeploymentServiceImpl] Finished deploy campaign message conversion. 10 converted.
02:45:04,681 ERROR [AdMaxDeploymentRequestMDB] Error queuing request, attempting rollback.
02:45:05,059 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX -3f57e17f:aa46:50b8e702:961c in state CANCEL
02:45:09,717 INFO [AdMaxDeploymentServiceImpl] Starting deploy campign message conversion
02:45:09,894 INFO [AdMaxDeploymentServiceImpl] 12 Campaign(s) to process.
02:45:09,896 INFO [ContentRepositoryServiceImpl] Inside ContentRepositoryServiceImpl Constructor : EntityId [0] BusinessTypeID [0] BusinessLocationID [0]
02:45:10,011 INFO [CustomerBusinessAreaServiceBD] start readAll...
02:45:10,235 WARN [CustomerFieldDAOSpringJDBC] 1395 13823 24369 BUSINESS_LOCATION_ID CUSTOMER
02:45:10,614 WARN [CustomerFieldDAOSpringJDBC] 1395 13815 24361 PAGE_TYPE CUSTOMER
02:45:10,690 WARN [CustomerFieldDAOSpringJDBC] 1395 13817 24363 PARAGRAPH_INDEX CUSTOMER
02:45:10,760 WARN [CustomerFieldDAOSpringJDBC] 1395 13821 24367 PARAGRAPH_INDEX CUSTOMER
02:45:10,835 WARN [CustomerFieldDAOSpringJDBC] 1395 13819 24365 PARAGRAPH_INDEX CUSTOMER
02:45:10,913 WARN [CustomerFieldDAOSpringJDBC] 1395 13813 24357 BUSINESS_AREA_ID CUSTOMER
02:45:10,913 WARN [CustomerFieldDAOSpringJDBC] 1395 13813 24359 IMAGE_ID CUSTOMER
02:45:10,985 WARN [CustomerFieldDAOSpringJDBC] 1395 13849 24381 PAGE_TYPE CUSTOMER
02:45:11,058 WARN [CustomerFieldDAOSpringJDBC] 1395 13851 24383 PARAGRAPH_INDEX CUSTOMER
02:45:11,135 WARN [CustomerFieldDAOSpringJDBC] 1395 13853 24385 PARAGRAPH_INDEX CUSTOMER
02:45:11,207 WARN [CustomerFieldDAOSpringJDBC] 1395 13855 24387 PARAGRAPH_INDEX CUSTOMER
02:45:11,282 WARN [CustomerFieldDAOSpringJDBC] 1395 13839 24379 BUSINESS_LOCATION_ID CUSTOMER
02:45:11,670 WARN [CustomerFieldDAOSpringJDBC] 1395 13837 24375 BUSINESS_AREA_ID CUSTOMER
02:45:11,670 WARN [CustomerFieldDAOSpringJDBC] 1395 13837 24377 IMAGE_ID CUSTOMER
02:45:11,734 WARN [CustomerFieldDAOSpringJDBC] 1395 13859 24393 BUSINESS_LOCATION_ID CUSTOMER
02:45:12,095 WARN [CustomerFieldDAOSpringJDBC] 1395 13869 24395 PAGE_TYPE CUSTOMER
02:45:12,167 WARN [CustomerFieldDAOSpringJDBC] 1395 13871 24397 PARAGRAPH_INDEX CUSTOMER

Hmm I was testing w/ master, which has Jeph's jboss messaging changes, though I haven't updated my local jboss. Jeph, do you think this could have been caused by my jboss being out of sync? Looking at the log above, there are definitely issues with onMessage() so I'm guessing that explains this (hopefully). Let me try to reproduce with 1.5, which doesn't have the jboss messaging changes.

Still get the error after cleanly installing the build and dropping/recreating the jboss DB. However I just tried for a customer with 4 core campaigns and while the ADS jboss portion took longer than expected, it did eventually send the message to the listener. Needs more investigation. My guess is that if I had let the customer with 12 campaigns run for longer it too would eventually send the message. Either way, something is up.