CONCURRENT_MODIFICATION error occurs intermittently on deployment

CONCURRENT_MODIFICATION error occurs intermittently on deployment.

After I installed bigmgr build 28 and deployed my 1st order, I saw the following errors in the log though the order got deployed successfully.
I am unable to reproduce this again. Orders that were deployed with the same/different campaigns again had no such errors.

Listener log

2014-04-23 07:51:22.338 (2) [pool-5-thread-2]: 1 Errors found
2014-04-23 07:51:22.338 (3) [pool-5-thread-2]: Retrying based on class DatabaseError
2014-04-23 07:51:22.339 (3) [pool-5-thread-2]: Encountered Recoverable API Error, fieldpath=, trigger= apierrortype=DatabaseError, reason=DatabaseError.CONCURRENT_MODIFICATION . The request will be retried.
2014-04-23 07:51:22.361 (3) [pool-5-thread-1]: TextAdDAO(accountID 554) Request contained 4 adds, 0 updates, 0 deletes.
2014-04-23 07:51:22.361 (3) [pool-5-thread-1]: TextAdDAO(accountID 554) DB updates: 4 items updated, 4 pending changes deleted, 0 pending updates deleted, 0 errors cleared
2014-04-23 07:51:46.676 (3) [Thread-9]: Heartbeat #15 (1398239506670) received in 0.006s
2014-04-23 07:51:52.340 (3) [pool-5-thread-2]: Caught recoverable API Error, 4retries left
2014-04-23 07:51:52.717 (3) [pool-5-thread-2]: TextAdDAO(accountID 554) Request contained 4 adds, 0 updates, 0 deletes.
2014-04-23 07:51:52.717 (3) [pool-5-thread-2]: TextAdDAO(accountID 554) DB updates: 4 items updated, 4 pending changes deleted, 0 pending updates deleted, 0 errors cleared
2014-04-23 07:52:08.332 (3) [pool-6-thread-1]: KeywordChangeRequest (accountID 554): Sending 96 add requests
2014-04-23 07:52:08.332 (3) [pool-6-thread-2]: KeywordChangeRequest (accountID 554): Sending 27 add requests

Jboss log

Line 847: 07:52:49,546 INFO [ErrorDAOSpringJDBC] getOperationErrorsResponse [SELECT gce_feed.name as 'category', `reason`, `trigger`, `field` FROM gce_feedPendingChange INNER JOIN gce_feedError ON gce_feedPendingChange.`feedID` = gce_feedError.`feedID` INNER JOIN gce_feed ON gce_feed.id=gce_feedError.`feedID` INNER JOIN gce_campaignFeed ON gce_campaignFeed.feedID=gce_feed.id WHERE gce_campaignFeed.campaignID IN ('2062', '2061') AND gce_feedPendingChange.isFailed='true']
Line 848: 07:52:49,546 DEBUG [JdbcTemplate] Executing SQL query [SELECT gce_feed.name as 'category', `reason`, `trigger`, `field` FROM gce_feedPendingChange INNER JOIN gce_feedError ON gce_feedPendingChange.`feedID` = gce_feedError.`feedID` INNER JOIN gce_feed ON gce_feed.id=gce_feedError.`feedID` INNER JOIN gce_campaignFeed ON gce_campaignFeed.feedID=gce_feed.id WHERE gce_campaignFeed.campaignID IN ('2062', '2061') AND gce_feedPendingChange.isFailed='true']

soap log

<soap:Body>
<soap:Fault>
<faultcode>soap:Server</faultcode>
<faultstring>[DatabaseError.CONCURRENT_MODIFICATION @ com.google.ads.api.services.campaignmgmt.common.error.DatabaseError.<init>(Datab]</faultstring>
<detail>
<ApiExceptionFault xmlns="https://adwords.google.com/api/adwords/cm/v201402">
<message>[DatabaseError.CONCURRENT_MODIFICATION @ com.google.ads.api.services.campaignmgmt.common.error.DatabaseError.<init>(Datab]</message>
<ApplicationException.Type>ApiException</ApplicationException.Type>
<errors xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="DatabaseError">
<fieldPath/>
<trigger/>
<errorString>DatabaseError.CONCURRENT_MODIFICATION</errorString>
<ApiError.Type>DatabaseError</ApiError.Type>
<reason>CONCURRENT_MODIFICATION</reason>
</errors>
</ApiExceptionFault>
</detail>
</soap:Fault>
</soap:Body>
</soap:Envelope>

Stacie's comments:

From Google (https://developers.google.com/adwords/api/training/development-techniques/error-handling):
Server-side errors
During request processing, server side errors are also possible:
• CONCURRENT_MODIFICATION
The error occurred due to concurrent modification of the same object by different threads. The operation has failed and the request needs to be retried.
If your application is multi-threaded, avoid allowing different threads to modify the same AdGroup, Campaign, etc. Please note this error can also occur even if you use only one thread for requests.

"In the past, you might have gotten a CONCURRENT_MODIFICATION error even when using a single-threaded application to modify one account at a time. The problem was likely due to background processing, such as approvals, which also modified the account’s entities. We’ve made some changes that should reduce the likelihood of these errors occurring " (http://adwordsapi.blogspot.com/2011/03/concurrency-management-in-adwords-api.html)

Looks like this error is something we see from time to time and is nothing to worry about. The code correctly caught and retried it. While we are multithreaded I don't believe we allow different threads to modify the same AdGroup, Campaign, etc.

Fjej's comment:

It's probably worth opening a bug to keep an eye on it in case it happens again. If it doesn't happen again (or happens very infrequently) we can close it at the end of the project as long as it doesn't happen during performance testing. if it's a recurring thing that happens frequently we should probably consider increasing it. it's possible the API changed behavior. it's also possible the AdWords API was under heavy usage at the time and it took their backend processes longer than usual to do the AdGroup data replication they do behind the scenes. If we try to add an Ad or Keyword to the AdGroup while the AdGroup data is being replicated across Google's servers, that is what triggers the CONCURRENT_MODIFICATION exception. I hope that helps clarify why it happens and why it's not a huge issue if it happens infrequently.