Bill's queries to find sources, campaign, searchEngineGroup, bidTarget searchEngineGroup: *************************** 2. row *************************** id: 750012 accountID: 289 description: Branding_Content shortCode: NULL paused: false searchEngineAccountID: 15277 searchEngineIdentifier: 390964207 medium: search searchEngineStatus: ok searchEngineStatusText: ENABLED keywordContentMaxCpc: 12 keywordMaxCpc: 0.5 siteMaxCpc: 1.43 siteMaxCpm: 0 2 rows in set (0.02 sec) and this source: mysql> select * from sources where accountID=289 and searchEngineAccountID=15277 and description like 'Content Network' and searchEngineStatusText='Content Network' and searchEngineGroupID=750012\G *************************** 1. row *************************** id: 97049862 altKey: NULL accountID: 289 siteID: 0 searchEngineAccountID: 15277 keywordID: 0 distributionID: 3 searchEngineIdentifier: Content Network searchEngineGroupIdentifier: 390964207 searchEngineMatchType: content searchEngineGroupID: 750012 isBiddable: true campaignID: 0 description: Content Network type: 7 keyword: Content Network sourceURL: cost: 0.05 maxCost: 12 waypointID: NULL landingURL: passParams: trackPerformance: true searchEngineStatus: ok searchEngineStatusText: Content Network lastChecked: 0000-00-00 00:00:00 1 row in set (0.04 sec) and this bidTarget: mysql> select * from bidTargets where sourceID=97049862\G *************************** 1. row *************************** id: 82226682 siteID: 0 distributionID: 3 sourceID: 97049862 bidMgrAccountID: 0 sbbStrategyID: 0 active: 1 targetPosition: 1 minBid: 0.000 maxBid: 0.050 portfolioTargetPosition: NULL portfolioMaxBid: NULL targetCPA: NULL estimatedCR: NULL estimatedCRLevel: unknown estimatedCREWMALevel: NULL targetROI: 0.48 estimatedRPC: NULL estimatedRPCLevel: unknown estimatedRPCEWMALevel: NULL admaxMin: 0.05 admaxCap: 6 minCPC: 0 lastPosition: 0.0 lastBid: 0.050 lastCTR: 0.000000 lastBidType: unknown lastSetBid: 0.050 paused: false createTime: 0000-00-00 00:00:00 lastModTime: 0000-00-00 00:00:00 lastDataTime: NULL lastSetBidTime: 2010-05-14 02:47:49 lastBidTimePeriodID: NULL prevID: 0 nextID: 0 markedForUpdate: clean 1 row in set (0.03 sec) and I see that the last successful bid reported here is 5/14 with 784 rows in bid history for: select * from bidHistory where bidTargetID=82226682\G lots recent. But the only bid change I can find Google side is from 4/25 made by the AM for this target. He set the bid to 12.00 where it remains. mysql> select * from bidHistory where bidTargetID = 82226682 and time > '2010-05-01' order by time; +-----------+---------------------+-------------+---------+------------+---------------------+ | id | time | bidTargetID | lastBid | lastSetBid | lastBidTimePeriodID | +-----------+---------------------+-------------+---------+------------+---------------------+ | 880172552 | 2010-05-01 01:24:34 | 82226682 | 0.050 | 0.050 | NULL | | 880442012 | 2010-05-01 18:30:12 | 82226682 | 12.000 | 0.050 | NULL | | 880938012 | 2010-05-01 22:37:59 | 82226682 | 0.050 | 0.050 | NULL | | 881238242 | 2010-05-02 18:46:02 | 82226682 | 12.000 | 0.050 | NULL | | 881724672 | 2010-05-02 23:06:56 | 82226682 | 0.050 | 0.050 | NULL | | 882010952 | 2010-05-03 18:45:58 | 82226682 | 12.000 | 0.050 | NULL | | 882489802 | 2010-05-03 22:27:44 | 82226682 | 0.050 | 0.050 | NULL | | 882779802 | 2010-05-04 18:41:16 | 82226682 | 12.000 | 0.050 | NULL | | 883287962 | 2010-05-04 22:46:34 | 82226682 | 0.050 | 0.050 | NULL | | 884379502 | 2010-05-06 18:46:46 | 82226682 | 12.000 | 0.050 | NULL | | 884903562 | 2010-05-07 04:46:33 | 82226682 | 0.050 | 0.050 | NULL | | 885194512 | 2010-05-07 19:05:42 | 82226682 | 12.000 | 0.050 | NULL | | 885693102 | 2010-05-07 23:35:18 | 82226682 | 0.050 | 0.050 | NULL | | 885987842 | 2010-05-08 18:43:13 | 82226682 | 12.000 | 0.050 | NULL | | 886424502 | 2010-05-08 21:56:31 | 82226682 | 0.050 | 0.050 | NULL | | 886774912 | 2010-05-09 18:41:21 | 82226682 | 12.000 | 0.050 | NULL | | 887242572 | 2010-05-09 22:59:34 | 82226682 | 0.050 | 0.050 | NULL | | 888290372 | 2010-05-11 19:58:23 | 82226682 | 12.000 | 0.050 | NULL | | 889005802 | 2010-05-13 13:57:43 | 82226682 | 0.174 | 0.174 | NULL | | 890060842 | 2010-05-13 19:08:07 | 82226682 | 0.050 | 0.050 | NULL | | 890110032 | 2010-05-13 19:55:05 | 82226682 | 12.000 | 0.050 | NULL | | 890805092 | 2010-05-14 02:47:49 | 82226682 | 0.050 | 0.050 | NULL | | 892901642 | 2010-05-14 19:51:24 | 82226682 | 12.000 | 0.050 | NULL | | 892967792 | 2010-05-14 20:21:14 | 82226682 | 0.050 | 0.050 | NULL | | 893477822 | 2010-05-15 19:14:37 | 82226682 | 12.000 | 0.050 | NULL | | 893966062 | 2010-05-15 23:27:55 | 82226682 | 0.050 | 0.050 | NULL | | 894238832 | 2010-05-16 18:59:03 | 82226682 | 12.000 | 0.050 | NULL | | 894703632 | 2010-05-16 23:50:03 | 82226682 | 0.050 | 0.050 | NULL | | 895292192 | 2010-05-17 19:55:32 | 82226682 | 12.000 | 0.050 | NULL | | 895330202 | 2010-05-17 20:53:47 | 82226682 | 0.050 | 0.050 | NULL | +-----------+---------------------+-------------+---------+------------+---------------------+ Seems like it alternates between the $12.00 bid and the $0.05 bid. The $0.05 bid is the AdMax bid. I don't know why it would be alternating. I can look into it some more, but it looks like the $12.00 bid isn't being set successfully. SEBidMgr seems to be getting a few errors for LZ. Maybe those are preventing the $12.00 content bid from being set. Ross 2010-05-18 12:20:27.255 (3) [P1T6]: Handling Account (14297) "New - Incorporation" for user (739) "LegalZoom" 2010-05-18 12:20:27.978 (3) [P1T1]: Caught other/unrecoverable API Exception: "com.thesearchagency.searchengines.SearchEngineCommandException" 2010-05-18 12:20:27.979 (3) [P1T1]: ...Bulk mutate job with id[50955607] seUserID[739] removed from queue 2010-05-18 12:20:27.979 (2) [P1T1]: Exception [Exception setting bids for group identifier:378957727]:com.thesearchagency.searchengines.SearchEngineCommandException: com.thesearchagency.searchengines.SearchEngineCommandException: java.lang.Throwable: Results were not successful for reason(s): [EntityNotFound.INVALID_ID @ operations[0].operations[0].operand.criterion.id; trigger:'293762669', EntityNotFound.INVALID_ID @ operations[0].operations[6].operand.criterion.id; trigger:'293757749'] at com.thesearchagency.searchengines.google.GoogleAdWordsAPI.setBids(GoogleAdWordsAPI.java:1651) at com.thesearchagency.searchengines.google.GoogleAPISearchEngine.setGroupBids(GoogleAPISearchEngine.java:1810) at com.thesearchagency.searchengines.SearchEngineBidManager$SearchEngineAccountWorker.run(SearchEngineBidManager.java:990) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: com.thesearchagency.searchengines.SearchEngineCommandException: java.lang.Throwable: Results were not successful for reason(s): [EntityNotFound.INVALID_ID @ operations[0].operations[0].operand.criterion.id; trigger:'293762669', EntityNotFound.INVALID_ID @ operations[0].operations[6].operand.criterion.id; trigger:'293757749'] at com.thesearchagency.searchengines.google.GoogleAdWordsAPI.processResults(GoogleAdWordsAPI.java:1969) at com.thesearchagency.searchengines.google.GoogleAdWordsAPI.setBids(GoogleAdWordsAPI.java:1628) ... 8 more Caused by: java.lang.Throwable: Results were not successful for reason(s): [EntityNotFound.INVALID_ID @ operations[0].operations[0].operand.criterion.id; trigger:'293762669', EntityNotFound.INVALID_ID @ operations[0].operations[6].operand.criterion.id; trigger:'293757749'] ... 10 more 2010-05-18 12:20:28.492 (3) [P1T7]: Completed account "CYN Content" 2010-05-18 12:20:28.509 (3) [P1T7]: Handling Account (40652) "New - Incorporation - Local/State" for user (739) "LegalZoom" Here are the recent bid change requests: +---------------------+-----------+-------------+-------------+ | requestTime | newMaxBid | newAdMaxMin | newAdMaxCap | +---------------------+-----------+-------------+-------------+ | 2010-01-27 21:16:49 | NULL | NULL | NULL | | 2010-03-02 10:57:01 | NULL | 0.05 | NULL | | 2010-03-24 15:43:57 | NULL | NULL | NULL | | 2010-03-30 15:56:22 | 1.850 | NULL | NULL | | 2010-04-07 17:26:13 | 1.230 | NULL | NULL | | 2010-04-19 17:57:29 | 0.510 | NULL | NULL | | 2010-04-26 11:13:27 | 12.000 | NULL | NULL | | 2010-05-03 09:12:36 | 0.050 | NULL | NULL | +---------------------+-----------+-------------+-------------+ IBM shows an AdMax bid of $0.04, which would be adjusted to $0.05. Hmm - it looks like the $12.00 bid should have gone away on 2010-05-03. (This is where Bill says, "That's what I've been saying!!") Here is the changehistory from Google on the account. Patrick (the AM) made two changes on the 14th. 12.00 -> .05 --- the "I am freaking out about this" .05-->12.00 -- the "I am freaking out but want engineering to track this down bid" Bill: I am going to tell him to make the changes he needs to and hope we do not override them as the account is having some issues - this is happening in multiple content based ad groups. It is the content network bid in a group with both placements and keywords where the campaign is enabled for all placementss. Ross: I see 8 jobs today running in production for LZ for bidding on Google. I doubt Patrick actually ran 8 jobs today. There are some exceptions but I am wondering if we have a bid job caught in a loop.