Google Paging in performance report downloads issue

Here is the change set which introduces Paging in performance report downloads. As agreed with Joel, we are downloading in chunk of 8000 at a time for SMS.

http://murky-svn.ri.thesearchagency.com/trac-tsa/changeset?old_path=%2Ftrunk%2FTheSearchAgency%2Fsrc%2Fbidmgr%2Fsrc%2Fcom%2Fthesearchagency%2Fsearchengines%2Fgoogle%2FGoogleAdWordsAPI.java&old=2605&new_path=%2Ftrunk%2FTheSearchAgency%2Fsrc%2Fbidmgr%2Fsrc%2Fcom%2Fthesearchagency%2Fsearchengines%2Fgoogle%2FGoogleAdWordsAPI.java&new=2605

Regards.

From: Ramesh Ramakrishnan
Sent: 11 October 2012 12:00
To: 'Ross Gardner'; 'Jeff Theroux'; Manasa Denning; Ajay Nagaraj
Cc: 'Jeffrey Collemer'; 'Autumn Francesca'
Subject: FW: Adwords v201206 upgrade issue
Importance: High

Dear All,

We deployed v201206 last evening and ended up with an issue while downloading thousands of performance reports. Apparently in 201206, we MUST use pagination for pulling any records more than 10K. Though Joel says this was there even in 201109, we never had a pagination issue previously.

Ross/Manasa, Not sure if Optus has that many accounts, but it is worth using the pagination strategy to make it future proof.

Ffej, I heard form Ajay that there are no pagination implementation at Agency. Worth considering these.

We will be making the code change on pagination (the max limit I agreed with Joel is 8000 though 10K is the upper limit). Ajay will share the code may be tomorrow, so you guys can take a look in to it and implement in the respective projects.

Let me know if you have any questions.

Cheers.

From: Ramesh Ramakrishnan
Sent: 11 October 2012 11:43
To: 'Alok Shukla'; Ajay Nagaraj; Chris Burton; 'Gurpreet Singh'; Rahul Honakande; 'Pradeep Nair'
Subject: FW: Adwords v201206 upgrade issue
Importance: High

Dear All

Joel came back with the root cause of the issue. And he has asked us to use pagination with “8000” records as the limit (not 5000 as mentioned in his email).

At any point, to get the total account, the ManagedCustomerService will be returning a variable called TotalNumEntries with valye. This will have the TOTAL no of accounts. And based on this number, we MUST be processing a pagination.For example, if the number in TotalNumEntries is 82,000, we should have the following pagination,

selector.setPaging(new Paging(0, 8000));
selector.setPaging(new Paging(8000, 8000));
selector.setPaging(new Paging(16000, 8000));
selector.setPaging(new Paging(24000, 8000));
selector.setPaging(new Paging(32000, 8000));
selector.setPaging(new Paging(40000, 8000));
selector.setPaging(new Paging(48000, 8000));
selector.setPaging(new Paging(56000, 8000));
selector.setPaging(new Paging(64000, 8000));
selector.setPaging(new Paging(72000, 8000));
selector.setPaging(new Paging(80000, 8000));
selector.setPaging(new Paging(88000, 8000));

So, what this effectively means is, we pull batch of records in an increment of 8000. Even if the last statement is we are pulling between 80,000 and 88,000, since we have only 82000 accounts, we will be pulling only 2000.

The API details are in here,
https://developers.google.com/adwords/api/docs/reference/latest/ManagedCustomerService.Paging

After the deployment, please work on the fix. Let’s try to get this to production by 16/10.

Regards.

From: Joel Toby joeltoby@google.com
Sent: 11 October 2012 11:23
To: Ramesh Ramakrishnan
Cc: Autumn Francesca; Ajay Nagaraj
Subject: Re: Adwords v201206 upgrade issue

Hi Ramesh,

I have just just run some tests with the Yell MCC. As expected (due to the large size of the Yell MCC), I received the same error: SizeLimitError.RESPONSE_SIZE_LIMIT_EXCEEDED.

However, I was able to resolve this by adding paging to my selector:
selector.setPaging(new Paging(0, 5000));
In my example above, I am downloading the results in batches of 5000 accounts. My understanding is that any value less than 10,000 accounts should work.

The managed customer page returned by the ManagedCustomerService.get request contains a TotalNumEntries value. This value represents the total number of accounts in the MCC (not just the 5,000 on this page). Therefore, I suggest that you use this value to set the paging in your requests.

For example, if you are requesting results in pages of 5,000 (as I have done above), and you have a total of 17,000 accounts, you would use the TotalNumEntries to loop through your requests. Effectively, you would make 4 identical requests with different paging start values:

selector.setPaging(new Paging(0, 5000));
selector.setPaging(new Paging(5000, 5000));
selector.setPaging(new Paging(10000, 5000));
selector.setPaging(new Paging(15000, 5000));

Please let me know if you have any further questions.

Kind regards,

Joel

Joel Toby |

Developer Advocate EMEA |

joeltoby@google.com |

+353 (0)1 543 2287

Google Ireland Ltd. Gordon House
Barrow Street, Dublin 4, Ireland
Registered in Dublin, Ireland, Registration Number: 368047
On 11 October 2012 10:01, Ramesh Ramakrishnan <Ramesh.Ramakrishnan@thesearchagency.co.uk> wrote:
Hello Joel,

We upgraded all our services to AdWords v201206 version last evening. This morning when working on ManagedCustomerService’s get(Selector) method, we encountered the following exception, SizeLimitError.RESPONSE_SIZE_LIMIT_EXCEEDED.

I am attaching the request and response XMLs herewith. Could you let us know the root cause of this issue?

There is a 2 year old thread that talks about this error. The latest from this thread says its an open issue.

https://groups.google.com/forum/?fromgroups=#!searchin/adwords-api/SizeLimitError.RESPONSE_SIZE_LIMIT_EXCEEDED/adwords-api/L-CBcQm18z8/vHc1sBWrskYJ

I have revoked back the production to v201109 till we fix this issue. Let me know if you need further info.

Best Regards.

Ramesh Ramakrishnan | Technical Account Manager