Publisher Feeds not processing files

We have a “publisher datafeed” feature in Admax, where our India Team pull reports from Display Vendors (Facebook in this case) daily. They format in a specific way and then upload into Admax: Home -> Upload 3rd Party Data

We have a process that looks for csv files in all publisher account folders and processes them. This runs every couple hours.

Starting around 11/13, the system was unable to read the data in the files for some reason. This has occurred in two different accounts.

For the newer unprocessed files, I would see:

2012-11-26 14:09:41.274 (3) [pool-1-thread-1]: Skipped line # 1
2012-11-26 14:09:41.274 (3) [pool-1-thread-1]: Skipped line # 2

Or:

2012-11-26 14:02:21.695 (1) [pool-1-thread-1]: Exception [parsing double "Impressions", cleaned to "Impressions"]:java.lang.NumberFormatException: For input string: "Impressions"
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1224)
at java.lang.Double.valueOf(Double.java:447)
at java.lang.Double.<init>(Double.java:539)
at com.carldunham.jst.util.MathUtil.parseDouble(MathUtil.java:643)
at com.thesearchagency.datafeeds.generic.PublisherConversionColumnGenerator.getPASImpressions(PublisherConversionColumnGenerator.java:161)
at com.thesearchagency.datafeeds.generic.behaviors.BehaviorFacade.getPublisherConversionInformation(BehaviorFacade.java:186)
at com.thesearchagency.datafeeds.generic.PublisherGenericDataFeed.getConversion(PublisherGenericDataFeed.java:479)
at com.thesearchagency.datafeeds.generic.PublisherGenericDataFeed.processArray(PublisherGenericDataFeed.java:297)
at com.thesearchagency.datafeeds.generic.PublisherGenericDataFeed.process(PublisherGenericDataFeed.java:753)
at com.thesearchagency.datafeeds.BaseDataFeed.processDataFile(BaseDataFeed.java:857)
at com.thesearchagency.datafeeds.generic.PublisherGenericDataFeed.processDataFile(PublisherGenericDataFeed.java:540)
at com.thesearchagency.datafeeds.BaseDataFeed.run(BaseDataFeed.java:421)
at com.thesearchagency.datafeeds.generic.PublisherFeedDispatcher.run(PublisherFeedDispatcher.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$

The data in the new versus old files looked the same. I actually copied the data for a new file locally. On XML-07, I deleted this file, copied an older file and renamed it to the date of the file I deleted. I then replaced the data into this file from my local copy. I ran this and it worked fine.

This is what led me to believe it is the format somehow.

To recreate, start on XML-07, and for Oodle Classifieds – Facebook, go to:

/data/var-local-tsa/publishers/812/177

You’ll see all the previously uploaded files. 812 is the account ID and 177 is the distribution ID. When a file is processed, it gets renamed with “.done” at the end. When a file is initially uploaded, it is just a .csv file. That is what the process looks for. To rerun previously uploaded files, you need to remove the .done. You can run this for multiple files to remove the .done:

for old in 812_177_2012093*.csv.done; do mv $old `basename $old .csv.done`.csv; done

Files before 9/29 worked fine. You can rename some files before and after.

Then run this to process:

nohup /usr/local/tsa/bidmgr/load-all-publishers.sh -d4 2>/tmp/load-all-publishers-2012-11-26.log &

You’ll see errors after but not before. India had re-uploaded files after 9/29 last week. Viking River Cruises (299) is the other account and was left un-touched. It stopped working after 11/13, which is why I initially thought it could pertain to the release.

In Admax, you can pull an account report by day by selecting the Display Channel and Facebook to see if the data is in.