Uploaded image for project: 'AdMaxLocal'
  1. AdMaxLocal
  2. AML-3336

Continuous processing of log on missing data in input file

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Rushmore
    • Fix Version/s: Rushmore
    • Component/s: Data Loader
    • Labels:
      None
    • Environment:

      aml merchant lead: aml-merchant-leads-distribution-1.0.0-20150130.194726-5-bin.tar.gz

    • Sprint:
      Sprint 4

      Description

      1) Dropped a file with a blank row
      Eg.,
      28/1/2015,Test_order_02,30
      27/1/2015,totesttime,31

      23/1/2015,Test_order_18,33

      (Note: Eg data may not match the data in the log file)

      Result: Kept logging error continuously until I stopped camel process
      Shouldn’t this be handled not to log error continuously? (Though it is a minor issue)

      [TSAApp@aml-qa3-core1 aml-merchant-leads]$ tail -f aml-merchant-leads.log
      [ main] MainSupport INFO Apache Camel 2.14.0 starting
      [ main] SpringCamelContext INFO Apache Camel 2.14.0 (CamelContext: camel-1) is starting
      [ main] ManagedManagementStrategy INFO JMX is enabled
      [ main] DefaultTypeConverter INFO Loaded 178 type converters
      [ main] SpringCamelContext INFO AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
      [ main] SpringCamelContext INFO StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
      [ main] SpringCamelContext INFO Route: leads-data-loader started and consuming from: Endpointfile:///var/local/tsa/leads/transformed?delay=5000&exclude=.*%5C.camelLock&move=done
      [ main] SpringCamelContext INFO Route: leads-csv-handler started and consuming from: Endpointfile:///var/local/tsa/leads/incoming?delay=5000&exclude=.*%5C.camelLock&move=done
      [ main] SpringCamelContext INFO Total 2 routes, of which 2 is started.
      [ main] SpringCamelContext INFO Apache Camel 2.14.0 (CamelContext: camel-1) started in 0.785 seconds
      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      15/1/2015,Test_order_9,160
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      15/1/2015 cd6d285c-7623-4b54-8421-3ed9b9fc4c81 160

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      14/1/2015,3310-4,161
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      14/1/2015 6bcb2ac3-fb89-4cc1-89d5-3942b4beca28 161

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      14/1/2015,3310-3,162
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      14/1/2015 c45600a4-378e-44d5-bc97-c1b3399b8f0d 162

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      13/1/2015,3205-6,163
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      13/1/2015 3c30760a-6a75-4661-a546-174ee5252805 163

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      12/1/2015,abc,164
      //var/local/tsa/leads/incoming MerchantLeadsCSVHandler ERROR Unable to connect to database
      org.springframework.dao.EmptyResultDataAccessException: Incorrect result size: expected 1, actual 0
      at org.springframework.dao.support.DataAccessUtils.requiredSingleResult(DataAccessUtils.java:71)
      at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:732)
      at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:747)
      at com.thesearchagency.merchant.leads.MerchantLeadsCSVHandler.getUUID(MerchantLeadsCSVHandler.java:54)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:408)
      at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:279)
      at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:252)
      at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:167)
      at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105)
      at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:67)
      at org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:103)
      at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)
      at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
      at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:589)
      at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:517)
      at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:226)
      at org.apache.camel.processor.Splitter.process(Splitter.java:104)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
      at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
      at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
      at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
      at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      09/1/2015,Test_order_6,165
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      09/1/2015 2f8ef882-dd01-4eb0-80aa-53f2bb53d720 165

      [ar/local/tsa/leads/transformed] ExecProducer INFO Executing ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false]
      [ar/local/tsa/leads/transformed] ExecProducer INFO The command ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false] had exit value 0
      [ar/local/tsa/leads/transformed] leads-data-loader INFO cmd succeeded. result is: 0
      ^C
      [TSAApp@aml-qa3-core1 aml-merchant-leads]$ clear
      [TSAApp@aml-qa3-core1 aml-merchant-leads]$ clear
      [TSAApp@aml-qa3-core1 aml-merchant-leads]$ tail -f aml-merchant-leads.log

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      09/1/2015,Test_order_6,165
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      09/1/2015 2f8ef882-dd01-4eb0-80aa-53f2bb53d720 165

      [ar/local/tsa/leads/transformed] ExecProducer INFO Executing ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false]
      [ar/local/tsa/leads/transformed] ExecProducer INFO The command ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false] had exit value 0
      [ar/local/tsa/leads/transformed] leads-data-loader INFO cmd succeeded. result is: 0
      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      15/1/2015,Test_order_9,160
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      15/1/2015 cd6d285c-7623-4b54-8421-3ed9b9fc4c81 160

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      14/1/2015,3310-4,161
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      14/1/2015 6bcb2ac3-fb89-4cc1-89d5-3942b4beca28 161

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      14/1/2015,3310-3,162
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      14/1/2015 c45600a4-378e-44d5-bc97-c1b3399b8f0d 162

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      13/1/2015,3205-6,163
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      13/1/2015 3c30760a-6a75-4661-a546-174ee5252805 163

      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:

      //var/local/tsa/leads/incoming DefaultErrorHandler ERROR Failed delivery for (MessageId: ID-aml-qa3-core1-59941-1422947881679-0-11 on ExchangeId: ID-aml-qa3-core1-59941-1422947881679-0-17). Exhausted after delivery attempt: 1 caught: java.lang.IllegalArgumentException: No records have been defined in the CSV

      Message History
      ---------------------------------------------------------------------------------------------------------------------------------------
      RouteId ProcessorId Processor Elapsed (ms)
      [leads-csv-handler ] [leads-csv-handler ] file:///var/local/tsa/leads/incoming?delay=5000&exclude=.*%5C.camelLock&move=d [ 94]
      [leads-csv-handler ] [log3 ] [log ] [ 0]
      [leads-csv-handler ] [unmarshal1 ] [unmarshal[org.apache.camel.model.dataformat.BindyDataFormat@3e9d9edd] ] [ 1]

      Exchange
      ---------------------------------------------------------------------------------------------------------------------------------------
      Exchange[
      Id ID-aml-qa3-core1-59941-1422947881679-0-17
      ExchangePattern InOnly
      Headers

      {breadcrumbId=ID-aml-qa3-core1-59941-1422947881679-0-11, CamelFileAbsolute=true, CamelFileAbsolutePath=/var/local/tsa/leads/incoming/call-leads-test-input.txt, CamelFileLastModified=1422948406000, CamelFileLength=118, CamelFileName=call-leads-test-input.txt, CamelFileNameConsumed=call-leads-test-input.txt, CamelFileNameOnly=call-leads-test-input.txt, CamelFileParent=/var/local/tsa/leads/incoming, CamelFilePath=/var/local/tsa/leads/incoming/call-leads-test-input.txt, CamelFileRelativePath=call-leads-test-input.txt, CamelRedelivered=false, CamelRedeliveryCounter=0}

      BodyType String
      Body
      ]

      Stacktrace
      ---------------------------------------------------------------------------------------------------------------------------------------
      java.lang.IllegalArgumentException: No records have been defined in the CSV
      at org.apache.camel.dataformat.bindy.csv.BindyCsvDataFormat.unmarshal(BindyCsvDataFormat.java:200)
      at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:67)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
      at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:589)
      at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:517)
      at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:226)
      at org.apache.camel.processor.Splitter.process(Splitter.java:104)
      at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
      at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
      at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
      at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
      at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
      at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
      at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      09/1/2015,Test_order_6,165
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      09/1/2015 2f8ef882-dd01-4eb0-80aa-53f2bb53d720 165

      //var/local/tsa/leads/incoming GenericFileOnCompletion WARN Rollback file strategy: org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@15fddb33 for file: GenericFile[/var/local/tsa/leads/incoming/call-leads-test-input.txt]
      [ar/local/tsa/leads/transformed] ExecProducer INFO Executing ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false]
      [ar/local/tsa/leads/transformed] ExecProducer INFO The command ExecCommand [args=[/var/local/tsa/config/com.thesearchagency.datafeeds.calldata.properties, /var/local/tsa/leads/transformed/call-leads-test-input.txt, 1], executable=/usr/local/tsa/bidmgr/leads-data-loader.sh, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=false] had exit value 0
      [ar/local/tsa/leads/transformed] leads-data-loader INFO cmd succeeded. result is: 0
      //var/local/tsa/leads/incoming leads-csv-handler INFO row in:
      15/1/2015,Test_order_9,160
      //var/local/tsa/leads/incoming leads-csv-handler INFO row out:
      15/1/2015 cd6d285c-7623-4b54-8421-3ed9b9fc4c81 160

        Attachments

          Activity

            People

            • Assignee:
              atul.sanagar Atul Sanagar (Inactive)
              Reporter:
              varshini.padmanabhan Varshini Padmanabhan (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: