在我的应用程序中,正在使用apache骆驼处理txt文件,应用程序使用交换对象将整个文件读入字符串,并在每行上进行分界符添加,并使用apache骆驼的交换对象将其作为整个字符串写入文件。我低于OutOfMemoryError。 ***我观察到此异常是在交换对象完成文件处理之后以及应将文件移动到另一个文件夹(端点)时发生的。

我的应用程序具有16GB的堆空间支持。处理前的文件大小为685MB,添加分隔符后可能会增加200 MB。
有人能帮我一下吗?先感谢您。

from("file:<filepath>/processed/INPUT?filter=#fileFilter&delay=120000&exclusiveReadLockStrategy=#fileReadLockStrategy&delete=true&autoCreate=true")
                    .threads(10, 10)
                    .process(fileProcessor).log("Came out of file processor")
                    .recipientList(header(DESTIONATION_PATH));


从INPUT文件夹和fileProcessor中读取文件的简单途径是通过在其中添加delimter来处理文件内容。 DESTIONATION_PATH是将放置文件的路径。我可以看到它正在打印日志“文件处理器超出限制”,下面是当文件即将发送到目标文件夹时出现异常。

Error has occurred: 
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
    at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1842) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:385) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:64) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
    Suppressed: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
        ... 26 common frames omitted
    ***Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit***
        at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
        at java.lang.StringCoding.encode(StringCoding.java:344)
        at java.lang.String.getBytes(String.java:918)
        at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198)
        at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55)
        at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63)
        at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299)
        at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188)
        at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115)
        at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333)
        at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305)
        at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
        at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80)
        at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
        at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638)
        at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
        at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173)
        at org.apache.camel.processor.RecipientList.process(RecipientList.java:133)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43)
        at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157)
        at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87)
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300) ~[na:1.8.0_171]
    at java.lang.StringCoding.encode(StringCoding.java:344) ~[na:1.8.0_171]
    at java.lang.String.getBytes(String.java:918) ~[na:1.8.0_171]
    at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
    at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]

最佳答案

很抱歉在这里迟到。我能够解决上述问题。
当我仔细查看日志时,它在标准Java类“ StringCoding.java”中失败。

我的代码中的罪魁祸首是以下两行:

String content = exchange.getIn().getBody(String.class); 
//Add deleimters in each line content string & write back the whole string in file.
//derive new content string from original content
exchange.getOut().setBody(content);// Camel stores content in String & sends it via route using character array. 


分析:

Java将字符串作为字符数组存储在后台。根据逻辑,我们在每行中添加了一些距离。
我观察到(原始文件中的字符数+添加的分隔符数)> Integer.MAX_VALUE。
因此,这里的Java无法将整个字符串存储到字符数组中,并且骆驼无法将文件放入
在目标文件夹路径中。发生这种情况是因为修改后的文件中的字符数超过了最大整数大小。

解:

File contentFile = exchange.getIn().getBody(File.class);
//Process the contentFile as file line by line & write the records back in file.
exchange.getOut().setBody(contentFile);// Camel stores content in File & sends it via route using InputStream. 


我保持上面的代码,并且它起作用了,我们没有看到OutofMemory问题,因为在通过路由发送文件时,骆驼不在哪里使用数组。

感谢您的阅读。快乐编码... !!!

08-04 15:33