Große CSV mit Apache Camel + Aggregator

Question

generieren große Dateien mit Apache Camel mit Aggregator mit einer besseren Leistung als diesen Beitrag: Large Files with Apache Camel

Mein Körper hat 33.352 Zeilen .. und mit completionSize="1000" und completionTimeout="2500" die letzte Datei verpasste die letzten 352 Zeilen

<camel:split streaming="true"> 
    <camel:simple>${body}</camel:simple> 
    <camel:marshal> 
    <camel:csv quote='"' quoteDisabled="false" headerDisabled="true" /> 
    </camel:marshal> 
    <camel:aggregate strategyRef="setfepCsvStringBodyAggregator" completionSize="1000" completionTimeout="2500"> 
    <camel:correlationExpression> 
     <constant>true</constant> 
    </camel:correlationExpression> 
    <to uri="file:{{setfep_dir_inprogress}}/?fileName={{setfep_filename_clientes}}.txt&amp;fileExist=Append" /> 
    </camel:aggregate> 
</camel:split> 

Schluss Datei hat 33.000 Zeilen 352.

[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 
[INFO ] org.apache.camel.util.CamelLogger.log - Complete by=size rows=1000 

fehlen, wenn ichverwendenund completionInterval="2500" meine letzte Datei hat 33155 Hä fehlt 197 Zeilen.

[Camel thread #0 - AggregateTimeoutChecker] [INFO ] CamelLogger.log - Complete by=interval rows=566 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[Camel thread #0 - AggregateTimeoutChecker] [INFO ] CamelLogger.log - Complete by=interval rows=43 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[Camel thread #0 - AggregateTimeoutChecker] [INFO ] CamelLogger.log - Complete by=interval rows=401 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[Camel thread #0 - AggregateTimeoutChecker] [INFO ] CamelLogger.log - Complete by=interval rows=768 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[Camel thread #0 - AggregateTimeoutChecker] [INFO ] CamelLogger.log - Complete by=interval rows=377 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 
[main] [INFO ] CamelLogger.log - Complete by=size rows=1000 

Wie behebt man das?

Apache Camel 2.19.0

Answer 1

Gotcha!

ich als Unit-Test läuft ... endet der Kontext vor dem vollständigen completionInterval oder completionTimeout

, wenn ich eine Verzögerung in meinem Weg habe ich schrieb die insgesamt 33.352 Zeilen sehen konnte.

<camel:delay> 
    <constant>5000</constant> 
</camel:delay> 

Aber in der Produktion nicht notwendig ist, weiterhin der Kontext am Leben oder wir die Option forceCompletionOnStop verwenden können:

<camel:aggregate strategyRef="setfepCsvStringBodyAggregator" forceCompletionOnStop="true" completionSize="1000" completionInterval="4000">