Verwendung kafka-producer-perf-test.sh wie Produzent Config bei kafka_2.10-0.8.2.0

Question

bin/kafka-producer-perf-test.sh --messages 1000000 --message-size 1000 --topics kafka_producertest_1replication_1partitions --broker-list mesos006:9092,mesos007:9092,mesos008:9092 --threads 1 --compression-codec 3 --batch-size 100000 

Ausnahme setzen:

Exception in thread "main" kafka.common.InvalidConfigException: Batch size = 100000 can't be larger than queue size = 10000 
     at kafka.producer.ProducerConfig$.validateBatchSize(ProducerConfig.scala:39) 
     at kafka.producer.ProducerConfig$.validate(ProducerConfig.scala:29) 
     at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:116) 
     at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:56) 
     at kafka.producer.OldProducer.<init>(BaseProducer.scala:59) 
     at kafka.tools.ProducerPerformance$ProducerThread.<init>(ProducerPerformance.scala:210) 
     at kafka.tools.ProducerPerformance$$anonfun$main$1.apply$mcVI$sp(ProducerPerformance.scala:58) 
     at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) 
     at kafka.tools.ProducerPerformance$.main(ProducerPerformance.scala:57) 
     at kafka.tools.ProducerPerformance.main(ProducerPerformance.scala) 

config/producer.properties:

queue.buffering.max.messages=100000 

Bei der Verwendung von kafka-producer-perf-test.sh, wie man alle Parameter des Produzenten einstellt. Oder geben Sie ihm eine Konfigurationsdatei (zum Beispiel: config/producer.properties) wie tow make config/producer.properties Wirkung?

kafka-producer-perf-test.sh Parameter

[root@mesos006 kafka_2.10-0.8.2.0]# bin/kafka-producer-perf-test.sh 
Missing required argument "[topics]" 
Option         Description        
------         -----------        
--batch-size <Integer: size>   Number of messages to write in a  
              single batch. (default: 200)   
--broker-list <hostname:port,..,  REQUIRED: broker info (the list of  
    hostname:port>       broker host and port for bootstrap. 
--compression-codec <Integer:   If set, messages are sent compressed 
    supported codec: NoCompressionCodec  (default: 0)       
    as 0, GZIPCompressionCodec as 1,            
    SnappyCompressionCodec as 2,             
    LZ4CompressionCodec as 3>              
--csv-reporter-enabled     If set, the CSV metrics reporter will 
              be enabled       
--date-format <date format>    The date format to use for formatting 
              the time field. See java.text.  
              SimpleDateFormat for options.   
              (default: yyyy-MM-dd HH:mm:ss:SSS) 
--help         Print usage.       
--hide-header       If set, skips printing the header for 
              the stats        
--initial-message-id <Integer: initial The is used for generating test data, 
    message id>        If set, messages will be tagged with 
              an ID and sent by producer starting 
              from this ID sequentially. Message 
              content will be String type and in 
              the form of 'Message:000...1:xxx...' 
--message-send-gap-ms <Integer:   If set, the send thread will wait for 
    message send time gap>     specified time between two sends  
              (default: 0)       
--message-size <Integer: size>   The size of each message. (default:  
              100)         
--messages <Long: count>    The number of messages to send or  
              consume (default:      
              9223372036854775807)     
--metrics-dir <metrics dictory>   If csv-reporter-enable is set, and  
              this parameter isset, the csv   
              metrics will be outputed here   
--new-producer       Use the new producer implementation. 
--producer-num-retries <Integer>  The producer retries number (default: 
              3)         
--producer-retry-backoff-ms <Integer> The producer retry backoff time in  
              milliseconds (default: 100)   
--reporting-interval <Integer: size> Interval at which to print progress  
              info. (default: 5000)     
--request-num-acks <Integer>   Number of acks required for producer 
              request to complete (default: -1)  
--request-timeout-ms <Integer>   The produce request timeout in ms  
              (default: 3000)      
--show-detailed-stats     If set, stats are reported for each  
              reporting interval as configured by 
              reporting-interval     
--sync         If set, messages are sent    
              synchronously.      
--threads <Integer: number of threads> Number of sending threads. (default: 1) 
--topics <topic1,topic2..>    REQUIRED: The comma separated list of 
              topics to produce to     
--vary-message-size      If set, message size will vary up to 
              the given maximum. 

Answer 1

die Dokumentation an das Skript angebracht Überprüfen Sie verwenden:

usage: producer-performance [-h] --topic TOPIC --num-records NUM-RECORDS [--payload-delimiter PAYLOAD-DELIMITER] --throughput THROUGHPUT 
          [--producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]] [--producer.config CONFIG-FILE] (--record-size RECORD-SIZE | --payload-file PAYLOAD-FILE) 

This tool is used to verify the producer performance. 

optional arguments: 
    -h, --help    show this help message and exit 
    --topic TOPIC   produce messages to this topic 
    --num-records NUM-RECORDS 
         number of messages to produce 
    --payload-delimiter PAYLOAD-DELIMITER 
         provides delimiter to be used when --payload-file is provided. Defaults to new line. Note that this parameter will be ignored if --payload-file is not 
         provided. (default: \n) 
    --throughput THROUGHPUT 
         throttle maximum message throughput to *approximately* THROUGHPUT messages/sec 
    --producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...] 
         kafka producer related configuration properties like bootstrap.servers,client.id etc. These configs take precedence over those passed via --producer.config. 
    --producer.config CONFIG-FILE 
         producer config properties file. 

    either --record-size or --payload-file must be specified but not both. 

    --record-size RECORD-SIZE 
         message size in bytes. Note that you must provide exactly one of --record-size or --payload-file. 
    --payload-file PAYLOAD-FILE 
         file to read the message payloads from. This works only for UTF-8 encoded text files. Payloads will be read from this file and a payload will be randomly 
         selected when sending messages. Note that you must provide exactly one of --record-size or --payload-file. 

Ich denke, das Argument, das Sie suchen, ist:

--producer.config CONFIG-FILE 
         producer config properties file. 

Ich habe es nicht versucht, aber es ist es wert, es zu versuchen.

Answer 2

$bin/kafka-producer-perf-test.sh --topic my_topic --num-records 50 --throughput 10 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 1