Failing Kafka Verbraucher Leistungstest

Question

Der Befehl auszuführen:

./bin/kafka-consumer-perf-test.sh --topic test_topic --messages 500000 --zookeeper zookeepernode --show-detailed-stats --consumer.config ./conf/connect-distributed.properties 

führt zu

{metadata.broker.list=kafkanode1:9093,kafkanode2:9093,av3l338p.kafkanode3:9093, request.timeout.ms=30000, client.id=perf-consumer-68473, security.protocol=SASL_SSL} 

WARN Fetching topic metadata with correlation id 0 for topics [Set(test_topic)] from broker [BrokerEndPoint(1003,kafkanode3,9093)] failed (kafka.client.ClientUtils$) 
java.io.EOFException 
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99) 


[...] 
WARN Fetching topic metadata with correlation id 0 for topics [Set(test_topic)] from broker [BrokerEndPoint(1001,kafkanode1,9093)] failed (kafka.client.ClientUtils$) 
java.nio.channels.ClosedByInterruptException 
    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) 

[...] 
WARN Fetching topic metadata with correlation id 0 for topics [Set(test_topic)] from broker [BrokerEndPoint(1002,kafkanode2,9093)] failed (kafka.client.ClientUtils$) 
java.nio.channels.ClosedChannelException 
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:122) 

Answer 1

Versuchen Sie, fügen Sie das Flag "--new-Verbraucher":

./bin/kafka-consumer-perf-test.sh --new-consumer --broker-list <broker-host>:9093 --topic <topic-read-from> --messages <number-of-messages> --show-detailed-stats --consumer.config consumer.properties --zookeeper <zookeeper-host>:2181 

Die consumer.properties sieht so aus (einschließlich SASL_SSL- und Kerberos-Konfiguration):

zookeeper.connect=<zookeeper-host>:2181 

# timeout in ms for connecting to zookeeper 
zookeeper.connection.timeout.ms=6000 

#consumer group id 
group.id=<consumer-group> 

security.protocol=SASL_SSL 
ssl.keystore.location=<path-to-keystore-file> 
ssl.keystore.password=<password> 
ssl.truststore.location=<path-to-truststore-file> 
ssl.truststore.password=<password> 

sollte die Ausgabe wie folgt aussehen:

time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 
2017-06-01 11:29:15:194, 0, 1,6726, 4,1816, 5000, 12500,0000 
2017-06-01 11:29:15:207, 0, 3,3916, 859,4761, 10000, 2500000,0000 
2017-06-01 11:29:15:208, 0, 5,1009, 1709,3201, 15000, 5000000,0000 
2017-06-01 11:29:15:209, 0, 6,7850, 1684,1164, 20000, 5000000,0000 
2017-06-01 11:29:15:291, 0, 8,5101, 21,2977, 25000, 61728,3951 
2017-06-01 11:29:15:460, 0, 10,2138, 10,0808, 30000, 29585,7988 
2017-06-01 11:29:15:462, 0, 11,9258, 1711,9637, 35000, 5000000,0000 
2017-06-01 11:29:15:463, 0, 13,6348, 1709,0416, 40000, 5000000,0000 
2017-06-01 11:29:15:673, 0, 15,3255, 8,0511, 45000, 23809,5238 
2017-06-01 11:29:15:962, 0, 17,0417, 5,9384, 50000, 17301,0381 
2017-06-01 11:29:15:963, 0, 18,7520, 1710,2280, 55000, 5000000,0000 
2017-06-01 11:29:15:963, 0, 20,4568, Infinity, 60000, Infinity 
2017-06-01 11:29:16:090, 0, 22,1223, 13,1138, 65000, 39370,0787 
2017-06-01 11:29:16:282, 0, 23,8131, 8,8062, 70000, 26041,6667