You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Steps:
We send/produce large sets of messages to Kafka of around 20 to 30 KB size each messages in JSON format and producing around 200 messages / sec for 1 hour duration. We have 3 Kafka Brokers running and I am trying to consume the messages from these 3 Kafka Brokers from the same topic using the above code. The problem is that sometimes some of the messages gets truncated, I am not sure why it happen ?
Activity
rgowdathreatmetrixcom commentedon Jun 6, 2016
Snippet code is below:
from kafka import KafkaConsumer
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer
Steps:
We send/produce large sets of messages to Kafka of around 20 to 30 KB size each messages in JSON format and producing around 200 messages / sec for 1 hour duration. We have 3 Kafka Brokers running and I am trying to consume the messages from these 3 Kafka Brokers from the same topic using the above code. The problem is that sometimes some of the messages gets truncated, I am not sure why it happen ?
ned-deily commentedon Jun 6, 2016
Sorry, kafka is a third-party package, not part of the Python Standard Library. Suggest you pursue this with the Kafka project:
http://kafka-python.readthedocs.io/en/master/support.html