Skip to content

Kafka Python Consumer Messages gets truncated #71424

Closed

Description

@rgowdathreatmetrixcom
BPO 27237
Nosy @ned-deily

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-06-06.00:12:30.561>
created_at = <Date 2016-06-06.00:06:03.398>
labels = ['library', 'performance']
title = 'Kafka Python Consumer Messages gets truncated'
updated_at = <Date 2016-06-06.00:12:30.559>
user = 'https://bugs.python.org/rgowdathreatmetrixcom'

bugs.python.org fields:

activity = <Date 2016-06-06.00:12:30.559>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2016-06-06.00:12:30.561>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2016-06-06.00:06:03.398>
creator = 'rgowda@threatmetrix.com'
dependencies = []
files = []
hgrepos = []
issue_num = 27237
keywords = []
message_count = 2.0
messages = ['267487', '267490']
nosy_count = 2.0
nosy_names = ['ned.deily', 'rgowda@threatmetrix.com']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue27237'
versions = ['Python 2.7']

Activity

  1. rgowdathreatmetrixcom commented on Jun 6, 2016

    rgowdathreatmetrixcommannequin
    MannequinAuthor

    Snippet code is below:
    from kafka import KafkaConsumer
    from kafka.client import KafkaClient
    from kafka.consumer import SimpleConsumer

    consumer = KafkaConsumer('eventdetails_ingestion' , group_id='1',bootstrap_servers=‘xxxx:9092', max_partition_fetch_bytes=10555555)
    for msg in consumer:
        try:
            jValue = json.loads(str(msg.value))
       except ValueError:
            fileErr.write(str(msg.value)+"\n")

    Steps:
    We send/produce large sets of messages to Kafka of around 20 to 30 KB size each messages in JSON format and producing around 200 messages / sec for 1 hour duration. We have 3 Kafka Brokers running and I am trying to consume the messages from these 3 Kafka Brokers from the same topic using the above code. The problem is that sometimes some of the messages gets truncated, I am not sure why it happen ?

  2. added
    stdlibStandard Library Python modules in the Lib/ directory
    performancePerformance or resource usage
    on Jun 6, 2016
  3. ned-deily commented on Jun 6, 2016

    @ned-deily
    Member

    Sorry, kafka is a third-party package, not part of the Python Standard Library. Suggest you pursue this with the Kafka project:

    http://kafka-python.readthedocs.io/en/master/support.html

  4. transferred this issue fromon Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions