Questions tagged [apache-kafka]

Apache Kafka is a distributed streaming platform designed to store and process high-throughput data streams.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

At high load getting “Received invalid metadata error in produce request ” kafka producer

I have a producer writing a million messages in loop to kafka. After processing around half million messages (not exact, varies in different runs) it starts giving Received invalid metadata error in ...
0
votes
0answers
7 views

How to skip an Avro serialization exception in Confluent KafkaStreams API?

I have a Kafka application that is written by Confluent KafkaStreams Java api. It reads data from Mysql binlog and do some stuff that is irrelevant to my question. The problem is one particular row ...
0
votes
0answers
15 views

KafkaIO Connector “go” SDK Available?

I am working on building a data ingestion pipeline using Apache Beam "go" SDK. My pipeline is to consume data from Kafka queue and persist the data to Google Cloud Bigtable (and/or to another Kafka ...
1
vote
1answer
18 views

How can i apply a filter in this avro schema using Apache Flink

Hello guys i'm deserializing an Avro Kafka message like this: FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer("conekta.public.person", new ...
1
vote
0answers
14 views

Parallelize transformation step in kakfa

In my kafka application I'm currently reading from a topic. The incoming messages containing BasicRequest are filtered and two additional transformations are applied. This result is then written in ...
2
votes
1answer
26 views

If i set 'compression.type' at topic level and producer level, which takes precedence

I'm trying to understand the 'compression.type' configuration and my question is, If i set 'compression.type' at topic level and producer level, which takes precedence?
0
votes
0answers
10 views

Can kafka streams state store be used for debouncing events?

I have a particular use case wherein I want to wait for a few seconds to capture similar messages and send the latest one. I'm going through this book - Kafka Streams in Action by William P. Bejeck ...
0
votes
0answers
12 views

Knative Eventing KafkaSource issue

Ive got an issue with Knative Eventing and I would like to have your help as I am fresh about it. I am using Knative v0.10.0, minikube v1.5.2, Docker v19.03.1 and Kafka 2.3.1 (used as the source of ...
0
votes
0answers
14 views

Can't connect to Kafka using CONDUKTOR UI

I have been connecting to KAFKA using CONDUKTOR Application. But somehow Conduktor connection is disabled (yellow highlighted in the screenshot), and I am unable to connect to kafka, even after ...
0
votes
1answer
12 views

Kafka Connect: can multiple standalone connectors write to the same HDFS directory?

For our pipeline, we have about 40 topics (10-25 partitions each) that we want to write into the same HDFS directory using HDFS 3 Sink Connectors in standalone mode (distributed doesn't work for our ...
1
vote
2answers
20 views

How to handle exceptions and message reprocessing in apache kafka

I have a kafka cluster. There is only one topic and to this topic 3 different consumer groups are taking the same messages from the topic and processing differently according to their own logic. is ...
0
votes
1answer
25 views

Dynamic Filtering of list in Java

I am developing a java application that receives realtime data (Kafka/Kafka Streams) from sources. In my case study, I need to filter the names that is part of my list. The list has 10 million names ...
0
votes
1answer
20 views

Disabling Kafka Listeners for a particular Spring Boot test

How to disable @KafkaListener instances in @SpringBootTest tests in applications with Spring Boot (2.2+) and Spring Kafka (2.4+)? The goal is to disable Kafka listeners in particular tests, so that ...
1
vote
2answers
18 views

Kafka Connect, JDBC connector password in clear text. How to avoid?

I have a jdbc source connector that I'm using, and I've been using Postman to test and set this. The password for the db connection is in clear text, which is fine as long as I'm the only one seeing ...
-1
votes
1answer
23 views

Error connecting to local Bitnami Docker Kafka from Spring Boot application

Spring Boot (version 2.2) application with Spring Kafka (version 2.4) can't establish a connection with Bitnami Docker Kafka (version 2) executed from the official docker-compose.yml version: '2' ...
0
votes
0answers
12 views

Why are my storm topology not acking when i send tuple to ElasticSearch

I'm new to using Storm, I've just started a Data Architect training course and it's in this context that I'm facing the problem that brings me to you today. I'm receiving messages from kakfa via a ...
0
votes
0answers
10 views

How to create Asynchronous Camel-kafka consumer?

I have a route in camel to consumer from kafka. It is consuming and producing with a TPS of 2000 if the incoming message is 18000 TPS. so the consumer topic has consumer lag. If I keep max.poll....
0
votes
0answers
20 views

Communicate with Kafka connectors

I have to make a survey about the fellows Kafka connectors : Fivetran Tray.io TIBCO Azure Logic App Power Automate I want to know if these tools communicate with Kafka using REST and / or ...
0
votes
0answers
21 views

Spring-kafka-listener: Offsets remain after last message using manual acknowledge

in my current project I'm using spring-kafka to send, transform and retrieve messages. The producer sends messages of type A to a kafka-topic topic-A, then a kafka-streams instance reads the messages ...
0
votes
0answers
22 views

Why kafka show error when try see offset?

I run command: > /bin/kafka-consumer-groups.sh --bootstrap-server 5.189.239.3:9092 > --group consumer-group1 --describe and get this error: Error: Executing consumer group command failed ...
0
votes
2answers
40 views

Integrate kafka Consumer spring batch

I have a Kafka Consumer developed in spring-boot and i am able to read the messages from the topic. I want to integrate it with the Spring batch because i want to create an batch file. I am not sure ...
0
votes
0answers
16 views

Debezium Kafka Connect Error - closing the kafka producer with timeoutMillis

I'm getting an error on Debezium connect producer, and not sure where I'm making mistake or missing out something.Below are the properties for my connector and docker file. Is it possible that docker ...
0
votes
0answers
30 views

Alternative of Confluent REST Proxy

We have some applications which want to communicate with Kafka using REST API calls to both consume and produce messages. If we do not want to use Confluent REST Proxy, what are the options ?
0
votes
2answers
27 views

Deserialize avro in apache flink using confluent

i'm trying to deserialize an Avro message from kafka on apache flink i'm currently doing it by implementing the DeserializationSchema interface but it is depecrated is there a better form to achieve ...
0
votes
1answer
50 views

Reactive Messaging: Emit events when needed (using Kafka)

I am just learning Quarkus and Reactive Messaging. I am trying to message between two components. The examples I have found have demonstrated streams that have a known data set that is streamed or are ...
0
votes
1answer
21 views

Processing a maximum number of tasks per consumer in a topic, for a given time window?

My producer generates n tasks from a single input message and publishes these on topic. The requirement is that out of all the individual consumers in the consumer group of topic, no one of them ...
0
votes
2answers
38 views

kafka MirrorMaker 2.0 does not replicate data

I am trying to replicate Kafka cluster with MirrorMaker 2.0. I am using following mm2.properties: # mm2.properties clusters=source, dest source.bootstrap.servers=kafka-1:9092,kafka-2:19092,kafka-3:...
0
votes
2answers
34 views

How to send Kafka message from one topic to another topic?

suppose my producer is writing the message to Topic A...once the message is in Topic A, i want to copy the same message to Topic B. Is this possible in kafka?
0
votes
1answer
18 views

kafka connect mongo on kafka MSK

I am using Kafka MSK in AWS. So we don't have native kafka connect with all required connectors like on confluent. Actually I work with kakfa mongo connector and I want to find a way to push the ...
1
vote
2answers
35 views

Running a single kafka s3 sink connector in standalone vs distributed mode

I have a kafka topic "mytopic" with 10 partitions and want to use S3 sink connector to sink records to an S3 bucket. For scaling purposes it should be running on multiple nodes to write partitions ...
0
votes
3answers
43 views

Kafka Producer API: bootstrap.servers property value

I am learning Kafka Producer API, and inside tutorials, they have mentioned "bootstrap.servers" is mandatory property to specify currently running brokers(by comma-separated value). I doubted why the ...
-1
votes
1answer
32 views

How to solve the classNotFoundException warnings when running Kafka Connect on windows?

I have checked multiple topics on the StackOverflow and Kafka Jira board and found that these warnings can be suppressed from connect-log4j.properperties file by setting reflections to log at error ...
0
votes
1answer
56 views

Kafka consumer in group skips the partitions

I've a single consumer which consumes a topic. Topic has 6 partitions. Single consumer assigned to the group. I do poll like below Consumer.poll(10000) I exit the consumer fetch when no records ...
0
votes
0answers
44 views

Listening to multiple topics on single consumer

I have recently started working on Apache Kafka. One thing which I keep on seeing on various blogs is that multiple topics are configured to the same listener. My question is, Is it a good practice ...
0
votes
0answers
8 views

librdkafka: rd_kafka_assignment returns offset -1001 for assigned partitions

When i query my consumer for the assigned Topic Partition List all of the Partitions in the result has an offset of -1001. If i print out the offset of received Message, the Offset is set to the right ...
1
vote
1answer
27 views

SSD or HDD for Kafka Brokers? ( Using SSD for Kafka )

Kafka is fast because it uses sequential writing techniques on HDD. If I use SSD for Kafka Brokers, do I get faster performance? As far as I know, SSD works differently than HDD. And I think with ...
1
vote
0answers
36 views

Spring-Kafka consumer coordinator unavailable or invalid looping continuously

I have three pods in a Kubernetes node. Pod-1: Single node Kafka broker Pod-2: Single node Zookeeper Pod-3: Several Spring Boot Kafka enabled applications Earlier on we were on Docker with similar ...
0
votes
1answer
14 views

Kafka : Switching source from MySQL to Kafka

Event data coming from both MySQL and Kafka topics. We will load historical data from MySQL and then switch to Kafka for current data.Mysql IDs also flows into Kafka . So messages contain event IDs . ...
0
votes
2answers
25 views

Does even number of Kafka brokers affect ensemble operation and leader election

we are little confuse about if is valid or not to use kafka cluster with even numbers the story begin when we build kafka cluster with 6 machines and trying to use the kafka Reassignment partitions ...
0
votes
0answers
12 views

Cannot run multiple test classes simultaneously if both the tests use KafkaConsumer bean

I have two separate test classes. Both of them when boot up individually create two beans each of instance KafkaConsumer. These tests are for event listeners which use these consumers to poll from the ...
0
votes
1answer
24 views

Kafka Auto-Commit doesn't effect for slower transactions

Using Spring Boot Kafka - v 2.2.1.RELEASE (which uses kafka-clients.jar v 2.3.1) I'm using confluent v 5.1.2 for my Kafka Broker. I've got enabled the autocommit to true and autocommit interval ...
3
votes
1answer
29 views

Scala Jar file not working properly for Kafka Streams app

I made a jar file for a Kafka Streams application coded in Scala using both IntelliJ IDEA itself and then SBT. Following are the problems I have been facing and so far I haven't succeeded: First, I ...
-2
votes
0answers
12 views

Kafka batch processing and Transformation data in pyspark, Write to HDFS

Im using pyspark to read Kafka data, as batch processing and on the fly Im trying to do transformations (aggregations/join with different data set) as well and planning on writing it to Hadoop. ...
1
vote
1answer
18 views

KafkaAvroDeserializer failing with Kyro Exception

I have written a consumer to read Avro's generic record using a schema registry. FlinkKafkaConsumer010 kafkaConsumer010 = new FlinkKafkaConsumer010(KAFKA_TOPICS, new ...
0
votes
0answers
39 views

Kafka Consumer- ClassCastException java

My Kafka consumer throws an exception when trying to process messages in a batch(i.e process list of messages) Error Message is java.lang.ClassCastException: class kafka.psmessage.PMessage cannot ...
-1
votes
1answer
37 views

“kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection” ONLY DURING LISTING TOPICS

I found few questions with similar topic but different context: I can connect to create a Topic but I can't list the topics because I got the error mentioned below (as far as I could see, people were ...
0
votes
0answers
21 views

Kafka consumer best practice with kafka dot net

I create a new web service (asp .net core) that consume messages from Kafka using .NET client for Kafka(confluent-kafka-dotnet) and I searched pretty much about example and best practices for consume ...
0
votes
2answers
26 views

Consuming wierd data from Kafka docker image

Hello guys i'm using the confluent kafka docker confluentinc/cp-server:5.4.0 I'm producing data from a postgresql, when i make a consumer and want to retireve the kafka message it appears in a ...
0
votes
0answers
48 views

Sending COBOL record to Kafka Topic

I have a COBOL batch program that needs to place messages into a Kafka topic. I know that there are functions available to allow me to dump a file or DB2 table into a Kafka topic from the mainframe. ...
2
votes
2answers
60 views

Kafka producer is connecting to localhost instead of the real IP

Please help I'm wondering why the kafka producer always connect to the localhost however there the broker ip is not the localhost. So, is there any help ? any ideas ? import org.apache.kafka.clients....