Questions tagged [apache-kafka]

Apache Kafka is a distributed streaming platform designed to store and process high-throughput data streams.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Apache Kafka broker's OOM when rebalancing partitions

We have a Kafka region cluster of 14 brokers (on AWS instances), 53 topics by 128 partitions each. Last time when we added 3 new brokers, during partitions rebalance multiple brokers became OOM and ...
0
votes
0answers
17 views

Find total number of topic in kafka cluster

I have a kafka cluster with 3 brokers, I would like to know the total number of topics in the cluster. Is there a script or any command to find out the total.
0
votes
0answers
4 views

Spring cannot autowire KafkaTemplate when running tests with Embedded Kafka

I had the following test class with Spring Boot 2.0.8 and Spring Cloud Finchley.SR2: @EnableKafka @RunWith(SpringRunner.class) @SpringBootTest(classes = { FormNotificationService.class, ...
0
votes
0answers
6 views

Usign ConfluentMetricsReporter with CDK

We are using Confluent's Control Center alongside CDK (Cloudera Distributed Kafka). It is configurable to use the ConfluentMetricsReporter with CDK, so that Control Center can monitor the Kafka ...
0
votes
1answer
18 views

Kafka consumer group, set offset to 0 when consumer group is created

I'm creating a new Kafka consumer like this in Java (some code omitted for brevity) final Properties props = new Properties(); props.put(ConsumerConfig.GROUP_ID_CONFIG, "group2"); final ...
0
votes
1answer
27 views

prepare method executing multiple times

Hi I am creating a Topology using apache-storm in which my Spout is collecting data from Kakfa Topic and sending it to a bolt. I am doing some validation over the tuple and emitting stream again for ...
0
votes
0answers
17 views

kafka brokers intermittent traffic

i have kafka cluster with version 0.10.2 i try to generete 25 mb/sec to my topic wich have 400 partitions, the cluster behavior is very strange the followers stop receiving messages each 5 min but ...
0
votes
0answers
14 views

How do I sink the geospatial data from kafka topic to postgres?

I have been trying to sink the geospatial data from kafka topic to postgres db with postgis enabled. So the geospatial data is in form of array [lat, lng] at the moment in kafka topic. I want to sink ...
0
votes
0answers
12 views

Docker-secor BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker

I'm using docker-secor to get messages from kafka topic to google cloud storage. The app seems to start normally and creates all the consumers and watchers for the topics. But then fails with the next ...
0
votes
0answers
10 views

Hyperldegr composer network failed-SERVICE_UNAVAILABLE

I start an Hyperldger composer network. It worked a long time without any issue . But now it face an issue for the orderer server. The errors that(Orderer logs) Rejecting broadcast of normal message ...
0
votes
0answers
10 views

Kafka to Spark StructuredStreaming Getting exception on dataframe.WriteStream()

Using Spark 2.3.0 (using java) along with Kafka 2.2.0 for structured streaming, I am getting below exception: Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: No ...
0
votes
1answer
9 views

Spring Kafka test with embedded Kafka failed on deleting logs

I am using spring kafka with the embedded kafka for JUnit test, it gives an error for every test on windows: Error deleting C:\Users:LXX691\AppData\Local\Temp\kafka-1103610162480947200/.lock: The ...
0
votes
0answers
9 views

Kafka Streams' State Store throws FileNotFoundException

I am having the following exception from Kafka Streams (Version: 2.0.1, # of application instances: 9) after restarting the pods in Kubernetes cluster (/tmp location is its default value): java.io....
1
vote
2answers
14 views

Apache Storm: How to micro batch events from Kafka Spout

How can i micro batch events in kafka spout to reduce IO calls in the bolts that follow? The expectation is: emit a batch of maximum size 100 using events in kafka but wait maximum of 1 second to form ...
2
votes
1answer
12 views

kafka connector update rest api not update sink data

I am running a sink with wrong transform which produce wrong data in sink mysql table but when i correct sink configuration with PUT rest API and run it, it run successfully but not update existing ...
0
votes
1answer
28 views

Spring Boot Cannot Find EmbeddedKafkaBroker Bean

I am writing an integration tests for Kafka. For that, I am using Spring's EmbeddedKafka. Here's my class: @Component @EmbeddedKafka( partitions = 1, topics = {"Topic}, brokerProperties = ...
2
votes
1answer
25 views

Caused by: io.debezium.text.ParsingException: extraneous input 'ASC' expecting

I am running source kafka connector but unfortunately i am getting below error: {"name":"supplier-central","connector":{"state":"RUNNING","worker_id":"192.168.208.4:8083"},"tasks":[{"id":0,"state":"...
0
votes
0answers
21 views

Kafka connect in docker

I am trying to set up a Dockerized environment running kafka-connect and having a standalone connector (s3-sink). I am using docker-compose and Kafka, zookeeper and my producer all works beautifully, ...
0
votes
0answers
18 views

XML to Avro Schema

I am trying to export thousands of data records( telecom signalling parameters) to a 3rd party Kafka. The only possibility is to export it using AVRO serialization and I have them in xml/csv. My ...
0
votes
0answers
7 views

Connect to Kafka from K6 using api key or user/password

I am trying to connect my Kafka which is protected by api-key for K6 runs. The original documentation on K6 website doesn't point on how to connect to Kafka using API KEY. Here is the link for ...
0
votes
0answers
12 views

How can I use Kafka connect to import a db table with Long as topic key

I am trying to import a database table with Kafka Connect (Debezium and Postgres) into a Kafka topic and I need to have my topic key as a Long. I was unable to find a LongConvertor which led me to ...
0
votes
0answers
5 views

How to configure Kafka REST Proxy to minimize the number of acknowledges?

How to configure Kafka REST Proxy to minimize the number of acknowledges? I went through REST Proxy Configuration Options but I cannot find anything about this. Generally I am asking about this, as ...
0
votes
0answers
22 views

Kafka Streams: TimeoutException: Failed to update metadata after 60000 ms

I'm running a Kafka Streams application to consumer one topic with 20 partitions at traffic 15K records/sec. The application does a 1hr windowing and suppress the latest result per window. After ...
0
votes
2answers
38 views

Spring Boot Kafka Startup error “Connection to node -1 could not be established. Broker may not be available.”

I am trying to start Spring-Kafka with Spring Boot 2.1.7.RELEASE on localhost with Java 12. Getting the Error: "org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=inter]...
0
votes
0answers
16 views

What is the equivalent of Flink's tumbling/session window join in Kafka Streams?

According to documentation of Kafka Streams, we can apply KStream-to-KStream joins only by defining a lower and upper bound: KStream<String, String> joined = left.join(right, (leftValue, ...
0
votes
0answers
18 views

Failed SSL handshake in Kafka Producer

I'm trying to set up a Kafka broker with SSL enabled communication. I followed the guide as described in the Kafka documentation here under 7.2: https://kafka.apache.org/documentation/#security When ...
0
votes
0answers
17 views

Kafka appender in log4j2 is not setting “key” value to message

I was implementing log4j2 with kafka appender and successfully able to consume the log messages through remote consumer. But the "key" attritube in kafka appender of log4j2 is not working. Kafka ...
1
vote
0answers
29 views

How to check streaming data record already there in historical/persisted data in spark streaming?

For my PoC, I am using spark-sql 2.4.x with Kafka. I have a streaming company data coming from Kafka topic. Company data Which has "company_id" ,"created_date" ,"field1" , "field2" and etc as fields. ...
0
votes
1answer
40 views

How do I know which partition to consume in Kafka?

I have one topic with 3 partitions in Kafka broker 1 partitions have message and 2 partitions are empty, how do I know which partition to consume in one call? First I assign a TopicPartition with ...
0
votes
0answers
21 views

Kafka Streams state store changelog, how does max.message.bytes affect the internal changelog topic?

I have a Kafka Streams application running with Kafka library 0.11 (cluster running on Kafka 2) that uses a state store for an .aggregate() call on the end of the chain of transformations. Monitoring ...
0
votes
0answers
13 views

debezium error: “option ”include-unchanged-toast“ = ”0“ is unknown”

I have a question about debezium and I was stuck for more than two weeks. I use debezium to pull data from remote PostgreSQL every minute, and I always see the below error as output: "option "...
0
votes
0answers
10 views

Kafka connect, JdbcSinkConnector not consistently auto creating tables

I have been trying to get JdbcSinkConnector and JdbcSourceConnector working but seem to get some intermittent problems. Here is my workflow. I create a table in my DB and create a source connector, ...
0
votes
1answer
39 views

How can I change the leader between consumers?

If I have a topic and multiple consumers with the same groupId, only one consumer should receive the messages (the leader). I want to chose which consumer would be the leader and receives messages in ...
0
votes
0answers
13 views

How to reach kafka with a producer situated in minikube

So, i'm out of ideas. i would reach a kafka cluster in my localpc with a producer (written in python with kafka-python library) situated on minikube. the producer code is: byte_log = str.encode(f"...
0
votes
0answers
15 views

Why I can not connect a Siddhi app to secure Kafka?

I am creating and environment with Siddhi and Kafka. Without security they worked, and now I want to deploy it with secure Kafka. For that I followed the guide. Logstash (if you do not know what it is,...
0
votes
0answers
15 views

Error while connecting MongoDB as source to Kafka

I am trying to connect MongoDB as source to Kafka server but when I run Debezium Mongo source connector, I get error. I don't understand why timed out? [2019-08-22 13:28:58,194] INFO Cluster ...
0
votes
0answers
16 views

Kafka migrate offsets

I have a new Kafka cluster. Offset is used as a sequence number. How can I migrate offsets from old to new cluster that a sequence won't be destroyed? Also it is doesn't matter if sequence will be ...
0
votes
0answers
14 views

Kafka connect single message transform

I searched about the question through out my web and reading several articles and came here as a final resort. I have configured a kafka connect Mongodb sink and I want to transform the message by ...
0
votes
0answers
12 views

Handle Deserialization Error (Dead Letter Queue) in a kafka consumer

After some research i found few configuration i can use to handle this. default.deserialization.exception.handler - from the StreamsConfig errors.deadletterqueue.topic.name - from the SinkConnector ...
0
votes
0answers
17 views

Getting BSON converter issue while saving message from Kafka topic to MongoDB using Kafka Connect

I am using Confluent Kafka Community version. I am trying to save data into MongoDB from Kafka topic using Kafka Connect. I have setup a CentOS VM where I have configured Confluent Kafka and MongoDB. ...
-1
votes
0answers
26 views

how to create a docker for the kafka commands Im currently using [on hold]

I currently have the following command list to run kafka locally: # Launch zookeeper cd bin/zookeeper-server-start.sh config/zookeeper.properties # Initialise Kafka Cluster with four ...
0
votes
0answers
12 views

Connecting Schema-Registry Docker Container to AWS MSK

I am not able to connect schema registry to MSK. I am running Docker in a separate instance and have connections enabled to MSK services. Tried docker image from confluent docker.io/confluentinc/cp-...
0
votes
0answers
22 views

Deleting a kafka topic

What is the best way to delete a topic when topic deletion is disabled in server properties. i tried doing that using zookeeper CLI, while zookeeper get topic says topic does not exist, kakfa ...
0
votes
2answers
35 views

Apache Kafka Java consumer does not receive message for topic with replication factor more than one

I'm starting on Apache Kakfa with a simple Producer, Consumer app in Java. I'm using kafka-clients version 0.10.0.1 and running it on a Mac. I created a topic named replicated_topic_partitioned with ...
0
votes
1answer
17 views

Can't figure out setting for inter.broker.listener.name in Kafka with SSL

I am attempting to configure Kafka nodes with SSL (TLS) inter-nodes and between nodes and clients but run into configuration problems. Kafka version is 2.3.0. My relevant settings are: - ...
0
votes
0answers
7 views

Timed out waiting for connection while in state : CONNECTING

I have a 3 node cluster for Kafka and zookeeper. Both Kafka and zookeeper cluster use same 3 machines. When I start the Kafka service, it fails to start if all 3 zookeeper servers are not up. Why is ...
0
votes
0answers
16 views

Logs when running Confluent dockers

I am running Confluent Kafka docker cp-all-in-one. I want to insert some topic data to MS SQL server. I create new DB and table. Using Confluent Control Center I added JdbcSinkConnector. Connector ...
0
votes
0answers
12 views

Kafka / ZooKeeper ACL Auditing

I am looking for a way to keep an audit log of ACL changes in our Kafka / ZooKeeper setup. I am using the Confluent bundle. I already increased all LOG_LEVEL in the log4j.properties to DEBUG, but I ...
0
votes
0answers
33 views

kafka is failing to start because cluster/id is deleted from Zookeeper

Apache Kafka is failing to start. It shows in its logs "Failed to get cluster id from Zookeeper. This can happen if /cluster/id is deleted from Zookeeper." How can I check the "/cluster/id"? ...
0
votes
0answers
25 views

Pattern for getting data from kafka through web application [on hold]

I am wondering what the pattern looks like to get data from kafka on a web application. The use case is to get a price from a microservice. Do i produce a getPrice message and send to kafka, that ...