kafka consumer cannot connect

See also questions close to this topic

• How to do profiling inside a docker container

  I am running some application inside a Docker container and want to do some profiling to see statistics like what functions in the application consumes most of the CPU and etc. The Linux Perf tool looks like a good fit, but it seems Perf cannot be used in Docker containers since it does not have the permission to access the hardware counters. Any suggestion on how to use Perf in the container, or what other tools can be used? Thanks!
• How to configure dynamically provisioned Docker agents

  I installed Docker on Windows 10, and I pulled jenkins docker from Docker Hub. Next, I started my jenkins docker,

  docker run --rm -u root -p 8080:8080 -v my_host_path:/var/jenkins_home jenkins
  

  Next, I used Manage Jenkins and Manage Plugins to install Docker plugin, then went to the Configure page and tried to add Docker Cloud.

  After I entered Docker Host URI : tcp://127.0.0.1:2375, I wanted to "Test Connection", but unfortunately got failed.

  I tried to follow the instruction as below link: How to find "Docker Host URI" to be used in Jenkins "Docker Plugin"?

  But I can't not find any docker setting file under /etc/default/* in my jenkins container, so I can't set the DOCKER_OPTS argument.

  Could someone give me any advise? Thank you !
• Creating a docker image to scaffold django projects

  I'm build a docker image that can scaffold out a docker project using this Dockerfile:

  FROM python:3
  
  WORKDIR /usr/src/app
  
  CMD ["pip","install Django"]
  
  CMD ["django-admin", "startproject hello_world_django"]
  

  I build the image using:

  docker build django-scaffold .
  

  and run it using

  docker run django-scaffold
  

  Error message:

  docker: Error response from daemon: OCI runtime create failed: 
  container_linux.go:348: starting container process caused "exec: 
  \"django-admin\": executable file not found in $PATH": unknown.
  

  When I run this docker image, I am informed that the command django-admin is not available.

  How can I add django-admin to the path within the container?
• how to set kafka connect auto.offset.reset with rest api

  I have create a sink kafka connect that convert data to other storage; I want to set auto.offset.reset as latest when new connector is created with kafka connect rest api; I have set consumer.auto.offset.reset: latest in configs;

  json { "name": "test_v14", "config": { "name": "test_v14", "consumer.auto.offset.reset": "latest", "connector.class": "...", ... } }

  But when task started, kafka consumer still poll records from earliest; So is any other ways to set auto.offset.reset as latest;
• I want to use the Kafka Streams tumbling window in a different way

  !!Test environment

  • A producer who sends a message with the key 'A' once a second.

  • Kafka Streams (Java Code) tumbling window every 10 seconds.

  • This code saves the number of messages in the window in Ktable.

  • I'd like to simply give a list of applications to print out the messages I receive.

  !! Current Output (timestemp, key, Number of messages per tumbling window)

  (1544079700000L, 'A', 10)

  (1544079710000L, 'A', 10)

  (1544079720000L, 'A', 10)

  ...

  !! the output I want(timestemp, key, Value List)

  (1544079700000L, ‘A', ,=, [a,b,c,d,e,f,g,h,i,j])

  (1544079710000L, ‘A', ,=, [k,l,m,n,o,p,u,x,y,z])

  (1544079710000L, ‘A', ,=, [a,c,w,d,r,f,a,s,w,d])

  ...

  !! code

  package io.github.timothyrenner.kstreamex.tumblingwindow;
  import org.apache.kafka.clients.producer.KafkaProducer;
  import org.apache.kafka.clients.producer.ProducerRecord;
  import org.apache.kafka.common.serialization.Serdes;
  import org.apache.kafka.streams.KafkaStreams;
  import org.apache.kafka.streams.KeyValue;
  import org.apache.kafka.streams.StreamsConfig;
  import org.apache.kafka.streams.kstream.KStream;
  import org.apache.kafka.streams.kstream.KTable;
  import org.apache.kafka.streams.kstream.Windowed;
  import org.apache.kafka.streams.kstream.TimeWindows;
  import org.apache.kafka.streams.kstream.KStreamBuilder;
  import java.util.Properties;
  import java.util.Random;
  /** Demonstrates tumbling windows.
   *
   * @author Timothy Renner
   */
  public class TumblingWindowKafkaStream {
      /** Runs the streams program, writing to the "long-counts-all" topic.
       *
       * @param args Not used.
       */
      public static void main(String[] args) throws Exception {
          Properties config = new Properties();
          config.put(StreamsConfig.APPLICATION_ID_CONFIG,
              "tumbling-window-kafka-streams");
          config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
              "localhost:9090,localhost:9091,localhost:9092");
          config.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG,
              "localhost:2181");
          config.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG,
              Serdes.ByteArray().getClass().getName());
          config.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG,
              Serdes.Long().getClass().getName());
          KStreamBuilder builder = new KStreamBuilder();
          KStream<byte[], byte[]> longs = builder.stream(
              Serdes.ByteArray(), Serdes.ByteArray(), "t2");
          // The tumbling windows will clear every ten seconds.
          KTable<Windowed<byte[]>, Long> longCounts =
              longs.groupByKey()
                   .count(TimeWindows.of(10000L)
                                     .until(10000L),
                          "t2-counts");
          // Write to topics.
          longCounts.toStream((k,v) -> k.key())
                    .to(Serdes.ByteArray(),
                        Serdes.Long(),
                        "t2-counts-all");
          KafkaStreams streams = new KafkaStreams(builder, config);
          streams.start();
          } // Close main.
  } // Close TumblingWindowKafkaStream.
  
• Apache Kafka Effective Broadcast

  In Apache Kafka concerning the first image below, if each partition contained the same message with different key, that is the single same message in each partition, then does the first image represents more efficient broadcast than the second one?

  With efficient broadcast I mean which one has a new delay sublinear to the new number of consumers, when we have more consumers. Or both of them have linear or upper-linear delay?

  I would like also to ask if it is possible to have several partitions with the same single message with different key since we know from the documentation of Kafka that each partition consists from different data.

  Is there any other more efficient scheme of broadcast using Apache Kafka, or any Apache Kafka API that implements a sublinear delay broadcast mechanism?