kafka consumer cannot connect

See also questions close to this topic

• Cannot access webapp deployed in Docker container

  I'm currently getting into Docker (or trying to), so I set up a little webapp. I used Eclipse to create a maven-Project, built a .war-file and deployed it (to check if it works) locally on a Glassfish server. This worked, and I could access the webapp via

  localhost:8080/myApp
  

  Now I created a Dockerfile, built a docker image from that and ran the whole thing via

  docker run -d -p 8080:8080 myApp
  

  This also works, and

  docker ps -a
  

  shows that a container is running. However, when I try to access localhost:8080/myApp now, I get

  ERR_CONNECTION_REFUSED

  I tried this with Chrome, Firefox and Internet Explorer.

  docker exec -it <myContainer> bash
  

  gives me a shell inside the container (as expected). When I do

  curl -v localhost:8080/myApp
  

  here, I get the output I expect (the page from myApp).

  So for me, this means that the container is running, the glassfish server is running inside the container, and myApp has been deployed "properly" to said server, yet still I cannot access myApp from outside the container.

  docker ps -a
  

  yields this, btw:

  CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                                                       NAMES
  9514b1ab6dd0        myApp              "sh /start.sh"      9 minutes ago       Up 9 minutes        4848/tcp, 8181/tcp, 0.0.0.0:8080->8080/tcp   admiring_cori
  

  The Dockerfile does of course contain the line

  EXPOSE 8080
  

  Any ideas what might be wrong here?
• Visual Studio Code PHP Debug does not stop on breakpoints for Docker project

  I cannot get the VS Code extension "Php debug" to stop on any breakpoints. I am running the php project (that I want to debug) from Docker on my Ubuntu laptop. Any advice greatly appreciated.

  My set up:

  • PHP version on Docker: 7.1
  • XDebug version on Docker: 2.6.1
  • PHP Debug version in VS Code (on my laptop): 1.12.6

  My VS Code launch.json file is:

  {
      "version": "0.2.0",
      "configurations": [
          {
              "name": "Listen for XDebug",
              "type": "php",
              "request": "launch",
              "port": 9000,
              "log": true,
              "pathMappings": {
                  "/var/www/html": "/home/chris/my-test-debugging-project"
              }
          },
          {
              "name": "Launch currently open script",
              "type": "php",
              "request": "launch",
              "program": "${file}",
              "cwd": "${fileDirname}",
              "port": 9000
          }
      ]
  }
  

  my /usr/local/etc/php/conf.d/xdebug.ini config is:

  zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20160303/xdebug.so
  xdebug.remote_enable=1
  xdebug.remote_autostart=1
  xdebug.remote_host=172.17.0.2
  xdebug.remote_log=/var/www/html/xdebug.log
  xdebug.remote_connect_back=0
  xdebug.remote_port=9000
  

  XDebug logfile (from setting xdebug.remote_log in php.ini):

      Log opened at 2018-10-14 05:47:16
      I: Connecting to configured address/port: 172.17.0.2:9000.
      W: Creating socket for '172.17.0.2:9000', poll success, but error: Operation now in progress (29).
      E: Could not connect to client. :-(
      Log closed at 2018-10-
  
  14 05:47:16
  

  The PHP Debug log output (from setting "log": true in launch.json):

  <- launchResponse
  Response {
  seq: 0,
  type: 'response',
  request_seq: 2,
  command: 'launch',
  success: true }
  
• Cannot pull Docker image out of OpenShift cluster

  I would like to pull a Docker image that was built inside an OpenShift Container Platform 3.9 cluster out of that cluster. To this end I try the following:

  username=$(oc whoami)
  api_token=$(oc whoami -t)
  docker login -u $username -p $api_token my-cluster:443
  image=$(oc get is/my-is -o jsonpath='{.status.tags[0].items[0].dockerImageReference}')
  docker pull $image
  

  Now docker login works, but docker image produces the error message

  lookup docker-registry.default.svc on 1.2.3.4: no such host

  where 1.2.3.4 is a placeholder for my local nameserver according to /etc/resolv.conf and $image is of the form docker-registry.default.svc:5000/registry/my-is@sha256:my-id. Am I doing something wrong or could it be that the cluster administrator must first expose the registry (but should it not be exposed by default)? If I try oc get svc -n default as suggested here I get this error message:

  User "my-user" cannot list services in project "default"

  So what steps are needed (preferably without intervention by the cluster's administrator) for me successfully pulling out that image? Would the situation change if the pull occurred in a container also executing inside the OpenShift cluster?
• KAFKA: Failed to update metadata after 60000 ms

  I am new to KAFKA and I know that this question has been asked multiple times on stack overflow but none of the solutions worked for me so here I am trying my luck with asking the same question again. I have downloaded and installed KFKA on Centos7 VM. The VM is on my laptop. When I run the KAFKA producer and consumer from the command line, it works fine. Next step, I wanted to create a Java Producer but it always timeout with the following exception.

      Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
      at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1186)
      at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:880)
      at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803)
      at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:690)
      at com.soft.teradata.KafkaProducerExample.runProducer(KafkaProducerExample.java:40)
      at com.soft.teradata.KafkaProducerExample.main(KafkaProducerExample.java:55)
  Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
  

  The Java code for the producer is:

  package com.soft;
  import java.util.Properties;
  import java.util.concurrent.Future;
  import org.apache.kafka.clients.producer.Producer;
  import org.apache.kafka.clients.producer.KafkaProducer;
  import org.apache.kafka.clients.producer.ProducerRecord;
  import org.apache.kafka.clients.producer.RecordMetadata;
  public class SimpleProducer {
      public static void main(String[] args) throws Exception {
          try {
              String topicName = "Hello-Kafka";
              Properties props = new Properties();
              props.put("bootstrap.servers", "192.168.xxx.xxx:9092");
              props.put("acks", "all");
              props.put("retries", 1);
              props.put("batch.size", 16384);
              props.put("linger.ms", 1);
              props.put("buffer.memory", 33554432);
              props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
              props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
              Producer<String, String> producer = new KafkaProducer<String, String>(props);
              Future<RecordMetadata> f=producer.send(new ProducerRecord<String, String>(topicName, "Eclipse"));
              System.out.println("Message sent successfully");
              producer.close();
          } catch (Exception e) {
              System.out.println(e.getMessage());
          } 
              System.out.println("Successful");
  
      }
  } 
  

  For bootstrap.server, i have even tried the following:

  props.put("bootstrap.servers", "PLAINTEXT://192.168.xxx.xxx:9092");

  Please note that I am executing the java code from Eclipse on my laptop and KAFKA is installed on a CENTOS7 VM on my laptop. 192.168.xxx.xxx is the IP address of the CENTOS7 VM. I have noticed that 192.168.xxx.xxx:9092 (telnet 192.168.xxx.xxx 9092) is inaccessible from my laptop. I added the port to firewall but still no success.

  firewall-cmd --zone=public --add-port=9092/tcp --permanent
          firewall-cmd --reload
          firewall-cmd --list-ports
  

  The version of KAFKA is 2.12-2.0.0 and I have added the following jars to my Eclipse Classpath:

  • kafka-clients-2.0.0.jar
  • lz4-java-1.4.1.jar
  • slf4j-api-1.7.25.jar
  • snappy-java-1.1.7.1.jar

  Thanks a lot for the help in advance :)

  Regadrs, DIRSHAH.
• Kafka behind Traefik on Kubernets

  I am trying to configure a Kafka cluster behind Traefik but my producers and client (that are outside kubernetes) don't connect to the bootstrap-servers. They keep saying:

  "no resolvable boostrap servers in the given url"

  Actually here is the Traefik ingress:

  {
      "apiVersion": "extensions/v1beta1",
      "kind": "Ingress",
      "metadata": {
          "name": "nppl-ingress",
          "annotations": {
              "kubernetes.io/ingress.class": "traefik",
              "traefik.frontend.rule.type": "PathPrefixStrip"
          }
      },
      "spec": {
          "rules": [
              {
                  "host": "" ,
                  "http": {
                      "paths": [
                          {
                              "path": "/zuul-gateway",
                              "backend": {
                                  "serviceName": "zuul-gateway",
                                  "servicePort": "zuul-port"
                              }
                          },                      
                          {
                              "path": "/kafka",
                              "backend": {
                                  "serviceName": "kafka-broker",
                                  "servicePort": "kafka-port"
                              }
  
  [..]
      }
  

  What I give to the kafka consumers/producers is the public IP of Traefik. Here is the flow: [Kafka producers/consumers] -> Traefik(exposed as Load Balancer) -> [Kafka-Cluster]

  Is there any solution? Otherwise I was thinking to add a kafka-rest proxy (https://docs.confluent.io/current/kafka-rest/docs/index.html) between Traefik and the kafka brokers but I think isn't the ideal solution.
• Having trouble with Ubuntu .profile and .bashrc

  I am new to linux. I am currently going through a setup tutorial for Kafka online. It says to add the path of my kafka bin directory as follows to my .profile file which I did as below:

  # ~/.profile: executed by the command interpreter for login shells.
  # This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
  # exists.
  # see /usr/share/doc/bash/examples/startup-files for examples.
  # the files are located in the bash-doc package.
  # the default umask is set in /etc/profile; for setting the umask
  # for ssh logins, install and configure the libpam-umask package.
  #umask 022
  
  # if running bash
  if [ -n "$BASH_VERSION" ]; then
  # include .bashrc if it exists
  if [ -f "$HOME/.bashrc" ]; then
  . "$HOME/.bashrc"
  fi
  fi
  
  # set PATH so it includes user's private bin directories
  DOCKER="/usr/local/bin/docker-compose"
  PATH="$HOME/bin:$HOME/.local/bin:$PATH:$DOCKER"
  # Ubuntu make installation of Ubuntu Make binary symlink
  PATH=/home/username/.local/share/umake/bin:$PATH
  cat ~/.ssh/config.d/* > ~/.ssh/config
  PATH="$PATH:/home/username/softwares/kafka/kafka_2.11-1.0.0/bin"
  

  After that, i did an echo on $PATH, I could see the kafka path added to the PATH. Post which, i typed kafka and then tab, then i could see kafka related optional commands.

  The tutorial then tells to edit .bashrc file. It was already there. It was quite a big file. I added at the end of the file the below line(as per the tutorial):

  . ~/.profile
  

  After that, i opened another terminal as per the tutorial to see that I am still getting all the kafka related options after kafka and tab. I saw that on the terminal, i was not getting my username and it was a blank terminal window. I then edited the .bashrc to remove the line I added and then tried to open a new terminal, and I could see my username on the terminal. Then i closed all the terminals. Opened a fresh one and typed kafka and tab, and I didnt get any options as earlier. I then opened the .profile file and i could see the kafka path still added. Then, I tried to echo $PATH, and this time, the kafka path was not there.

  I am really confused what is happening here. Could you please explain a bit and let me know how to load the .profile every time i open a terminal and that why do I not see the kafka path anymore , when i do an echo on PATH.