Can't connect to dockerized kafka (used wurstmeister/kafka-docker) from outside the server
I'm trying to connect kafka, which is running on docker conatiner (on EC2 server).
I cloned wurstmeister/kafka-docker, and edited docker-compose-single-broker.yml
file like below.
version: "2"
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "8092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 111.11.11.111 # (my EC2 server ip address)
KAFKA_CREATE_TOPICS: "test-topic:1:1" # topic_name:partition:replica
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
I changed port 9092:9092
to 8092:9092
, because my EC2 server doesn't allow port over 9000.
Ip 111.11.11.111
is an example.
With this docker-compose, I successfully started kafka and zookeeper container.
Then, I used console-producer and console-consumer on same EC2 server to test kafka connection like below.$ bin/kafka-console-producer.sh --broker-list 111.11.11.111:8092 --topic test-topic
$ bin/kafka-console-consumer.sh --bootstrap-server 111.11.11.111:8092 --topic test-topic --from-beginning
It worked well on my EC2 server, but I couldn't do the same on my local shell. It seems like it doesn't allow access from outside the server. How can I change my docker-compose file to make it possible to access from other server?
do you know?
how many words do you know
See also questions close to this topic
-
best way to add a spring boot web application with JSPs to a docker container?
so i have a spring boot web application, it uses JSPs, and im supposed to put it in a container .
my question is what is the best way ? ive tried to copy the project and then run it there using a wmvn like so : dockerfile:FROM openjdk:8-jdk-alpine ADD . /input-webapp WORKDIR /input-webapp EXPOSE 8080:8080 ENTRYPOINT ./mvnw spring-boot:run
which works, but it takes a long time getting the dependencies and feels messy .
and ive tried to package it into a jar, and then copy the jar only, and run it : dockerfile:
FROM openjdk:8-jdk-alpine ADD target/input-webapp-0.0.1-SNAPSHOT.jar input-webapp-0.0.1-SNAPSHOT.jar EXPOSE 8080:8080 ENTRYPOINT ["java","-jar","input-webapp-0.0.1-SNAPSHOT.jar"]
but this way it cant see the JSPs, or at least i think this is the problem, as i get a 404.
so is there a better way ? can i copy the jsps plus the jar to make it work? thanks
-
build spring boot (mvnw) with docker can not use cache
Spring Boot Docker Experimental Features Docker 18.06 comes with some “experimental” features, including a way to cache build dependencies. To switch them on, you need a flag in the daemon (dockerd) and an environment variable when you run the client. With the experimental features, you get different output on the console, but you can see that a Maven build now only takes a few seconds instead of minutes, provided the cache is warm.
my dockerfile can not use cache.
dockerfile
# syntax=docker/dockerfile:experimental FROM openjdk:8-jdk-alpine as build WORKDIR /workspace/app COPY mvnw . COPY .mvn .mvn COPY pom.xml . COPY src src RUN --mount=type=cache,target=/root/.m2 ./mvnw install -DskipTests -s .mvn/wrapper/settings.xml RUN mkdir -p target/extracted && java -Djarmode=layertools -jar target/*.jar extract --destination target/extracted FROM openjdk:8-jre-alpine ENV TZ Asia/Shanghai RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone RUN addgroup -S spring && adduser -S spring -G spring USER spring:spring ARG EXTRACTED=/workspace/app/target/extracted ARG JAVA_OPTS="-Xmx100m -Xms100m" COPY --from=build ${EXTRACTED}/dependencies/ ./ COPY --from=build ${EXTRACTED}/spring-boot-loader/ ./ COPY --from=build ${EXTRACTED}/snapshot-dependencies/ ./ COPY --from=build ${EXTRACTED}/application/ ./ ENTRYPOINT ["sh", "-c","java ${JAVA_OPTS} org.springframework.boot.loader.JarLauncher"]
run shell
DOCKER_BUILDKIT=1 docker build -t org/spring-boot .
every time use many minutes
-
PyQT5 doesn't work on docker ImportError: libsmime3.so: cannot open shared object file: No such file or directory
I have a Dockerfile with PyQT installed like below
FROM ubuntu:20.04 ENV DEBIAN_FRONTEND=noninteractive RUN adduser --quiet --disabled-password qtuser && usermod -a -G audio qtuser RUN apt-get update -y \ && apt-get install alsa -y \ && apt-get install -y python3-pyqt5 \ && apt-get install python3-pip -y && \ pip3 install pyqtwebengine WORKDIR /htmltopdf
I built my image like this
docker build -t html-to-pdf .
Then I ran my image like this
docker run --rm -v "$(pwd)":/htmltopdf -u qtuser -it html-to-pdf python3 htmlToPdfnew.py --url https://www.w3schools.com/howto/howto_css_register_form.asp
But I'm getting below error
Traceback (most recent call last): File "htmlToPdfnew.py", line 2, in <module> from PyQt5 import QtWidgets, QtWebEngineWidgets ImportError: libsmime3.so: cannot open shared object file: No such file or directory
I do NOT get that error in my PC.
below is my python code
import sys from PyQt5 import QtWidgets, QtWebEngineWidgets from PyQt5.QtCore import QUrl, QTimer from PyQt5.QtGui import QPageLayout, QPageSize from PyQt5.QtWidgets import QApplication import argparse def main(): url = '' parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument("--url", help="Type url") args = parser.parse_args() config = vars(args) url = config['url'] app = QtWidgets.QApplication(sys.argv) loader = QtWebEngineWidgets.QWebEngineView() loader.setZoomFactor(1) layout = QPageLayout() layout.setPageSize(QPageSize(QPageSize.A4Extra)) layout.setOrientation(QPageLayout.Portrait) loader.load(QUrl(url)) loader.page().pdfPrintingFinished.connect(lambda *args: QApplication.exit()) def emit_pdf(finished): QTimer.singleShot(2000, lambda: loader.page().printToPdf("test.pdf", pageLayout=layout)) loader.loadFinished.connect(emit_pdf) sys.exit(app.exec_()) if __name__ == '__main__': main()
so how do I resolve below error ?
Traceback (most recent call last): File "htmlToPdfnew.py", line 2, in <module> from PyQt5 import QtWidgets, QtWebEngineWidgets ImportError: libsmime3.so: cannot open shared object file: No such file or directory
-
kafka - How to read and process messages in a fault tolerant way?
I am new to Kafka, and I am trying to understand how a Kafka Consumer can consume and process a message from a Kafka topic without losing messages if the consumer fails in between.
For Example:
There is a kafka topic called
cart_checkout
which just holds an event when the user successfully checks out their cart. All it will do is get an event and send an email to the user that the their items are checked out.This is the happy path:
- Consumer gets an event from the topic.
- Read offset is committed to Kafka.
- Consumer calls app specific function to send email.
- Repeat.
What happens if the app fails during
Step 3
?If the consumer starts up then it will miss an event, am I right ? (Since the read message was committed)
The consumer can rewind but how will it know to rewind ?
RabbitMQ solution:
It seems easier to solve in RabbitMQ since we can control the ACKs to a message. But in Kafka, if we commit the read-offset we lose the message when the app restarts, if we don't commit the read-offset then the same message is sent to multiple consumers.
What is the best possible way to deal with this problem ?
-
Kafka unable to connect to Zookeeper in docker-compose.yml
I am using the Zookeeper and Kafka images provided by the Debezium project in a
docker-compose.yml
file. When I bring up the containers, I see a lot of error logs in both the Zookeeper and Kafka containers.Zookeeper log:
zookeeper_1 | 2022-05-06 17:28:50,926 - WARN [NIOWorkerThread-10:NIOServerCnxn@370] - Exception causing close of session 0x0: Unreasonable length = 308375649
Kafka log:
2022-05-06 17:28:50,975 - ERROR [main:TopicCommand$@44] - org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
I do not know what is causing these errors or how to resolve them.
My
docker-compose.yml
file and links to the Docker images are below.docker-compose.yml
version: '3.3' services: zookeeper: hostname: zookeeper image: debezium/zookeeper:1.8 ports: - 2181:2181 - 2888:2888 - 3888:3888 kafka: hostname: kafka image: debezium/kafka:1.8 ports: - 9092:9092 environment: - ZOOKEEPER_CONNECT=zookeeper:2181 - CREATE_TOPICS=kafka_connect_config:1:1:compact,kafka_connect_offsets:1:1:compact,kafka_connect_status:1:1:compact depends_on: - zookeeper
Debezium Kafka
Debezium ZookeeperIf it helps, I am on an Ubuntu machine.
-
Error while Upgrading Confluent Connector for S3 Sink
I am currently working on a project where we have a already setup system with Confluent sink connectors for s3 and Elasticsearch. Current Version which we are using are :-
Kafka Version :- 2.4.0 Confluent Connector for S3 :- 5.3.1 Confluent Connector for Elasticsearch :- 5.3.1
The whole setup helps us to get data from kafka topic to sink(s3 and elastic). We are in process of upgrading the whole setup, though the goal is to get to the latest version but because of some technical issues we have to gradually upgrade the version. Now we are trying to upgrade to
Kafka :- 2.5.0 Confluent S3 sink :- 5.5.1 Confluent Elastic sink connector:- 5.5.1
This whole setup is deployed in Kubernetes, while deploying the upgraded image i am seeing below error
java.lang.abstractmethoderror: receiver class com.sun.jersey.api.uri.uribuilderimpl does not define or inherit an implementation of the resolved method 'abstract javax.ws.rs.core.uribuilder uri(java.lang.string)' of abstract class javax.ws.rs.core.uribuilder.
This is causing the connector to fail and exit. Am I missing out something here ??? Any one in community has encountered this issue ?? Any help to resolve this issue is greatly appreciated.
-
how do i store maven dependencies inside docker image using a Dockerfile
so im making a spring boot application that im supposed to put in and run from a docker container, and i would like to build the whole image using a docker file.
im using this dockerFile:FROM openjdk:8-jdk-alpine ADD . /analytics-service WORKDIR /analytics-service ENTRYPOINT ./mvnw spring-boot:run
when i create the image it just copies the files, and only after i run it, it starts downloading all the maven dependencies. which takes a while, given that i will be running a few containers. so how do i do it ? i want it to get all the dependencies when the image is created, so when i create a container it doesnt start downloading.
-
Uncaught Error: Class 'Service\UsersService' not found
I'm trying to run my php-based application, which connects to a mysql DB, launched through docker-compose.
i get this message when i connect to localhost:
Fatal error: Uncaught Error: Class 'Service\UsersService' not found in /var/www/html/dic/users.php:3 Stack trace: #0 /var/www/html/web/index.php(3): require() #1 {main} thrown in /var/www/html/dic/users.php on line 3
user.php
<?php return new Service\UsersService( require "../config/db-connection.php" );
UsersService.php
<?php namespace Service; use Entity\User; use PDO; class UsersService { protected $db; public function __construct(PDO $db) { $this->db = $db; } public function getLastJoined(): array { return $this->db->query( "SELECT id, joined, name FROM user ORDER BY joined DESC LIMIT 10" )->fetchAll(PDO::FETCH_CLASS, User::class); } public function getById(string $id): ?User { $user = $this->db->query("SELECT id, joined, name FROM user WHERE id = " . $this->db->quote($id))->fetchObject(User::class); return ($user === false) ? null: $user; } }
index.php
<?php $lastJoinedUsers = (require "../dic/users.php")->getLastJoined(); switch (require "../dic/negotiated_format.php") { case "text/html": (new Views\Layout( "Twitter - Newcomers", new Views\Users\Listing($lastJoinedUsers), true ))(); exit; case "application/json": header("Content-Type: application/json"); echo json_encode($lastJoinedUsers); exit; } http_response_code(406);
I appreciate any help!!
-
Docker compose specified port not mapping to connect to the health check endpoint
I am a newbie to docker. This is the layout I am working with. I have 2 projects in the docker compose file. A dotnet6 web api and a mvc web app. In the web api startup.cs I have the following code to get health statuses using AspNetCore.HealthChecks.UI, If I browse to the endpoint I get a json object of health check values (this part works)
app.UseEndpoints(endpoints => { endpoints.MapControllers(); endpoints.MapHealthChecks("/hc", new HealthCheckOptions { Predicate = _ => true, ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse }); });
I have a WebStatus .Net core MVC app whose job is to display health stats using the healthcheck UI. The appsetting for this is:
{ "HealthChecks-UI": { "HealthChecks": [ { "Name": "TestApp API Health Check", "Uri": "https://localhost:7100/hc" }, { "Name": "TestApp API Health Check (Local)", "Uri": "https://localhost:7141/hc" } ], "Webhooks": [ { "Name": "", "Uri": "", "Payload": "", "RestoredPayload": "" } ], "EvaluationTimeOnSeconds": 10, "MinimumSecondsBetweenFailureNotifications": 60 }, "Logging": { "LogLevel": { "Default": "Information", "Microsoft": "Warning", "Microsoft.Hosting.Lifetime": "Information" } }, "AllowedHosts": "*" }
And the docker compose file is as shown below
version: '3.4' services: testapp.api: environment: - ASPNETCORE_ENVIRONMENT=Development - ASPNETCORE_URLS=https://+:443;http://+:80 ports: - "8005:80" - "7100:443" volumes: - ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro - ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro webstatus: environment: - ASPNETCORE_ENVIRONMENT=Development - ASPNETCORE_URLS=https://+:443;http://+:80 - HealthChecksUI__HealthChecks__0__Name=Test App API Health Check - HealthChecksUI__HealthChecks__0__Uri=https://testapp.api/hc ports: - "8006:80" - "7101:443" depends_on: - testapp.api volumes: - ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro - ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
If I start both the projects without docker, everything works, the WebStatus app shows the health status of the api as expected! However it doesn't work when I use docker compose to run this. The url I use to get to the webstatus app is https://localhost:7101/healthchecks-ui#/healthchecks