log4j ERROR found in kafka connecting to zookeeper

I am trying to connect kafka to zookeeper on three machines, one is my laptop and other two are virtual machines. When I attempted initiating kafka using

bin/kafka-server-start.sh config/server.properties &

However, it printed the following error:

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/local/kafka/bin/../logs/server.log (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
    at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
    at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
    at kafka.utils.Log4jControllerRegistration$.<init>(Logging.scala:25)
    at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala)
    at kafka.utils.Logging$class.$init$(Logging.scala:47)
    at kafka.Kafka$.<init>(Kafka.scala:30)
    at kafka.Kafka$.<clinit>(Kafka.scala)
    at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [kafkaAppender].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/local/kafka/bin/../logs/kafka-request.log (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
    at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
    at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
    at kafka.utils.Log4jControllerRegistration$.<init>(Logging.scala:25)
    at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala)
    at kafka.utils.Logging$class.$init$(Logging.scala:47)
    at kafka.Kafka$.<init>(Kafka.scala:30)
    at kafka.Kafka$.<clinit>(Kafka.scala)
    at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [requestAppender].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/local/kafka/bin/../logs/kafka-authorizer.log (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
    at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
    at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
    at kafka.utils.Log4jControllerRegistration$.<init>(Logging.scala:25)
    at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala)
    at kafka.utils.Logging$class.$init$(Logging.scala:47)
    at kafka.Kafka$.<init>(Kafka.scala:30)
    at kafka.Kafka$.<clinit>(Kafka.scala)
    at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [authorizerAppender].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/local/kafka/bin/../logs/controller.log (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
    at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
    at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
    at kafka.utils.Log4jControllerRegistration$.<init>(Logging.scala:25)
    at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala)
    at kafka.utils.Logging$class.$init$(Logging.scala:47)
    at kafka.Kafka$.<init>(Kafka.scala:30)
    at kafka.Kafka$.<clinit>(Kafka.scala)
    at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [controllerAppender].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/local/kafka/bin/../logs/log-cleaner.log (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
    at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
    at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
    at kafka.utils.Log4jControllerRegistration$.<init>(Logging.scala:25)
    at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala)
    at kafka.utils.Logging$class.$init$(Logging.scala:47)
    at kafka.Kafka$.<init>(Kafka.scala:30)
    at kafka.Kafka$.<clinit>(Kafka.scala)
    at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [cleanerAppender].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /usr/local/kafka/bin/../logs/state-change.log (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
    at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
    at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
    at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
    at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
    at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
    at kafka.utils.Log4jControllerRegistration$.<init>(Logging.scala:25)
    at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala)
    at kafka.utils.Logging$class.$init$(Logging.scala:47)
    at kafka.Kafka$.<init>(Kafka.scala:30)
    at kafka.Kafka$.<clinit>(Kafka.scala)
    at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [stateChangeAppender].
[2021-01-11 11:29:34,962] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-01-11 11:29:35,126] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-01-11 11:29:35,126] INFO starting (kafka.server.KafkaServer)
[2021-01-11 11:29:35,126] INFO Connecting to zookeeper on jyy-lenovo-legion:2181,pro1:2181,pro2:2181 (kafka.server.KafkaServer)
[2021-01-11 11:29:35,141] INFO [ZooKeeperClient Kafka server] Initializing a new session to jyy-lenovo-legion:2181,pro1:2181,pro2:2181. (kafka.zookeeper.ZooKeeperClient)
[2021-01-11 11:29:35,144] INFO Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.ZooKeeper)
[2021-01-11 11:29:35,144] INFO Client environment:host.name=jyy-lenovo-legion (org.apache.zookeeper.ZooKeeper)
[2021-01-11 11:29:35,144] INFO Client environment:java.version=1.8.0_271 (org.apache.zookeeper.ZooKeeper)
[2021-01-11 11:29:35,144] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2021-01-11 11:29:35,144] INFO Client environment:java.home=/usr/local/jdk1.8.0_271/jre (org.apache.zookeeper.ZooKeeper)

My zoo.cfg:

tickTime=2000
initLimit=5
syncLimit=5
dataDir=/usr/local/zookeeper/zkData
clientPort=2181
server.1=pro1:2888:3888
server.2=pro2:2888:3888
server.3=jyy-lenovo-legion:2888:3888

My kafka/config/server.properties:

broker.id=0
host.name=jyy-lenovo-legion
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/local/kafka/tmp/kafka-logs
num.partitions=1
zookeeper.connect=jyy-lenovo-legion:2181,pro1:2181,pro2:2181
port=9092
group.initial.rebalance.delay.ms=0

Temporarily, I didn't find any solutions specific or close to my problem. Besides, I am new to kafka, zookeepero and hadoop-related big data tools, so this problem really troubles me. Could anyone offer some help?

1 answer

  • answered 2021-01-11 10:22 Mickael Maison

    These exceptions are not related to ZooKeeper. They are thrown by log4j as it's not allowed to write to the specified files. These should not prevent Kafka from running but obviously you won't get log4j logs.

    When starting Kafka with bin/kafka-server-start.sh, the default log4j configuration file, log4j.properties, is used. This attempts to write logs to ../logs/, see https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L194-L197

    In your case, this path is /usr/local/kafka/bin/../logs and Kafka is not allowed to write there.

    You can change the default path by setting the LOG_DIR environment variable to a path where Kafka will be allowed to write logs, for example:

    export LOG_DIR=/var/log/kafka/
    bin/kafka-server-start.sh config/server.properties &