Confused about commit and fetch consumer offsets in Kafka 1.x

I used the following Kafka consumer example code:

KafkaConsumer kconsumer = new KafkaConsumer(props);
ConsumerRecords<String, byte[]> records = kconsumer.poll(timeoutInMS);

I have two questions please:

(1) Where the offset commits were commited, to Kafka or Zookeer?(I knew it was committed to zookeeper for the old kafka version, but in new kafka version, it still go to ZK, but I even didn't indicate the zookeeper IP in the configuration file).

(2) How can I retrieve the committed offsets in case I need to re-consumer the data?

1 answer

  • answered 2018-01-11 22:14 ppatierno

    With newer Kafka version the offsets aren't saved in Zookeeper but in the __commit_offsets topic which is distributed across the Kafka broker of course. The Zookeeper node has no entries related to committed offesets and you don't need connection to it. Regarding retrieving latest committed offsets it's something that happens automatically when a partition is assigned to a consumer and such consumer gets (through the underline Kafka protocol) the latest committed offsets and will start to receive from there. You can use seekXXXX methods if you want to seek at beginning, at the end or at a specific offset in the partition.