kafka over 7 days reset offset experience

2 min readAug 8, 2022

background:

We are using kakfa as our message queue system, some case we may stop over 7 day to consume message, and then the consumer can not get any message

the situation is like:

we search “kafka consumer group not auto consume over 7 days”…

we found kafka broker will reset consumer offset ,if consumer does not pull/commit any message to topic in 7 days

ok , let test offset.retention.minutes is it work or not

experiment 1 set offset.retention.minutes=1 → and check consumer offset

— 1 consumer group , 1 consumer

we use kafkajs to get consumer offset https://kafka.js.org/docs/admin

| it seems over 1 min , the offset still not remove

we try to search again, the kafka has another setting for check the expiration of offset

offsets.retention.check.interval.ms

ok, let set offsets.retention.check.interval.ms = 1000(check log every seconds)

result: success , offset will be “-1” after 1 min later

when we send new message after 1 min , the consumer still can consumer new message, why ???

btw, so far, we get:

— — —

— many consumers in 1 consumer group

think about our production env, we have many consumers in consumer group,

so in this time, we launch many consumers in 1 consumer group

still set

offset.retention.minutes=1

offsets.retention.check.interval.ms=1000

result:

in our test case , even offset = -1, produce new message , consumer also can receive message…Why will happen this situation in production?

our conclusion：

May be at that time , consumers doing rebalance ( so the offset will keep -1)

set greater offset.retention.minutes (ex: 1 months) for low usage topic
if you need keep log same time as offset , you can set log.retention.hours greater also to keep kakfa log if need