Redis HA with Sentinel on Docker

I'm struggling now for a week with the Redis HA on Docker. I am not quite convinced if my intentions will even works. The documentation is understandable, but there are many examples which do not correspond to the documentation.

Well, what I want to do is set up a Redis Cluster with 1 master, 2 replicas and 3 Sentinel. This is hosted on a 192.168.1.10. I want to acess the Cluster via an App coming from 192.168.1.11. The RedisCluster is working properly without the Sentinel. Replication works fine.

When I start the sentinel, I get the following log entries on all 3 Redis-Container: redis-0, redis-1 and redis-2

1:S 22 Dec 2020 18:43:38.349 * Connecting to MASTER 172.20.0.2:6379
1:S 22 Dec 2020 18:43:38.350 * MASTER <-> REPLICA sync started
1:S 22 Dec 2020 18:43:38.350 * Non blocking connect for SYNC fired the event.
1:S 22 Dec 2020 18:43:38.350 * Master replied to PING, replication can continue...
1:S 22 Dec 2020 18:43:38.350 * Trying a partial resynchronization (request eac3aa540e767589e9673ae0ed844d985ed2abb2:1856).
1:S 22 Dec 2020 18:43:38.350 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master

I tried to follow this tutorial but it didn't work. Same behavier as described. These are my Docker commands

# Redis (with custom redis.conf will not work the replication) so i keep it simple this way.
docker run --name redis-0 -d --network redis -p 6379:6379 redis redis-server
docker run --name redis-1 -d --network redis -p 6380:6379 redis redis-server --slaveof redis-0 6379
docker run --name redis-2 -d --network redis -p 6381:6379 redis redis-server --slaveof redis-0 6379
# Sentinel
docker run -d --name sentinel-0 --network redis -v ${PWD}/sentinel-0:/etc/redis/  redis  redis-sentinel /etc/redis/sentinel.conf
docker run -d --name sentinel-1 --network redis -v ${PWD}/sentinel-1:/etc/redis/  redis  redis-sentinel /etc/redis/sentinel.conf
docker run -d --name sentinel-2 --network redis -v ${PWD}/sentinel-2:/etc/redis/  redis  redis-sentinel /etc/redis/sentinel.conf

These is the sentinel.conf

port 5000
# sentinel monitor <master-group-name> <ip> <port> <quorum>
sentinel monitor mymaster 172.20.0.2 6379 2
sentinel down-after-milliseconds mymaster 1000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

The Sentinel-Container have write access to the sentinel.conf.

Those are my iptables instructions

# Redis
/usr/sbin/iptables -t nat -A PREROUTING -p tcp --dport 6379 -j DNAT --to-destination 172.20.0.2:6379
/usr/sbin/iptables -t nat -A PREROUTING -p tcp --dport 6380 -j DNAT --to-destination 172.20.0.3:6379
/usr/sbin/iptables -t nat -A PREROUTING -p tcp --dport 6381 -j DNAT --to-destination 172.20.0.4:6379

# Sentinel
/usr/sbin/iptables -t nat -A PREROUTING -p tcp --dport 26379 -j DNAT --to-destination 172.20.0.5:6379
/usr/sbin/iptables -t nat -A PREROUTING -p tcp --dport 26380 -j DNAT --to-destination 172.20.0.6:6379
/usr/sbin/iptables -t nat -A PREROUTING -p tcp --dport 26381 -j DNAT --to-destination 172.20.0.7:6379

I'm well aware of the Documentation:

Since Sentinels auto detect replicas using masters INFO output information, the detected replicas will not be reachable, and Sentinel will never be able to failover the master, since there are no good replicas from the point of view of the system, so there is currently no way to monitor with Sentinel a set of master and replica instances deployed with Docker, unless you instruct Docker to map the port 1:1.

For the first problem, in case you want to run a set of Sentinel instances using Docker with forwarded ports (or any other NAT setup where ports are remapped), you can use the following two Sentinel configuration directives in order to force Sentinel to announce a specific set of IP and port:

sentinel announce-ip sentinel announce-port Note that Docker has the ability to run in host networking mode (check the --net=host option for more information). This should create no issues since ports are not remapped in this setup.

I just dont know where to place the announce-IP and Port and what the Value of those has to be. Also notice that the --net=host will not work, because i have 3 container each on the same Host:port.

How can I run Sentinel in a Docker environment which serves me for the Redis HA?

Thanks for the help!

EDIT:

I did a Failover test and have following result (same result on Sentinel 0, 1 and 2)

# docker exec -it sentinel-0 redis-cli -p 5000
127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster
1) "172.20.0.2"
2) "6379"
# docker stop redis-0
redis-0
# docker exec -it sentinel-0 redis-cli -p 5000
127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster
1) "172.20.0.2"
2) "6379"

1 answer

  • answered 2020-12-23 03:40 Joe

    follow your Docker commands and sentinel.conf, it works to me.

    1:S 23 Dec 2020 03:14:59.370 * Connecting to MASTER redis-0:6379
    1:S 23 Dec 2020 03:14:59.371 * MASTER <-> REPLICA sync started
    1:S 23 Dec 2020 03:14:59.371 * Non blocking connect for SYNC fired the event.
    1:S 23 Dec 2020 03:14:59.371 * Master replied to PING, replication can continue...
    1:S 23 Dec 2020 03:14:59.372 * Trying a partial resynchronization (request 5c52aa10610b365f29fec2968e095c5b49eb6136:43).
    1:S 23 Dec 2020 03:14:59.373 * Full resync from master: 1f843162cf808a500a5d57392baf585f6e1679a3:0
    

    Maybe you can check redis-0 logs,does it accept replica' ask.