Setup Redis Sentinel with TLS and password authentication enabled with Docker Compose

2024-10-14 2024-10-14 1681 words 8 minutes

Contents

Introduction

If a Redis Sentinel deployment is used in production it can be sometimes handy to have such a deployment also locally available for testing. Esp. with TLS enabled for a secure connection and password authentication it sometimes takes a bit to get the driver configuration right. So here is a possible solution to configure such a setup.

Basic setup

Of course you need docker and docker-compose standalone or compose plugin as a Docker plugin installed.

/etc/hosts

Communication between the Redis components uses host names. That means some entries in /etc/hosts are needed. Make sure that you’ve the following /etc/hosts entries:

plain

127.0.0.1 redis00 redis01 redis02 sentinel00 sentinel01 sentinel02

Directories

Lets create a directory that contains all files and directories needed:

bash

mkdir rs
cd rs

Next a few more directories are needed:

bash

mkdir bin
mkdir certs
mkdir redis{00,01,02}
mkdir sentinel{00,01,02}

The final directory structure will look like this (already includes files created further down below):

bash

.
├── bin
│   └── sentinel_entrypoint.sh
├── certs
│   ├── ca.crt
│   ├── ca.key
│   ├── ca.srl
│   ├── gen_certs.sh
│   ├── redis.crt
│   ├── redis.csr
│   ├── redis.key
│   └── req.conf
├── docker-compose.yml
├── redis00
│   └── redis.conf
├── redis01
│   └── redis.conf
├── redis02
│   └── redis.conf
├── sentinel00
│   └── sentinel.conf.tmpl
├── sentinel01
│   └── sentinel.conf.tmpl
├── sentinel02
│   └── sentinel.conf.tmpl
├── start.sh
└── stop.sh

Redis configuration

The redisXX (XX is a placeholder for 00,01 and 02) directories will contain the configuration file for every Redis process. Here is an example for redis00/redis.conf:

plain

dir /tmp

requirepass dapassword

port 0

# For redis01 set "6479" as value
# For redis02 set "6579" as value
tls-port 6379
tls-cert-file /certs/redis.crt
tls-key-file /certs/redis.key
tls-ca-cert-file /certs/ca.crt
tls-auth-clients no
tls-replication yes

# For redis01 set "redis01" as value
# For redis02 set "redis02" as value
replica-announce-ip redis00

# For redis01 and redis02 only:
#slaveof redis00 6379
#masterauth dapassword

Create redis01/redis.conf and redis02/redis.conf too. It’s basically the same content as redis00/redis.conf. Just change replica-announce-ip redisXX accordingly while XX is the number of the Redis process. Also adjust tls-port. For redis01 it’s 6479 and for redis02 it’s 6579. And finally for redis01/redis.conf and redis02/redis.conf you need two addtional lines. For these two configuration files (not for redis00/redis.conf!) uncomment the last two lines. As redis01 and redis02 should become replicas of redis00 these two lines are the same for redis01 and redis02.

Sentinel configuration

For Sentinel the directory structure looks similar. Create a configuration file called sentinel.conf.tmpl for sentinel00. Why the .tmpl suffix? Sentinel will change this file while it’s running to store the current state. So you would need to clean sentinel.conf after each test. But I’ll add a script that copies the sentinel.conf.tmpl file to sentinel.conf so that we’ve a fresh start each time. sentinel00/sentinel.conf.tmpl will look like this:

plain

dir "/tmp"
port 0
sentinel monitor myprimary redis00 6379 2
sentinel down-after-milliseconds myprimary 2000
sentinel failover-timeout myprimary 4000
sentinel auth-pass myprimary dapassword
requirepass "dapassword"
sentinel announce-hostnames yes
sentinel resolve-hostnames yes

# For redis01 set "26479" as value
# For redis02 set "26579" as value
sentinel announce-port 26379
# For redis01 set "sentinel01" as value
# For redis02 set "sentinel02" as value
sentinel announce-ip "sentinel00"

# For redis01 set "26479" as value
# For redis02 set "26579" as value
tls-port 26379
tls-cert-file /certs/redis.crt
tls-key-file /certs/redis.key
tls-ca-cert-file /certs/ca.crt
tls-replication yes
tls-auth-clients no

sentinel announce-ip "sentinel00" needs to be adjusted for every Sentinel configuration (for sentinel01/sentinel.conf.tmpl it’s sentinel announce-ip "sentinel01" and for sentinel02/sentinel.conf.tmpl it’s sentinel announce-ip "sentinel02"). Also adjust announce-port and tls-port for sentinel01 and sentinel02 accordingly. As you can see dapassword is the password used for the requirepass option. If you later use redis-cli to connect to Redis this password is needed.

Docker entrypoint for Sentinel

Later I’ll use the official Redis Docker image. For Sentinel the Docker entrypoint script needs to be overridden to be able to supply the --sentinel option to make Redis start in Sentinel mode. Lets create bin/sentinel_entrypoint.sh with the following content:

bash

#!/usr/bin/env bash

sleep 5

# Start Sentinel service
exec docker-entrypoint.sh "$@" --sentinel

Make sure to make the script executable:

bash

chmod 755 bin/sentinel_entrypoint.sh

Prepare TLS certificates

To enable secure communication between the Redis components and the clients a TLS certificate is needed of course. Lets change temporary to the certs folder. Create a file gen_certs.sh with the following content:

bash

#!/usr/bin/env bash

# Generate CA
openssl genrsa -out ca.key 2048
openssl req -new -x509 -nodes -sha256 \
  -key ca.key \
  -days 3650 \
  -subj "/C=DE/CN=EXAMPLE" \
  -out ca.crt

# Generate Redis certificate files
openssl req -config req.conf -newkey rsa:2048 \
  -nodes -sha256 -keyout redis.key -out redis.csr

openssl x509 -req -sha256 -days 3650 -in redis.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out redis.crt -extfile req.conf -extensions v3_req

Make it executable with chmod 755 gen_certs.sh.

Next create a file called req.conf. It contains some configuration for the certificate and also a list of host names this certificate is valid for:

plain

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
C = DE
ST = BY
L = CITY
OU = OUNIT
CN = redis00
[v3_req]
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = redis00
DNS.2 = redis01
DNS.3 = redis02
DNS.4 = sentinel00
DNS.5 = sentinel01
DNS.6 = sentinel02
DNS.7 = 127.0.0.1

Then lets execute gen_certs.sh in the certs folder. This will generate a few more files which already used in the configuration files above:

ca.crt
ca.key
ca.srl
redis.crt
redis.csr
redis.key

The docker-compose.yml file

Back in the main directory create docker-compose.yml file with the following content:

yaml

services:
  redis00:
    image: redis:7.2-bookworm
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./redis00/redis.conf:/usr/local/etc/redis/redis.conf
      - ./certs:/certs
    user: "${USER_ID}:${GROUP_ID}"
    ports:
      - "6379:6379"

  redis01:
    image: redis:7.2-bookworm
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./redis01/redis.conf:/usr/local/etc/redis/redis.conf
      - ./certs:/certs
    user: "${USER_ID}:${GROUP_ID}"
    ports:
      - "6479:6479"
    depends_on:
      - redis00

  redis02:
    image: redis:7.2-bookworm
    command: redis-server /usr/local/etc/redis/redis.conf
    volumes:
      - ./redis02/redis.conf:/usr/local/etc/redis/redis.conf
      - ./certs:/certs
    user: "${USER_ID}:${GROUP_ID}"
    ports:
      - "6579:6579"
    depends_on:
      - redis00

  sentinel00:
    image: redis:7.2-bookworm
    entrypoint: /usr/local/bin/entrypoint.sh redis-server /usr/local/etc/redis/sentinel.conf
    user: "${USER_ID}:${GROUP_ID}"
    volumes:
      - ./bin/sentinel_entrypoint.sh:/usr/local/bin/entrypoint.sh
      - ./sentinel00:/usr/local/etc/redis
      - ./certs:/certs
    ports:
      - "26379:26379"

  sentinel01:
    image: redis:7.2-bookworm
    entrypoint: /usr/local/bin/entrypoint.sh redis-server /usr/local/etc/redis/sentinel.conf
    user: "${USER_ID}:${GROUP_ID}"
    volumes:
      - ./bin/sentinel_entrypoint.sh:/usr/local/bin/entrypoint.sh
      - ./sentinel01:/usr/local/etc/redis
      - ./certs:/certs
    ports:
      - "26479:26479"

  sentinel02:
    image: redis:7.2-bookworm
    entrypoint: /usr/local/bin/entrypoint.sh redis-server /usr/local/etc/redis/sentinel.conf
    user: "${USER_ID}:${GROUP_ID}"
    volumes:
      - ./bin/sentinel_entrypoint.sh:/usr/local/bin/entrypoint.sh
      - ./sentinel02:/usr/local/etc/redis
      - ./certs:/certs
    ports:
      - "26579:26579"

  test:
    image: debian:stable-slim
    command: bash -c "sleep 86400"
    user: "${USER_ID}:${GROUP_ID}"

start.sh

Also in the main directory create start.sh file with this content:

bash

#!/usr/bin/env bash

USER_ID="$(id -u)"
export USER_ID

GROUP_ID="$(id -g)"
export GROUP_ID

function cleanup() {
  rm -f \
    sentinel00/sentinel.conf \
    sentinel01/sentinel.conf \
    sentinel02/sentinel.conf
}

trap cleanup EXIT

for ID in 00 01 02; do
  sudo cp "sentinel${ID}/sentinel.conf.tmpl" "sentinel${ID}/sentinel.conf"
  sudo chmod 666 "sentinel${ID}/sentinel.conf"
done

docker-compose up

Make it executable with chmod 755 start.sh.

stop.sh

And finally stop.sh:

bash

#!/usr/bin/env bash

USER_ID="$(id -u)"
export USER_ID

GROUP_ID="$(id -g)"
export GROUP_ID

docker-compose down

Make it executable with chmod 755 start.sh.

Please always use the ./start script to launch the container. It makes sure that the Sentinel container always have a fresh configuration file. Sentinel makes changes to that file every time it starts up or if a failover happens.

All ports are TLS enabled! So, if you make a connection make sure that you have TLS connection enabled. The password is dapassword for all Redis’ and all Sentinels. You also need to specify the Certificate Authority file ca.crt which is in the certs directory because the TLS certificate is self-signed and therefore your local Certificate Authority store is not able to verify the validity of the certificate without that file.

Start and logs

Once you’ve started all the Redis processes with ./start.sh you should see the following lines somewhere in the log output:

log

redis01-1     | 1:S 15 Oct 2024 19:34:00.385 * MASTER <-> REPLICA sync: Finished with success
redis02-1     | 1:S 15 Oct 2024 19:34:00.385 * MASTER <-> REPLICA sync: Finished with success

That means that the replication between redis00 (usually the primary instance at the very beginning) and the replicas redis01 and redis02 was successfully initialized.

Connect via redis-cli

Once you’ve started all the Redis processes with ./start.sh and if you have redis-cli command installed, you should be able to connect to one of the Redis instances like this:

bash

redis-cli -a dapassword --tls --cacert certs/ca.crt -p 6379

If you type info you should see this output (despite a long list of other output):

plain

...
# Replication
role:master
connected_slaves:2
slave0:ip=redis01,port=6479,state=online,offset=4041,lag=1
slave1:ip=redis02,port=6579,state=online,offset=4041,lag=1
...

That’s also a good indication that replication works as expected.

Same for one of Sentinel processes:

bash

redis-cli -a dapassword --tls --cacert certs/ca.crt -p 26379

Again if you type info when connected you should see something like this:

plain

...
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_tilt_since_seconds:-1
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=myprimary,status=ok,address=redis00:6379,slaves=2,sentinels=3

Insert some data

To insert some data or just for testing connectivity one can use a little Go program. Setup Go first. Create a script called install_go:

bash

#!/usr/bin/env bash

readonly GO_VERSION="1.23.2"

rm "go${GO_VERSION}.linux-amd64.tar.gz"
wget "https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz"
tar -xzf "go${GO_VERSION}.linux-amd64.tar.gz"
export PATH="${PWD}/go/bin":${PATH}
mkdir golang
cd golang
go mod init test

Run the script

bash

chmod 700 install_go
./install_go

Make sure go binary is in the right search path and enter golang directory:

bash

export PATH="${PWD}/go/bin":${PATH}
cd golang
which go

Create a test.go file with this content:

package main

import (
  "crypto/tls"
  "crypto/x509"
  "fmt"
  "github.com/go-redis/redis/v8"
  "io/ioutil"
  "time"
  "context"
)

func main() {
  sentinelHosts := []string{
"sentinel00:26379",
"sentinel01:26379",
"sentinel02:26379",
  }

  caCertFile := "../certs/ca.crt"

  caCert, err := ioutil.ReadFile(caCertFile)
  if err != nil {
    fmt.Println("Failed to load CA certificate:", err)
    return
  }

  caCertPool := x509.NewCertPool()
  caCertPool.AppendCertsFromPEM(caCert)

  options := &redis.FailoverOptions{
    MasterName:    "myprimary",
    SentinelAddrs: sentinelHosts,
    DialTimeout:   2 * time.Second,
    TLSConfig: &tls.Config{
      RootCAs:      caCertPool,
    },
    SentinelPassword: "dapassword",
    Password: "dapassword",
  }

  client := redis.NewFailoverClient(options)

  ctx := context.Background()

  statusCmd := client.Set(ctx, "akey", "avalue", 0)
  if statusCmd.Err() != nil {
    fmt.Printf("Failed to insert key: ", statusCmd.Err())
    return
  }
}

This will just connect to the Redis Sentinels to figure out the current primary. Afterwards it connects to the current primary and inserts one key called akey:

bash

go mod tidy
go run test.go

If you now connect to Redis (port 6379) again and run info keyspace you’ll see that there is now one key:

plain

# Keyspace
db0:keys=1,expires=0,avg_ttl=0

If you run get akey the answer value will be avalue. Remove the test key by executing del akey.