Install Mattermost Messaging on Kubernetes

If you know Slack messaging and want a something similar open source alternative then Mattermost might be something for you. It’s written in Go and therefore quite low on memory and CPU usage and also quite fast - at least on the server side, the client side could be a bit faster ;-). It supports also plugins which you can write on your own or use on of the plugins available at Mattermost Integrations.

Here are some integrations/plugins that I found quite useful:

matterbridge - Simple bridge between Mattermost, IRC, XMPP, Gitter, Slack, Discord, Telegram, Rocket.Chat, Hipchat (via xmpp), Matrix, Steam and ssh-chat. Has a REST API. Minecraft server chat support via MatterLink. twittermost - twittermost is a Mattermost bot that will announce the tweets of the tweeps it follows on twitter. Remind - A bot that schedules reminders for Mattermost. Github plugin - A GitHub plugin for Mattermost. Supports GitHub SaaS and Enterprise versions. mail2most - Watch emails and send them to Mattermost. Filter emails from mail accounts and send them to Mattermost. Matterpoll - Allows users to create poll by using a slash command.

How to install Mattermost on Kubernetes (K8s) is basically described in the official documentation: Installing Mattermost on Kubernetes. I basically used just that documentation but I did a few things differently and also used Traefik as ingress proxy instead of nginx in the official documentation. So lets get started!

Obviously you need a Kubernetes cluster running. Currently Kubernetes 1.15 or higher is needed (mainly because of the minio-operator). You can either install Mattermost via Helm or using a so called Operator which I’m using here as already mentioned. A K8s operator in general or the Mattermost operator in this case basically takes care of installing, setting up, configuring and handling software or services needed to run Mattermost. Mattermost also needs at least a MySQL database, MinIO (as object storage) and an ingress controller. The official documentation uses the nginx ingress controller but since I already have Traefik as K8s ingress proxy running I’ll use that one of course.

As already mentioned Mattermost needs a database to store all the messages. The official documentation uses the Presslabs MySQL Operator. Presslabs uses Percona Server for MySQL 5.7 because of backup improvements (eg. backup locks), monitoring improvements and some serviceability improvements (eg. utility user). Also of possible interest: The MySQL operator uses orchestrator, a tool developed by GitHub for MySQL replication topology management and high availability.

One thing upfront: MySQL and MinIO need persistent storage of course. If you keep the default settings then make sure you have a default storage class configured and in case you have different storage classes make sure that the default one is the one that you want to use for MySQL and MinIO. To get the available storage classes use this command:

1> kubectl get storageclasses

NAME                       PROVISIONER         AGE
hcloud-volumes (default)   csi.hetzner.cloud   245d

As you can see in my case I’m using Hetzner Cloud Storage. I installed the Hetzner CSI driver accordingly. It of course doesn’t matter which storage you use just make sure that you’ve a default storage class defined so that the operator can allocate persistent volumes accordingly.

All the operators run in their own namespace. So lets create one for the MySQL operator:

kubectl create ns mysql-operator

The next thing I did was downloading the mysql-operator.yaml file for reviewing:

wget https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/mysql-operator/mysql-operator.yaml

The file contains two secrets that should be changed to not use the default values:

data:
  TOPOLOGY_USER: "b3JjaGVzdHJhdG9y"
  TOPOLOGY_PASSWORD: "YW1QZHlGN1VmRA=="

The first one sets a user for the orchestrator which is needed to connect to the MySQL cluster and the second one sets a password for the orchestrator user to connect to MySQL cluster. Both values are base64 encoded and this are the defaults:

1> echo "b3JjaGVzdHJhdG9y" | base64 -d
orchestrator

1> echo "YW1QZHlGN1VmRA==" | base64 -d
amPdyF7UfD

So at least for the password we should use a different value. E.g. lets assume we use super_password as our password value the new base64 encoded password would be:

echo "super_password" | base64

Now we can replace the value of TOPOLOGY_PASSWORD with the new value.

I’m also someone who likes to have docker images in my private registry. This makes downloading images faster and I’m not depended on external resources to name two advantages. This is of course optional. You can skip all following docker pull -> tag -> push steps if you just want to use the quay.io image hub. Sadly I’ve to admit that putting the images in your own registry only works partly. The operators have hard coded values for some images like prom/mysqld-exporter or minio/minio:RELEASE.2020-01-03T19-12-21Z. I haven’t found a way to change that yet. It looks like that this image values are not exposed so that it is not possible to override them. But this needs to be further investigated.

As mentioned I want the images in my own Docker registry so I pull the mysql-operator images, tag them and push them to my private registry. E.g. (0.3.8 is the latest version as time of writing this blog post. For latest version see presslabs / mysql-operator.

export MYSQL_OPERATOR_VERSION="0.3.8"

for IMAGE in presslabs/mysql-operator presslabs/mysql-operator-sidecar presslabs/mysql-orchestrator
do
  docker pull "quay.io/${IMAGE}:${MYSQL_OPERATOR_VERSION}"
  docker tag "quay.io/${IMAGE}:${MYSQL_OPERATOR_VERSION}" "registry.domain.tld:5000/${IMAGE}:${MYSQL_OPERATOR_VERSION}"
  docker push "registry.domain.tld:5000/${IMAGE}:${MYSQL_OPERATOR_VERSION}
done

Of course you need to replace registry.domain.tld:5000 with the domain name and port of your private Docker registry. Now we can change all occurrences of the Docker images in the YAML file accordingly.

To pull from a private Docker registry your normally also need to add either a imagePullSecret or patch the namespace default service account. This is also optional and depends on your private Docker image registry requirements.

Now we can deploy the mysql-operator:

kubectl apply -n mysql-operator -f mysql-operator.yaml

Next we create a namespace for MinIO operator:

kubectl create ns minio-operator

Now download the YAML file for the MinIO operator:

wget https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/minio-operator/minio-operator.yaml

In this case I only made one change. Again I pulled the image, tagged it and pushed it to my private Docker registry (this is also optional):

docker pull minio/k8s-operator:1.0.6
docker tag minio/k8s-operator:1.0.6 registry.domain.tld:5000/minio/k8s-operator:1.0.6
docker push registry.domain.tld:5000/minio/k8s-operator:1.0.6

registry.domain.tld:5000 needs to be replaced of course. As mentioned above image pull secrets might be required.

Now we can deploy the minio-operator:

kubectl apply -n minio-operator -f minio-operator.yaml

According to the Mattermost documentation mentioned above installing nginx ingress controller would be the next thing. But since I already have Traefik as ingress I skip that step.

Next on the list is installing the Mattermost operator. First a namespace is needed:

kubectl create ns mattermost-operator

As with the other operators I first download the YAML file and have a look what needs to be adjusted:

wget https://raw.githubusercontent.com/mattermost/mattermost-operator/master/docs/mattermost-operator/mattermost-operator.yaml

Currently there is only one thing that I change here: image: mattermost/mattermost-operator:latest. I don’t like it to use latest tag. That’s a bad habit. Images should mostly always have a version assigned. The available versions can be found here: https://github.com/mattermost/mattermost-operator/releases. In my case v1.1.1 is the latest version. And once again I put the images in a private registry:

docker pull mattermost/mattermost-operator:v1.1.1
docker tag mattermost/mattermost-operator:v1.1.1 registry.domain.tld:5000/mattermost/mattermost-operator:v1.1.1
docker push registry.domain.tld:5000/mattermost/mattermost-operator:v1.1.1

After changing image: registry.domain.tld:5000/mattermost/mattermost-operator:v1.1.1 in the YAML file accordingly the operator can be installed (again you may need a image pull secret in this namespace):

kubectl apply -n mattermost-operator -f mattermost-operator.yaml

Next a installation manifest file is needed which basically defines what the operators should deploy e.g.:

apiVersion: mattermost.com/v1alpha1
kind: ClusterInstallation
metadata:
  name: mm-example-com
spec:
  size: 100users
  ingressName: mm.example.com
  ingressAnnotations:
    kubernetes.io/ingress.class: traefik
  version: 5.19.1
  mattermostLicenseSecret: ""
  database:
    storageSize: 10Gi
  minio:
    storageSize: 30Gi
  elasticSearch:
    host: ""
    username: ""
    password: ""

As you can see the value for metadata.name and spec.ingressName are basically the same. The first one specifies the name of your Mattermost installation in K8s and the second specifies the domain name that you’ll later put into your browser URL bar later to open your Mattermost website.

size can be 100users, 1000users, 5000users, 10000users, or 25000users. For me 100users is big enough. This parameter has also influences on database and memory sizings. Depending on this parameter the operators choose values database buffers e.g.

kubernetes.io/ingress.class: traefik annotation specifies that Traefik ingress controller should create an ingress accordingly and in my case Traefik will also request a free SSL certificate from Let’s Encrypt.

The latest Mattermost version currently is 5.19.1. You can find the latest releases here: https://github.com/mattermost/mattermost-server/releases storageSize for database and minio is of course something everyone needs to figure out on its own. Messages go into the database so with 10Gi you can store quite a lot of messages. Uploads like pictures and videos will end up in MinIO (the object storage). This of course can get quite big if you share a lot of videos or pictures. But 30Gi should be a good starting point. You can also specify an ElasticSearch host here. Using KubeDB Elasticsearch operator might be a good option to install Elasticsearch quickly. But I skip that one for now.

As mentioned already I use Traefik as ingress proxy so this will my entrypoint to Mattermost. If you’ve a static IP for Traefik you can already create a DNS entry before applying the manifest. In the manifest above the ingressName is mm.example.com. In case Let’s Encrypt is configured Traefik will fetch a SSL certificate for mm.example.com.

After the manifest is created save the content in a file called mattermost.yaml .e.g. Then create again a namespace and apply the manifest:

kubectl create ns mattermost
kubectl apply -n mattermost -f mattermost.yaml

This takes a while. You can use kubectl -n mattermost get all -o wide to see when all pods are ready.

After everything is up and running you can head over to the URL you specified for your Mattermost installation e.g. https://mm.example.com and finish the installation. The first user you create also becomes Mattermost administrator. As Mattermost is now exposed and reachable from the public internet you want to make sure to create that user quite fast to avoid surprises ;-) In general you should visit the System console and adjust all necessary settings.

One of the next things that you should think about is configuring a backup for the MySQL database and the MinIO object store. Also configuring network policies might be a good idea.