Kubernetes the Not So Hard Way With Ansible - Network policies with kube-router
CHANGELOG
2020-08-31
While parts of this blog post are still valid (and Kube-router is still a very nice and up2date Kubernetes networking solution) I switched to Cilium as my “all in one” networking solution as described in Kubernetes the not so hard way with Ansible - The worker. Besides network policies this thing just has everything ever needed for Kubernetes networking and still it’s easy to use. So this blog post is just left for reference reasons. It won’t be updated anymore.
If you followed my Kubernetes the Not So Hard Way With Ansible blog posts so far you should now have a pretty usable Kubernetes cluster running. What’s missing - but not strictly required - besides persistent storage (like GlusterFS) are Network Policies. From the Kubernetes documentation: A network policy is a specification of how groups of pods are allowed to communicate with each other and other network endpoints. NetworkPolicy resources use labels to select pods and define rules which specify what traffic is allowed to the selected pods.
Enter kube-router
As we already have Kubernetes networking up and running we only need a network plugin that implements the network policy specification. Calico is one of the different options but it’s a bit to heavy for what we need for our setup. CloudNative Labs kube-router provides a network policy implementation and can be activated basically with one switch. kube-router provides many more features like distributed load balancing and routing for pods but we will only use the firewall part which implement’s ingress (incoming traffic to pods) and egress (outgoing traffic from pods) firewall support. The nice thing here is that it only uses native Linux networking tools to accomplish this. So it makes it quite “easy” (well say it’s at least possible if you deep dive into Linux netfilter/iptables ;-) ) to debug if something strange happens. There is more information in Kube-router: Enforcing Kubernetes network policies with iptables and ipset. So let’s start ;-)
kube-router Role
We basically need quite a few Kubernetes resources that we already used for Traefik ingress. The first resource is a ClusterRole. kube-router needs some cluster wide permissions and we define this permissions in this ClusterRole. So lets create a variable kube_router_clusterrole in Ansible’s group_vars/all.yaml variable file (or where it fits best for you):
kube_router_clusterrole: |
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kube-router
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
- services
- nodes
- endpoints
verbs:
- get
- list
- watch
- apiGroups: ["networking.k8s.io"]
resources:
- networkpolicies
verbs:
- get
- list
- watch
- apiGroups: ["extensions"]
resources:
- networkpolicies
verbs:
- get
- list
- watch As you can see here we create a ClusterRole called kube-router. We specify no namespace as ClusterRole’s are not namespaced. There’re resources and verbs defined for three apiGroups. The fist one indicates the core API group as it’s value is "". For all three apiGroups we want to allow get,list and watch (the verbs) for the resources defined.
kube-router ClusterRoleBinding
Next we define a ClusterRoleBinding:
kube_router_clusterrolebinding: |
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kube-router
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-router
subjects:
- kind: ServiceAccount
name: kube-router
namespace: kube-system The name of the ClusterRoleBinding is kube-router. It’s basically the glue between the ClusterRole and the ServiceAccount (see below). This means in subjects we list users, groups or service accounts and in roleRef we assign basically what we want to allow. In the case above we allow everything defined in our kube-router ClusterRole.
kube-router ServiceAccount
I already mentioned the service account so let’s add a Ansible variable for it:
kube_router_serviceaccount: |
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-router
namespace: kube-system Only a few lines here. Later when we configure the kube-router DaemonSet we configure it to use this service account which enables kube-router to get all the information from the API server it needs to work properly. E.g. if new pods are created then kube-router gets the information (via so called watcher) that new pods are created and applies the network policy accordingly if needed (that’s the job of the different controller that kube-router provides depending what resource needs to be managed).
kube-router DaemonSet
And while talking about the DaemonSet… ;-) Here is the Ansible variable for it:
kube_router_daemonset: |
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: kube-router
namespace: kube-system
labels:
k8s-app: kube-router
spec:
template:
metadata:
labels:
k8s-app: kube-router
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: kube-router
hostNetwork: true
containers:
- name: kube-router
image: cloudnativelabs/kube-router:v0.0.20
args:
- --run-router=false
- --run-firewall=true
- --run-service-proxy=false
- --masquerade-all
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: cni-conf-dir
mountPath: /etc/cni/net.d
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
volumes:
- name: lib-modules
hostPath:
path: /lib/modules
- name: cni-conf-dir
hostPath:
path: /etc/cni/net.d A DaemonSet means that the specified container (cloudnativelabs/kube-router:v0.0.20 in our case / see https://github.com/cloudnativelabs/kube-router/releases for the latest releases) will run on every worker node. That’s what we need here as the firewall rules needs to be modified on this nodes accordingly to enforce the network policy.
DaemonSet config walkthrough
Let’s get quickly through the important DaemonSet config settings:
metadata:
name: kube-router
namespace: kube-system
labels:
k8s-app: kube-routerThat’s pretty obvious: We name the DaemonSet kube-router, run it in kube-system namespace and put a label on it.
Next comes the specification of the template that will be used by Kubernetes to create the pods on each worker node. The important parts here:
serviceAccountName: kube-router
hostNetwork: trueThis specifies that this DaemonSet should use the ServiceAccount we created above. This basically assigns kube-router the permissions which we defined in the kube-router ClusterRole. Additionally the pods should use the host’s network which is of course important as we need the firewall rules at this level and not somewhere above this “layer”.
containers:
- name: kube-router
image: cloudnativelabs/kube-router:v0.0.20
args:
- --run-router=false
- --run-firewall=true
- --run-service-proxy=false
- --masquerade-all
securityContext:
privileged: true
imagePullPolicy: IfNotPresentThis is basically the heart of the whole DaemonSet. This config will cause the cloudnativelabs/kube-router be pulled if it is no already on the worker node’s Docker image cache. The pods have privileged access because they need to do some “low level” tasks on the host which needs this kind of permissions. And we only turn on the firewall functionality of kube-router via --run-firewall=true argument.
As enforcing NetworkPolicy is a critical thing we can tell Kubernetes that these pods are kind of super important (see Guaranteed Scheduling For Critical Add-On Pods ). This means that Kubernetes tries it’s best to schedule such kind of pods over “normal” pods if resources are available. It will unschedule “normal” pods to get critical pods up and running. For this to work we need a few settings:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''and
tolerations:
- key: CriticalAddonsOnly
operator: ExistsRoll out NetworkPolicy’s resources with Ansible
Now to roll out kube-router get the playbook I created by cloning my ansible-kubernetes-playbooks repository e.g.:
git clone https://github.com/githubixx/ansible-kubernetes-playbooks.gitThen cd kube-router and run the playbook with
ansible-playbook install_or_update.ymlThis will install all the resources we defined above and of course the kube-router DaemonSet.
Some NetworkPolicy examples
Now you should be able to play around with Network Policies. One thing to start with is block all traffic between all pods (here for the default namespace):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- IngressSave the content in default-deny.yaml e.g. and run kubectl apply -f default-deny.yaml. Afterwards pods shouldn’t be able to communicate anymore. But this will also prevent Traefik (see /posts/kubernetes-the-not-so-hard-way-with-ansible-ingress-with-traefik/) ) to send requests to e.g. some nginx pods that deliver our static blog or whatever. So with this policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: best-blog-ever-allow-external
spec:
podSelector:
matchLabels:
app: best-blog-ever
ingress:
- from: []
ports:
- port: 80we will allow requests to port 80 from any ([]) source to pods that have the label app=best-blog-ever assigned. Adjust the settings to your need, save them into best-blog-ever-allow-external.yaml and run kubectl apply -f best-blog-ever-allow-external.yaml. This will allow our Traefik ingress again to access the pods that matches the app=best-blog-ever label.
Have fun! ;-)