In part 1 of this blog series, we started to build a KUDO operator for Galera, and I showed how we built up plans, steps and tasks to create the Galera bootstrap node. In this second part, we’ll extend the operator to deploy more nodes into our Galera cluster.
The next thing we need our operator to do is deploy the configuration required for the additional nodes to join the cluster initially. Let’s add some steps and tasks to do that :
phases:
- name: deploy
strategy: serial
steps:
- name: bootstrap_config
tasks:
- bootstrap_config
- name: bootstrap_service
tasks:
- bootstrap_service
- name: bootstrap_deploy
tasks:
- bootstrap_deploy
- name: firstboot_config
tasks:
- firstboot_config
tasks:
- name: bootstrap_config
kind: Apply
spec:
resources:
- bootstrap_config.yaml
- name: bootstrap_service
kind: Apply
spec:
resources:
- bootstrap_service.yaml
- name: bootstrap_deploy
kind: Apply
spec:
resources:
- bootstrap_deploy.yaml
- name: firstboot_config
kind: Apply
spec:
resources:
- galera_config.yaml
Now we need to create the yaml file which our firstboot_config task is going to apply.
MacBook-Pro:templates matt$ cat galera_config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Name }}-nodeconfig
namespace: {{ .Namespace }}
data:
galera.cnf: |
[galera]
wsrep_on = ON
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_sst_method = mariabackup
wsrep_cluster_address = gcomm://{{ .Name }}-bootstrap-svc,{{ $.Name }}-{{ $.OperatorName }}-0.{{ $.Name }}-hs{{ range $i, $v := untilStep 1 (int .Params.NODE_COUNT) 1 }},{{ $.Name }}-{{ $.OperatorName }}-{{ $v }}.{{ $.Name }}-hs{{end}}
wsrep_sst_auth = "{{ .Params.SST_USER }}:{{ .Params.SST_PASSWORD }}"
binlog_format = ROW
innodb.cnf: |
[innodb]
innodb_autoinc_lock_mode = 2
innodb_flush_log_at_trx_commit = 0
innodb_buffer_pool_size = 122M
Again this is going to add a ConfigMap to our cluster, with two data sections this time - some Galera specific configuration, and some configuration which is recommended for the InnoDB engine when using Galera.
As you can see the templating here is more complex than in our bootstrap configuration, so let’s break that down a bit. In our wsrep_cluster_address variable, we need the address of the service to connect to our bootstrap node, and then we need the addresses of each of the other nodes that will be in our cluster. That could be any arbitrary number that we will configure in our operator, so we need some code to iterate through a number of nodes and build that connection string. This list also needs to be comma separated. Here the addition of Sprig templating to KUDO comes into it’s own :
wsrep_cluster_address = gcomm://{{ .Name }}-bootstrap-svc,{{ $.Name }}-{{ $.OperatorName }}-0.{{ $.Name }}-hs{{ range $i, $v := untilStep 1 (int .Params.NODE_COUNT) 1 }},{{ $.Name }}-{{ $.OperatorName }}-{{ $v }}.{{ $.Name }}-hs{{end}}
Before we break this code down, let’s clarify a couple of assumptions that we can make :
- Our nodes will be in a StatefulSet, so we can predict the naming will be .Name-.OperatorName-number, where .Name and .OperatorName are created by KUDO.
- We will be creating a headless service to access them, which we will name as .Name-hs to ensure uniqueness, and we will document that later in this blog. That will give us predictable DNS entries for each node in our StatefulSet in the format .Name-.OperatorName-0.Name-hs
Working from those assumptions we can start to put together the connection string. Firstly we add the service address for our bootstrap node, which we created in the last blog, followed by a comma :
gcomm://{{ .Name }}-bootstrap-svc,
We also know there will always be a node ending in 0, since we must have at least 1 node, so we can add the DNS entry for that nodes service anyway
gcomm://{{ .Name }}-bootstrap-svc,{{ $.Name }}-{{ $.OperatorName }}-0.{{ $.Name }}-hs
For the next part, we need to define a value for the number of nodes we are going to deploy, in our params.yaml so we can refer to that here :
- name: NODE_COUNT
description: "Number of nodes to create in the cluster"
default: "3"
Now we can use that variable in the next step of the Sprig templating :
{{ range $i, $v := untilStep 1 (int .Params.NODE_COUNT) 1 }},{{ $.Name }}-{{ $.OperatorName }}-{{ $v }}.{{ $.Name }}-hs{{end}}
Here we are using the range function, part of standard Go templating, which gives us values and indexes in $v and $i, to which we are passing a range generated using the UntilStep function from Sprig. UntilStep will give us a list of integers, starting at 1, through and up to NODE_COUNT, in steps of 1. So if NODE_COUNT is 3, then the range generated by UntilStep here would be 1,2.
Note that we have to cast .Params.NODE_COUNT to an integer before we use it, since it’s just a string in our params file. Then we will print out each entry for the nodes in our cluster, preceded by a comma. As we are dealing with integers, we also need to precede the other string variables inside our function with $. Finally we close the templating code with {{end}}
Working out complex templating functions can be tricky, you might find it useful to have some stub Go code you can run when testing the templating. I have an example that I use at https://github.com/mattj-io/template_test, which just prints out the template to stdout.
We can check our new parameters and tasks are correctly defined using the CLI, which at least gives us some certainty that our YAML is correct :
MacBook-Pro:operator matt$ kubectl kudo package list plans .
plans
└── deploy (serial)
└── [phase] deploy (serial)
├── [step] bootstrap_config
│ └── bootstrap_config
├── [step] bootstrap_service
│ └── bootstrap_service
├── [step] bootstrap_deploy
│ └── bootstrap_deploy
└── [step] firstboot_config
└── firstboot_config
We can see our new step defined.
MacBook-Pro:operator matt$ kubectl kudo package list tasks .
Name Kind Resources
bootstrap_config Apply [bootstrap_config.yaml]
bootstrap_service Apply [bootstrap_service.yaml]
bootstrap_deploy Apply [bootstrap_deploy.yaml]
firstboot_config Apply [galera_config.yaml]
And the task is also defined
MacBook-Pro:operator matt$ kubectl kudo package list parameters .
Name Default Required
IST_PORT 4568 true
MYSQL_PORT 3306 true
MYSQL_ROOT_PASSWORD admin true
NODE_COUNT 3 true
REPLICATION_PORT 4567 true
SST_PASSWORD admin true
SST_PORT 4444 true
SST_USER root true
As well as our new parameter NODE_COUNT.
Let’s go ahead and deploy our operator again, and see what happens.
MacBook-Pro:operator matt$ kubectl kudo install .
operator.kudo.dev/v1beta1/galera created
operatorversion.kudo.dev/v1beta1/galera-0.1.0 created
instance.kudo.dev/v1beta1/galera-instance created
MacBook-Pro:operator matt$ kubectl kudo plan status --instance galera-instance
Plan(s) for "galera-instance" in namespace "default":
.
└── galera-instance (Operator-Version: "galera-0.1.0" Active-Plan: "deploy")
└── Plan deploy (serial strategy) [COMPLETE], last updated 2020-06-24 14:32:26
└── Phase deploy (serial strategy) [COMPLETE]
├── Step bootstrap_config [COMPLETE]
├── Step bootstrap_service [COMPLETE]
├── Step bootstrap_deploy [COMPLETE]
└── Step firstboot_config [COMPLETE]
So we can see our operator completed all the steps we had from the previous blog, as well as completing the new step firstboot_config. We expect this to have created a ConfigMap, so we can check it’s correctly been templated out.
MacBook-Pro:operator matt$ kubectl get configmaps
NAME DATA AGE
galera-instance-bootstrap 1 6m17s
galera-instance-nodeconfig 2 5m32s
MacBook-Pro:operator matt$ kubectl describe configmap galera-instance-nodeconfig
Name: galera-instance-nodeconfig
Namespace: default
Labels: heritage=kudo
kudo.dev/instance=galera-instance
kudo.dev/operator=galera
Annotations: kudo.dev/last-applied-configuration:
{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"galera-instance-nodeconfig","namespace":"default","creationTimestamp":null,"labe...
kudo.dev/last-plan-execution-uid: 35f18f81-bb38-4881-a132-e4741cb913a2
kudo.dev/operator-version: 0.1.0
kudo.dev/phase: deploy
kudo.dev/plan: deploy
kudo.dev/step: firstboot_config
Data
====
galera.cnf:
----
[galera]
wsrep_on = ON
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_sst_method = mariabackup
wsrep_cluster_address = gcomm://galera-instance-bootstrap-svc,galera-instance-galera-0.galera-instance-hs,galera-instance-galera-1.galera-instance-hs,galera-instance-galera-2.galera-instance-hs
wsrep_sst_auth = "root:admin"
binlog_format = ROW
innodb.cnf:
----
[innodb]
innodb_autoinc_lock_mode = 2
innodb_flush_log_at_trx_commit = 0
innodb_buffer_pool_size = 122M
Events: <none>
So we can see our ConfigMap is deployed to the cluster, and the templating has created the correct entry for wsrep_cluster_address when NODE_COUNT is defined as 3. Clean up the cluster once again, and let’s move on.
We are going to deploy our remaining nodes in a StatefulSet, and we’ll need a way to connect to them, so let’s add a Service. Firstly, define the steps, tasks and resources in our operator.yaml :
plans:
deploy:
strategy: serial
phases:
- name: deploy
strategy: serial
steps:
- name: bootstrap_config
tasks:
- bootstrap_config
- name: bootstrap_service
tasks:
- bootstrap_service
- name: bootstrap_deploy
tasks:
- bootstrap_deploy
- name: firstboot_config
tasks:
- firstboot_config
- name: cluster_services
tasks:
- cluster_services
tasks:
- name: bootstrap_config
kind: Apply
spec:
resources:
- bootstrap_config.yaml
- name: bootstrap_service
kind: Apply
spec:
resources:
- bootstrap_service.yaml
- name: bootstrap_deploy
kind: Apply
spec:
resources:
- bootstrap_deploy.yaml
- name: firstboot_config
kind: Apply
spec:
resources:
- galera_config.yaml
- name: cluster_services
kind: Apply
spec:
resources:
- hs-service.yaml
Now let’s create the hs-service.yaml which this step needs :
MacBook-Pro:templates matt$ cat hs-service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ .Name }}-hs
namespace: {{ .Namespace }}
labels:
app: galera
galera: {{ .Name }}
spec:
ports:
- port: {{ .Params.MYSQL_PORT }}
name: mysql
- port: {{ .Params.SST_PORT }}
name: sst
- port: {{ .Params.REPLICATION_PORT }}
name: replication
- port: {{ .Params.IST_PORT }}
name: ist
clusterIP: None
selector:
app: galera
instance: {{ .Name }}
So we are creating a headless service ( with no ClusterIP ) for all the ports Galera needs, with values pulled in from our params.yaml, and we are selecting on two labels, app and instance, to ensure uniqueness for each deployment. The reason for the headless service is that we don’t need a load balanced address for this service, we only need DNS entries for each node in our StatefulSet.
Now we can deploy again, and check the service has been correctly created.
MacBook-Pro:operator matt$ kubectl kudo plan status --instance galera-instance
Plan(s) for "galera-instance" in namespace "default":
.
└── galera-instance (Operator-Version: "galera-0.1.0" Active-Plan: "deploy")
└── Plan deploy (serial strategy) [COMPLETE], last updated 2020-06-24 14:50:51
└── Phase deploy (serial strategy) [COMPLETE]
├── Step bootstrap_config [COMPLETE]
├── Step bootstrap_service [COMPLETE]
├── Step bootstrap_deploy [COMPLETE]
├── Step firstboot_config [COMPLETE]
└── Step cluster_services [COMPLETE]
MacBook-Pro:operator matt$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
galera-instance-bootstrap-svc ClusterIP None <none> 3306/TCP,4444/TCP,4567/TCP,4568/TCP 2m51s
galera-instance-hs ClusterIP None <none> 3306/TCP,4444/TCP,4567/TCP,4568/TCP 2m28s
MacBook-Pro:operator matt$ kubectl describe service galera-instance-hs
Name: galera-instance-hs
Namespace: default
Labels: app=galera
galera=galera-instance
heritage=kudo
kudo.dev/instance=galera-instance
kudo.dev/operator=galera
Annotations: kudo.dev/last-applied-configuration:
{"kind":"Service","apiVersion":"v1","metadata":{"name":"galera-instance-hs","namespace":"default","creationTimestamp":null,"labels":{"app"...
kudo.dev/last-plan-execution-uid: 0374f5bc-27e9-4b28-ada9-371a149cf05b
kudo.dev/operator-version: 0.1.0
kudo.dev/phase: deploy
kudo.dev/plan: deploy
kudo.dev/step: cluster_services
Selector: app=galera,instance=galera-instance
Type: ClusterIP
IP: None
Port: mysql 3306/TCP
TargetPort: 3306/TCP
Endpoints: <none>
Port: sst 4444/TCP
TargetPort: 4444/TCP
Endpoints: <none>
Port: replication 4567/TCP
TargetPort: 4567/TCP
Endpoints: <none>
Port: ist 4568/TCP
TargetPort: 4568/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
We can see here, the additional service has been created correctly, is using the correct selectors, and it currently has no endpoints since we haven’t deployed any nodes yet.
Once again, let’s clean up the cluster and move on to our next step. Now we have our configuration and service defined, we can finally add our stateful set. Once again, add the steps, tasks and resources to our operator.yaml
steps:
- name: statefulset
tasks:
- statefulset
tasks:
- name: statefulset
kind: Apply
spec:
resources:
- statefulset.yaml
And let’s create the statefulset.yaml :
MacBook-Pro:templates matt$ cat statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ .Name }}-{{ .OperatorName }}
namespace: {{ .Namespace }}
labels:
galera: {{ .OperatorName }}
app: galera
instance: {{ .Name }}
annotations:
reloader.kudo.dev/auto: "true"
spec:
selector:
matchLabels:
app: galera
galera: {{ .OperatorName }}
instance: {{ .Name }}
serviceName: {{ .Name }}-hs
replicas: {{ .Params.NODE_COUNT }}
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: galera
galera: {{ .OperatorName }}
instance: {{ .Name }}
spec:
initContainers:
# Stop the image bootstrap script from trying to set up single master
- name: {{ .Name }}-init
image: busybox:1.28
command: ['sh', '-c', 'set -x; if [ ! -d /datadir/mysql ]; then mkdir /datadir/mysql; fi']
volumeMounts:
- name: {{ .Name }}-datadir
mountPath: "/datadir"
containers:
- name: mariadb
image: mariadb:latest
args:
- "--ignore_db_dirs=lost+found"
env:
# Use secret in real usage
- name: MYSQL_ROOT_PASSWORD
value: {{ .Params.MYSQL_ROOT_PASSWORD }}
ports:
- containerPort: {{ .Params.MYSQL_PORT }}
name: mysql
- containerPort: {{ .Params.SST_PORT }}
name: sst
- containerPort: {{ .Params.REPLICATION_PORT }}
name: replication
- containerPort: {{ .Params.IST_PORT }}
name: ist
livenessProbe:
exec:
command: ["mysqladmin", "-p{{ .Params.MYSQL_ROOT_PASSWORD }}", "ping"]
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
# Check we can execute queries over TCP (skip-networking is off).
command: ["mysql", "-p{{ .Params.MYSQL_ROOT_PASSWORD }}", "-h", "127.0.0.1", "-e", "SELECT 1"]
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
volumeMounts:
- name: {{ .Name }}-config
mountPath: /etc/mysql/conf.d
- name: {{ .Name }}-datadir
mountPath: /var/lib/mysql
volumes:
- name: {{ .Name }}-config
configMap:
name: {{ .Name }}-nodeconfig
items:
- key: galera.cnf
path: galera.cnf
- key: innodb.cnf
path: innodb.cnf
volumeClaimTemplates:
- metadata:
name: {{ .Name }}-datadir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{ .Params.VOLUME_SIZE }}
There’s a few things to note in this yaml file. Firstly, our StatefulSet is going to use persistent volumes for the MySQL data directory /var/lib/mysql, and we’re also going to mount the ConfigMaps we created earlier into /etc/mysql/conf.d where they will get picked up on startup. :
volumeMounts:
- name: {{ .Name }}-config
mountPath: /etc/mysql/conf.d
- name: {{ .Name }}-datadir
mountPath: /var/lib/mysql
volumes:
- name: {{ .Name }}-config
configMap:
name: {{ .Name }}-nodeconfig
items:
- key: galera.cnf
path: galera.cnf
- key: innodb.cnf
path: innodb.cnf
Since this is a StatefulSet, we also define a VolumeClaimTemplate :
volumeClaimTemplates:
- metadata:
name: {{ .Name }}-datadir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: {{ .Params.VOLUME_SIZE }}
This is going to use a configurable VOLUME_SIZE parameter that we need to add to our params.yaml
- name: VOLUME_SIZE
description: "Size of persistent volume"
default: "10M"
I’ve added a very small default size here just for testing. We’ll then mount that at the right place.
The MariaDB image by default has bootstrap scripts which try to set up a single node instance. For our use case we need to override that script, which works by checking for a mysql folder in /var/lib/mysql, indicating a database is already present and to skip that portion of the bootstrapping. We’ll override that by using an init container, mounting the data directory and creating the folder before those scripts run, which will allow the Galera process to run correctly and synchronise to the cluster as per the configuration we created in the ConfigMap.
initContainers:
# Stop the image bootstrap script from trying to set up single master
- name: {{ .Name }}-init
image: busybox:1.28
command: ['sh', '-c', 'set -x; if [ ! -d /datadir/mysql ]; then mkdir /datadir/mysql; fi']
volumeMounts:
- name: {{ .Name }}-datadir
mountPath: "/datadir"
We are using our NODE_COUNT parameter to set how many replicas should be in this StatefulSet
replicas: {{ .Params.NODE_COUNT }}
And we are also defining the ports using the parameters we set for those in the first of these blog posts.
ports:
- containerPort: {{ .Params.MYSQL_PORT }}
name: mysql
- containerPort: {{ .Params.SST_PORT }}
name: sst
- containerPort: {{ .Params.REPLICATION_PORT }}
name: replication
- containerPort: {{ .Params.IST_PORT }}
name: ist
Let’s go ahead and test this part of our operator - for testing purposes you may want to set NODE_COUNT to 1 in your params.yaml.
MacBook-Pro:operator matt$ kubectl kudo plan status --instance galera-instance
Plan(s) for "galera-instance" in namespace "default":
.
└── galera-instance (Operator-Version: "galera-0.1.0" Active-Plan: "deploy")
└── Plan deploy (serial strategy) [COMPLETE], last updated 2020-06-24 15:28:32
└── Phase deploy (serial strategy) [COMPLETE]
├── Step bootstrap_config [COMPLETE]
├── Step bootstrap_service [COMPLETE]
├── Step bootstrap_deploy [COMPLETE]
├── Step firstboot_config [COMPLETE]
├── Step cluster_services [COMPLETE]
└── Step statefulset [COMPLETE]
MacBook-Pro:operator matt$ kubectl get pods
NAME READY STATUS RESTARTS AGE
galera-instance-bootstrap-7566f8b69b-7fxxm 1/1 Running 0 2m50s
galera-instance-galera-0 1/1 Running 0 2m7s
galera-instance-galera-1 1/1 Running 0 82s
galera-instance-galera-2 1/1 Running 0 56s
All of our instances for our running cluster are now created. Let’s check the logs and see if Galera looks like it is working :
MacBook-Pro:operator matt$ kubectl logs galera-instance-galera-0
----SNIPPED---
2020-06-24 14:28:15 1 [Note] WSREP: ================================================
View:
id: b60a66f0-b626-11ea-a19c-0a391f30f86f:7175
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 1
members(4):
0: c01a8198-b626-11ea-aaa1-1632aa349ff0, galera-instance-bootstrap-7566f
1: d20464f9-b626-11ea-bd04-7aaebcdf5dd7, galera-instance-galera-0
2: e1dda89a-b626-11ea-82ed-f72190284b55, galera-instance-galera-1
3: f07db060-b626-11ea-8916-7e6c9ef1b8d5, galera-instance-galera-2
=================================================
2020-06-24 14:28:15 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-06-24 14:28:15 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 7175
2020-06-24 14:28:15 1 [Note] WSREP: Min available from gcache for CC from group: 7173
2020-06-24 14:28:16 0 [Note] WSREP: Member 3.0 (galera-instance-galera-2) requested state transfer from '*any*'. Selected 0.0 (galera-instance-bootstrap-7566f8b69b-7fxxm)(SYNCED) as donor.
2020-06-24 14:28:18 0 [Note] WSREP: (d20464f9-bd04, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-06-24 14:28:29 0 [Note] WSREP: 0.0 (galera-instance-bootstrap-7566f8b69b-7fxxm): State transfer to 3.0 (galera-instance-galera-2) complete.
2020-06-24 14:28:29 0 [Note] WSREP: Member 0.0 (galera-instance-bootstrap-7566f8b69b-7fxxm) synced with group.
2020-06-24 14:28:30 0 [Note] WSREP: 3.0 (galera-instance-galera-2): State transfer from 0.0 (galera-instance-bootstrap-7566f8b69b-7fxxm) complete.
2020-06-24 14:28:30 0 [Note] WSREP: Member 3.0 (galera-instance-galera-2) synced with group.
So at this point we can see our StatefulSet has come up correctly, and all of our nodes have joined the cluster ! Everything should be working correctly now internally in the cluster, but this is stil not production ready - our bootstrap node is no longer required, we need to enable external connectivity to the cluster, and add functionality to make sure we can safely scale up and down whilst still maintaining full operation. In the third part of this blog series, we’ll extend our operator to address all of those issues.