One of the most common deployments of services in k8s has always been a simple mysql-service. You google and you find a thousand examples out there.
But recently I started experimenting with debezium, and it requires a mysql with a custom
my.cnf file in order to enable full binary logs.
First I deployed this service on my
minikube with no issues at all:
apiVersion: v1 kind: Service metadata: name: mysql spec: ports: - port: 3306 clusterIP: None selector: app: mysql --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc labels: app: mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi --- apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: mysql spec: selector: matchLabels: app: mysql strategy: type: Recreate template: metadata: labels: app: mysql spec: containers: - image: debezium/example-mysql:0.10 name: mysql env: # Use secret in real usage - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-pass key: password ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pvc
But to my surprise, it would not start on GKE. Instead I would get this:
> [ERROR] --initialize specified but the data directory has files in it. Aborting.
What? Isn't Kubernetes meant to be the fabric on which we could easily port our workloads across different providers with no issues?
After a good amount of googling, it turns out that GKE Persistence Volume provider mounts the
/var/lib/mysql volume with a lost+found directory that is not empty. That does not happen on minikube. And mysql does not like that (not when you are not running with default my.cnf at least).
So to fix this, you need to pass
--ignore-db-dir=lost+found to the command line arguments of the container as in this snippet:
spec: containers: - image: debezium/example-mysql:0.10 args: - "--ignore-db-dir=lost+found" name: mysql
Since this took me a while to find, I thought it would be worth sharing for others that may bump into this.