WireGuard Mesh VPN Encryption for MariaDB Galera Cluster

This tutorial is going to show you how to build a WireGuard mesh VPN and use it to encrypt replication traffic in MariaDB Galera cluster.

Prerequisites

This is not a step-by-step tutorial. To use the knowledge in this article, it’s assumed that you have a basic understanding of how to set up WireGuard VPN and MariaDB Galera Cluster.

WireGuard Mesh VPN Encryption for MariaDB Galera Cluster

What’s a Mesh VPN?

In computer networking:

  • A mesh topology is where each node in the network has a direct connection to every other node.
  • A star topology is where a central node controls the flow of connection. Nodes in the network can’t connect to each other directly but have to communicate through the central node. A typical example is a home private network.

The most common VPNs in the market use the star topology. Each VPN client has to connect to a central VPN server. And WireGuard can be set up in this way.

A mesh VPN has no central server. Every node in the mesh VPN can connect to each other directly, so there’s no single point of failure. This is also known as peer-to-peer VPN, or point-to-point VPN (not to be confused with the PPTP VPN protocol). You can use it to build a distributed secure network for your servers running in the cloud. WireGuard is an open-source ultra-fast VPN protocol that can be used to build a full-mesh network.

WireGuard is super lightweight and currently doesn’t implement UDP hole punching. If you want two nodes to connect to each other with WireGuard, one of them has to listen on a public IP address to accept requests. So its mesh networking capability isn’t comparable to Nebula or ZeroTier, which allows you to build a full-mesh VPN for nodes behind NAT devices. However, the peer-to-peer feature in WireGuard works really well for servers running in data centers and it’s very easy to set up.

What’s MariaDB Galera Cluster?

Galera Cluster is a synchronous multi-master cluster for MySQL, MariaDB, and Percona database servers to implement high-availability data redundancy. A multi-master cluster allows read and writes to any node in the cluster. Data modifications on any node are replicated to all other nodes.

Why Use WirdGuard Mesh VPN with MariaDB Galera Cluster

Because the built-in encryption mechanism in MariaDB Galera cluster isn’t robust for production environment.

In 2019, I wrote a tutorial explaining how to encrypt MariaDB Galera replication traffic on the public Internet. However, the TLS settings for Galera cluster aren’t very stable. I have seen a parameter changed back and forth.

The encrypt variable for snapshot state transfer (SST) has 5 possible values: 0, 1, 2, 3, 4. 0 means encryption is disabled.

  • In 2019, the encryption mode for the mariabackup SST method should be set to 4. 1, 2 and 3 didn’t work.
  • Now in 2021, the MariaDB documentation says that you should use mode 2 or 3 for the mariabackup SST method.

I didn’t expect this change and it caused the following TLS connection error when I trying to join a new node to the cluster.

[ERROR] WSREP: client_handshake error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

encryption for mariadb galera cluster

It’s not easy to debug this error. And I still don’t know what’s wrong with my configurations. Whatsmore, changing the TLS settings requires restarting the entire cluster. So I gave up and started looking for alternative solutions. It occurred to me that I can use VPN to encrypt traffic and WireGuard allows me to build a peer-to-peer VPN, which is very suitable for use with the MariaDB Galera cluster.

WireGuard Mesh VPN Setup

It’s very easy to set up a mesh VPN with WireGuard. You simply need to list every peer in your WireGuard configuration file. I use /etc/wireguard/mesh.conf as the config file.

[Interface]
Address = 172.16.0.1/24
ListenPort = 51820
PrivateKey = RofXta1Y6D1mSP+KH35SUMymVWxffSZl5KQ1CyhBAlM=

[Peer]
PublicKey = ulkmsYKuoGf2G/5MGUZSrb8yBW1b+07zqzlnuP1A8MY=
AllowedIPs = 172.16.0.2/32
Endpoint = IP_Address_of_Node_2:51820
PersistentKeepalive = 25

[Peer]
PublicKey = d47QT5s5Rh75yQSvJbgzdTdma0qvin14ZvWjGx9nr0U=
AllowedIPs = 172.16.0.3/32
Endpoint = IP_Address_of_Node_3:51820
PersistentKeepalive = 25

My Galera cluster has 3 nodes, so I use also 3 nodes in my WireGuard configuration file. The private IP range 172.16.0.0/24 is used for the private network. It’s important that you add a ListenPort directive in the [Interface] section and an Endpoint directive in each [Peer] section. If you have multiple configuration files under /etc/wireguard/ directory, then only one of the config files should contain a ListenPort directive, because only one interface can bind to the 51820 UDP port.

Once you have configured all your WireGuard nodes, start the WireGuard interface on each node.

sudo systemctl start wg-quick@mesh.service

And enable auto-start at boot time.

sudo systemctl enable wg-quick@mesh.service

You should now be able to ping each VPN private IP address. If you can’t ping, check if your WireGuard is listening on UDP port 51820.

sudo ss -lnpu

Also, make sure you configure firewall to allow each node to access UDP port 51820.

Galera Cluster Setup

You simply need to delete the TLS encryption settings in your Galera configuration file. My full Galera cluster configurations is as follows.

[galera]
# Mandatory settings
wsrep_on                 = ON
wsrep_provider           = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_name       = "MariaDB Galera Cluster"
wsrep_cluster_address    = "gcomm://172.16.0.1,172.16.0.2,172.16.0.3"
binlog_format            = row
default_storage_engine   = InnoDB
innodb_autoinc_lock_mode = 2
innodb_force_primary_key = 1
innodb_doublewrite       = 1

# Allow server to accept connections on all interfaces.
bind-address = 0.0.0.0

# By default, MariaDB error logs are sent to journald, which can be hard to digest sometimes. 
# The following line will save error messages to a plain file. 
log_error = /var/log/mysql/error.log

# Optional settings
wsrep_slave_threads            = 4
innodb_flush_log_at_trx_commit = 0
wsrep_node_name                = node1
wsrep_node_address             = "172.16.0.1"

wsrep_provider_options="gcache.size = 1G; gcache.recover = yes;"

#SST method
wsrep_sst_method = mariabackup
wsrep_sst_auth = mariabackup:secret_password

[sst]
encrypt=0

Obviously, you need to change the wsrep_node_name and wsrep_node_address for each node in the Galera cluster. Also, make sure you configure firewall to allow traffic from the 172.16.0.0/24 network. If you use the UFW firewall on Ubuntu, then execute the following command on each node.

sudo ufw insert 1 allow in from 172.16.0.0/24

Wrapping Up

I hope this tutorial helped you encrypt replication traffic in MariaDB Galera cluster with WireGuard mesh VPN.

Rate this tutorial
[Total: 2 Average: 5]

9 Responses to “WireGuard Mesh VPN Encryption for MariaDB Galera Cluster

  • Thx for you´re use-&helpfull Tutorials!

  • Jalal Hajigholamali
    3 years ago

    Hi,
    Thanks a lot,
    nice article..

  • gopal tathe
    3 years ago

    Hi,
    Can you please explain which keys need to use in the public and private keys? I have three nodes, in that case will I need 3 different config files?

    • Xiao Guoan (Admin)
      3 years ago

      Each node needs only one pair of public/private keys, and only one configuration file. On each node, list every other peer in the cluster.

  • Thanks for the tutorial. Just to clarify: 172.16.0.0/24 is the Wireguard range? And for the Endpoint you need to use the existing Private IP of the Node? Thanks

    • Xiao Guoan (Admin)
      3 years ago

      You can use any private IP ranges for WireGuard VPN

      10.0. 0.0 ~ 10.255. 255.255
      172.16. 0.0 ~ 172.31. 255.255
      192.168. 0.0 ~ 192.168. 255.255

      as long as it’s not already used by another network interface on your server.

      The endpoint IP address in WireGuard configuration is the public IP address of other peers.

  • Jens T.
    2 years ago

    I set up my Galera cluster exactly like this. The wireguard interfaces have private IP addresses. The Galera wsrep_node_addresses have the same private IP addresses set. The cluster works. However, I have the problem that from time to time packets arrive on the external interface (eth0) at default Galera port 4567 from my cluster nodes. This should not be the case as Galera should only communicate via the wireguard interfaces. Have you observed this behavior and do you have a solution?

    • Xiao Guoan (Admin)
      2 years ago

      No. I don’t have this problem on my server. To fix it, I would simply set up a firewall rule to block Galera traffic on the eth0 interface.

  • Hi sorry for asking but theoreticly I can now install wp or a forum on those nodes an connect them to the nodes.
    node1 to wp1 ….
    so that I can use is a a geolocation one in nl, one in de and one in US and the db youd be synced but
    I have still a DE IP, NL IP and a US IP for geodns.

    thanks in advanced

Leave a Comment

  • Comments with links are moderated by admin before published.
  • Your email address will not be published.
  • Use <pre> ... </pre> HTML tag to quote the output from your terminal/console.
  • Please use the community (https://community.linuxbabe.com) for questions unrelated to this article.
  • I don't have time to answer every question. Making a donation would incentivize me to spend more time answering questions.

The maximum upload file size: 2 MB. You can upload: image. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here