Skip to content

Tidbits

Running ssh-agent as a systemd user service.

gnome-keyring and running i3 might sometimes make ed25419 keys not work so it might be useful to run an ssh-agent. And this is one way to do it.

This is a systemd user unit file that when enabled will run when you login to the system as your user.

Put this file in: ~/.config/systemd/user/ssh-agent.service

[Unit]
Description=OpenSSH private key agent
IgnoreOnIsolate=true

[Service]
Type=forking
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -a $SSH_AUTH_SOCK
ExecStartPost=/usr/bin/systemctl --user set-environment SSH_AUTH_SOCK=${SSH_AUTH_SOCK}

[Install]
WantedBy=default.target

Enable this unit file with systemctl --user enable ssh-agent.service You can also start it with: systemctl --user start ssh-agent.service

To use this unit file in all your shells you need this in your .bashrc or .profile

eval $(systemctl --user show-environment | grep SSH_AUTH_SOCK)
export SSH_AUTH_SOCK

And now you can use ssh-add to add your keys to this agent.

MariaDB Galera Cluster SST Config

When you are trying to rejoin or add a new node to a MariaDB Galera Cluster you might run in to problem that the node really don't want to join the cluster.

The log is usally found under: /var/log/mysql/mysql_error.log

The log on the joining node look something like this:

[ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
[ERROR] WSREP: SST request callback failed. This is unrecoverable, restart required.
[ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync - role 'joiner' - address '192.168.10.2' - datadir '/srv/data/mysql/' - parent '2495' - binlog '/var/log/mysql/mysql-bin' - binlog-index '/var/log/mysql/mysql-bin.index' - mysqld-args - wsrep_start_position=00000000–0000–0000–0000–000000000000:-1
[ERROR] WSREP: Process completed with error: wsrep_sst_rsync - role 'joiner' - address '192.168.10.2' - datadir '/srv/data/mysql/' - parent '2495' - binlog '/var/log/mysql/mysql-bin' - binlog-index '/var/log/mysql/mysql-bin.index' - mysqld-args - wsrep_start_position=00000000–0000–0000–0000–000000000000:-1: 1 (Operation not permitted)
[ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
SST stands for "State Snapshot Transfers" and this is the method used to send data from the donor node to the joiner node. There is different methods you can use for SST and the default one is rsync.
This error is when SST fails, and more info can be found on the donor side.
[ERROR] WSREP: Failed to read from: wsrep_sst_rsync - role 'donor' - address '192.168.10.2:4444/rsync_sst' - socket '/run/mysqld/mysqld.sock' - datadir '/srv/data/mysql/' - gtid '0c4ad108-b55b-11ea-8507-f36fd73d2301:5593775' - gtid-domain-id '0' - binlog '/var/log/mysql/mysql-bin' - binlog-index '/var/log/mysql/mysql-bin.index' - mysqld-args - wsrep_start_position=0c4ad108-b55b-11ea-8507-f36fd73d2301:2595925

This indicated that the donor node failed to establish a connection on the designated port somehow. It can be for many reasons, SELinux might prevent it, there can be firewalls, etc. Another reason might be that rsync timesout, since it is not built to transfer large amounts of data. So if this isnt done in 10 seconds (wich might be the case if you are joining a node on a running cluster). The solution I used when I ran into this problem was to switch to mariabackup instead of rsync for the SST.

You need to add the package: mariadb-backup

This is done with a few different changes in configuration. Mysql configuration has sections like [mysqld] [mariadb] etc. That is different parts of the configuration. You can have special configuration options for JUST mariadb-10.3 in [mariadb-10.3] etc.

Joiner:

My configuration resides in /etc/mysql/mariadb.conf.d/20-galera.conf and 50-server.conf

This is default for Debian10 and upstream MariaDB-packages. #20-galera.conf 

[mysqld]
wsrep_sst_method="mariabackup"
wsrep_sst_donor="node1.mariadb.cluster" #This is the name of the node, as mariadb names it.

#50-server.conf

[mariadb]
wsrep_sst_auth = mariabackup:vYej4BB0cpMn # This is a user:password (This random string is generated for this text)

In the running donor you have to add this user: 

CREATE USER 'mariabackup'@'localhost' IDENTIFIED BY 'vYej4BB0cpMn';
GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'mariabackup'@'localhost';
You can also add the wsrep_sst_auth to the running databases with this commmand.

SET GLOBAL wsrep_sst_auth = 'mariabackup:vYej4BB0cpMn';

Now you can try to restart your joining node and see if the SST method mariaback up works instead.

Logstash grok patterns for sssd

The logfiles for sssd has its own format, and is unable to send the logfiles native to a syslogservers. Either you have to install filebeat on each server, or you use the syslog-daemon probably already running and does not add another dependency to your systems. I chose the latter. It has four distinct parts.

  1. Rsyslog config
  2. Logstash syslog pre-match
  3. Detailed sssd match if block
  4. grok patterns.

I have put up a small github project with all the parts I made for sssd logs and enrichment for logstash and to get them into an ELK Stack Github repo