#1 2022-04-08 00:58:41

schwim

Member

From: Coastal VA, Murica

Registered: 2015-09-29

Posts: 342

Website

apt-get upgrade, mariadb will no longer start

Hello there everyone!

I've got an issue with mariadb after and upgrade, it fails to start with generic errors:

root@adolf:~# mysql -uroot
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
root@adolf:~# systemctl enable mariadb.service
root@adolf:~# mysql -uroot
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
root@adolf:~# systemctl restart mariadb
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

root@adolf:~# systemctl status mariadb
● mariadb.service - MariaDB 10.3.34 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mysql.service.d
└─limits.conf
Active: failed (Result: exit-code) since Thu 2022-04-07 17:40:58 EDT; 46min ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 4191 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 4192 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 4193 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && s
Process: 4238 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Main PID: 4238 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"

Apr 07 17:40:57 adolf systemd[1]: Starting MariaDB 10.3.34 database server...
Apr 07 17:40:57 adolf mysqld[4238]: 2022-04-07 17:40:57 0 [Note] /usr/sbin/mysqld (mysqld 10.3.34-MariaDB-0+deb10u1) starting as process 4238 ...
Apr 07 17:40:58 adolf systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Apr 07 17:40:58 adolf systemd[1]: mariadb.service: Failed with result 'exit-code'.
Apr 07 17:40:58 adolf systemd[1]: Failed to start MariaDB 10.3.34 database server.
lines 1-19/19 (END)

2022-04-07 18:32:11 0 [Note] InnoDB: Using Linux native AIO
2022-04-07 18:32:11 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-04-07 18:32:11 0 [Note] InnoDB: Uses event mutexes
2022-04-07 18:32:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-04-07 18:32:11 0 [Note] InnoDB: Number of pools: 1
2022-04-07 18:32:11 0 [Note] InnoDB: Using SSE2 crc32 instructions
2022-04-07 18:32:11 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2022-04-07 18:32:11 0 [Note] InnoDB: Completed initialization of buffer pool
2022-04-07 18:32:11 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-04-07 18:32:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=31617857011
2022-04-07 18:32:11 0 [ERROR] InnoDB: Missing MLOG_CHECKPOINT at 31617857011 between the checkpoint 31617857011 and the end 31617857036.
2022-04-07 18:32:11 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2022-04-07 18:32:12 0 [Note] InnoDB: Starting shutdown...
2022-04-07 18:32:12 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-04-07 18:32:12 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-04-07 18:32:12 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-04-07 18:32:12 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-04-07 18:32:12 0 [ERROR] Aborting

root@adolf:~#

relevant tail of journalctl -xe:

Apr 07 18:34:12 adolf dovecot[486]: auth-worker(14513): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL se
Apr 07 18:34:12 adolf dovecot[486]: auth-worker(14513): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL se
Apr 07 18:34:13 adolf postfix/submission/smtpd[8385]: warning: unknown[141.98.10.82]: SASL LOGIN authentication failed: Connection lost to authentication
Apr 07 18:34:13 adolf postfix/submission/smtpd[8385]: disconnect from unknown[141.98.10.82] ehlo=2 starttls=1 auth=0/1 quit=1 commands=4/5
Apr 07 18:34:21 adolf CRON[14493]: pam_unix(cron:session): session closed for user root

I've done a ton of Googling and I just don't seem to have much to go on other than

Process: 4238 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)

When I search that, I find a lot of results relating to corrupt log files or permission database but I'm afraid to backup the mysql dir and reinitiate it, for fear of breaking my dbs and users.  Folder contents found here:

Any thoughts on what my next step should be would be most welcome.  Thanks for your time!

---

Schw.im! A social site with an identity crisis.

Bearded_Blunder

Dodging A Bullet

From: Seat: seat0; vc7

Registered: 2015-09-29

Posts: 1,146

Re: apt-get upgrade, mariadb will no longer start

I'm afraid I can't help much with the issue of it not starting, all my database stuff is still MySQL on Windows boxen, & I'm certain your ability to search the issue is as good as mine.

However:

When I search that, I find a lot of results relating to corrupt log files or permission database but I'm afraid to backup the mysql dir and reinitiate it, for fear of breaking my dbs and users.

Since the database isn't starting, backing up is as simple as making a copy of the folder containing the database files, they're not in use & so not changing.

You can then try what you need & reverting to where you are now is as simple as stopping the database (if it's running but things are corrupted) and putting the copy back in place.

---

Blessed is he who expecteth nothing, for he shall not be disappointed...
If there's an obscure or silly way to break it, but you don't know what.. Just ask me

schwim

Member

From: Coastal VA, Murica

Registered: 2015-09-29

Posts: 342

Website

Re: apt-get upgrade, mariadb will no longer start

Hi there Blunder and thank you very much for your thoughts.

I've progressed to the point that I'm able to start mysql in recovery_mode=6.  I tried running a mysql check on all dbs but they all return as ok.  https://pastebin.com/kc109XB2

I'm continuing to Google to figure out where I go next.  It's a production machine so this is quite the pickle I've gotten myself in.

---

Schw.im! A social site with an identity crisis.

DeepDayze

Like sands through an hourglass...

From: In Linux Land

Registered: 2017-05-28

Posts: 1,901

Re: apt-get upgrade, mariadb will no longer start

Have you tried to roll back your system to before you did that apt upgrade?

---

Real Men Use Linux

schwim

Member

From: Coastal VA, Murica

Registered: 2015-09-29

Posts: 342

Website

Re: apt-get upgrade, mariadb will no longer start

Have you tried to roll back your system to before you did that apt upgrade?

Not yet.  I'm trying to sort this without going to that length.  I've just not searched the right phrase yet to have an idea of a next step.

---

Schw.im! A social site with an identity crisis.

nobody

The Great

Registered: 2015-08-10

Posts: 3,655

Re: apt-get upgrade, mariadb will no longer start

If you're using the galera cluster extension (wsrep) with MariaDB/MySQL you need to look at the logs the SQL cluster is dropping in /var/log. You have to bootstrap and also fix a galera cluster that has failed somehow using specific patterns/settings, see for example <https://galeracluster.com/library/docum … overy.html>. Partial logs won't help here.