You are not logged in.

#1 2020-08-04 21:10:00

corey.taylor
Member
From: Dresden
Registered: 2020-04-14
Posts: 27
Website

[SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

Hey guys,

SSH'd in to the new Lithium install I installed today from home and noticed that I had no SSD space left at all. There were a few huge logs in /var/log (e.g. messages, journal, etc.) of ~80Gb each which maxed out my 500Gb drive quicksmart. I tried opening them but even vim was barely running so just I just deleted them. Also noticed rsyslogd using a serious amount of resources and the system load was pretty high (3-4-ish):

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND          
27735 root      20   0  365092  91696   1768 S 201.1   0.0  27:53.83 rsyslogd 

Most of the RAM was in use too. To kill this, had to run the following:

systemctl stop syslog.socket rsyslog.service

I noticed a thread last year chatting about what sort of logging regime to use but I'm not good enough at Linux admin to know which choice would have been the best, nor whether this relates to my problem anyway.

I did run a resource-heavy thing today, opened about 190Gb of MD simulation trajectories which (almost) maxed out the RAM and did result in some swap usage. If its relevant, I let the installer set the swap space so on a 500Gb drive, it set the swap as 200Gb leaving 300Gb left for /. For years the rule of thumb was swap = twice the amount of RAM but is that a bit old hat? Any ideas whether this relates in any way?

Either way, does anyone know whether merely setting something like LOG_SIZE in /etc/default/rsyslog as one used to do with syslog will end the problem or is there some other thing I'm missing?

Cheers lads.

Last edited by corey.taylor (2020-08-05 07:44:06)

Offline

#2 2020-08-04 21:29:15

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,942

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

The rsyslogd service on modern Debian 10 (BL Lithium) is run by a systemd service file that contains ExecStart=/usr/sbin/rsyslogd -n -iNONE so I don't think a LOG_SIZE would work (unless there's an indirection between sysvinit-compat layers and systemd still, it's horrible, even Ubuntu 20.04 has this still built in). Because of the way rsyslog works with its internal queues you'd need logrotate to rotate/discard logs reliably (to send the SIGHUP to rsyslogd), or configure fixed size limits per queue in /etc/rsyslog.conf (much more arcane than systemd-journald)

You could instead create /var/log/journal to enable journald persistence (as described in the post you linked to) and configure then SystemMaxUse as described here https://wiki.archlinux.org/index.php/Sy … size_limit to set a fixed limit for your on-disk logs. After changing the config, reload the config by running sudo systemctl restart systemd-journald.service.

Then you'd stop, disable and mask rsyslogd to prevent it from ever starting again

sudo systemctl stop rsyslog.service
sudo systemctl disable rsyslog.service
sudo systemctl mask rsyslog.service

Journald will continue to capture your syslogs realiably, but within the fixed size limit you specified earlier. The difference is that now you won't have plaintext logs anymore but have to view logs using journalctl. Unless you see a clear advantage of plain text logs on your desktop, I think this is the cleanest solution to limiting log size strictly today.


Per aspera ad astra.

Offline

#3 2020-08-05 07:03:51

ohnonot
...again
Registered: 2015-09-29
Posts: 4,882
Website

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

200G of swap???
I think your installer went haywire there.
Used to be twice the physical RAM, but the more RAM you have the less you need that much. I have 8G and swap is also 8G.

PS: b = bit; I'm sure you meant B = byte? 8b=1B


BL quote proposals to this thread please.
my repos / my repos
---
Thank you for posting direct image links!

Offline

#4 2020-08-05 07:42:30

corey.taylor
Member
From: Dresden
Registered: 2020-04-14
Posts: 27
Website

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

Yeh I figured that would be far too much. I have ~200 gigaBytes of physical RAM (you're right - big B!) but as I said, I went with the install defaults in the first instance. When I can be arsed, I'll resize the swap space because I can't imagine a situation where I'd ever need that amount of swap space unless something has gone seriously wrong.

Thanks for the tips and info, guys. I restarted the journal service and masked rsyslog as well as setting a small SystemMaxSize (50Mb). Can't imagine a reason I'd really need plain text logs so let's see how things go but I'll mark the question as solved anyway.

Last edited by corey.taylor (2020-08-05 09:22:11)

Offline

#5 2020-08-05 12:19:39

rbh
Member
From: Sweden/Vasterbotten/Rusfors
Registered: 2016-08-11
Posts: 661

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

corey.taylor wrote:

Yeh I figured that would be far too much. I have ~200 gigaBytes of physical RAM (you're right - big B!) but as I said, I went with the install defaults in the first instance. When I can be arsed, I'll resize the swap space because

You could just as well disable the swap in /etc/fstab. Later on, you can delete the swap and instead create an smaller compressed ramdisk to swap against.

On my best equiped laptop with 12 GB ram, I have comented out the swap -partition in fstab. Never been close to getting little free RAM. If i ever would need swap, i can run in root terminal "swapon /dev/sdxy".

Last edited by rbh (2020-08-05 12:20:36)


// Regards rbh

Offline

#6 2020-08-05 18:03:35

ohnonot
...again
Registered: 2015-09-29
Posts: 4,882
Website

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

I think you should still figure out what swamps your logs.


BL quote proposals to this thread please.
my repos / my repos
---
Thank you for posting direct image links!

Offline

#7 2020-08-06 06:31:14

corey.taylor
Member
From: Dresden
Registered: 2020-04-14
Posts: 27
Website

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

Agreed. Wish I'd kept some of the output but yeah, even vim was struggling. As of now it all appears normal.

Maybe someone more knowledgeable can answer one thing, though. I've administered Debian systems before and the usual logs are there (Xorg, auth, kern, etc.) but I'd never seen a log just named messages or journal before. These were two of the big ones. Is this something new?

Offline

#8 2020-08-06 07:55:27

ohnonot
...again
Registered: 2015-09-29
Posts: 4,882
Website

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

corey.taylor wrote:

even vim was struggling

Use

less 

instead.

corey.taylor wrote:

I'd never seen a log just named messages or journal before. These were two of the big ones. Is this something new?

I have /var/log/messages as a log file.
/var/log/journal is a directory, it's where systemd stores its journal which can be accessed with

journalctl

I have no idea to what extent all logs are duplicated in the journal, but it is increasingly becoming the standard place to log & look for log messages.


BL quote proposals to this thread please.
my repos / my repos
---
Thank you for posting direct image links!

Offline

#9 2020-08-06 09:39:01

corey.taylor
Member
From: Dresden
Registered: 2020-04-14
Posts: 27
Website

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

Bah, sorry I meant Less. Eesh.

Cheers for the tip about logs too.

Offline

#10 2020-08-06 11:41:23

rbh
Member
From: Sweden/Vasterbotten/Rusfors
Registered: 2016-08-11
Posts: 661

Re: [SOLVED] Lithium: syslogd uses many resources, huge logs in /var/log

corey.taylor wrote:

I'd never seen a log just named messages or journal before. These were two of the big ones. Is this something new?

I can not remember being without /var/log/messages... (my first debian installation was with Debian 2.2 (Potato), 20 years ago. Ubuntu removed messegages and use only syslog some 10 years ago.


// Regards rbh

Offline

Board footer

Powered by FluxBB