You are not logged in.

#1 2019-11-12 13:10:30

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,553

Lithium/Buster journal persistency

Debian ships /etc/systemd/journald.conf with the setting

[journal]
Storage=auto

and does not create /var/log/journal. Unless rsyslogd is installed and always running, this means that the systemd journal is not persistent; see the man page:

Storage=
           Controls where to store journal data. One of "volatile", "persistent", "auto" and
           "none". If "volatile", journal log data will be stored only in memory, i.e. below the
           /run/log/journal hierarchy (which is created if needed). If "persistent", data will be
           stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created
           if needed), with a fallback to /run/log/journal (which is created if needed), during
           early boot and if the disk is not writable.  "auto" is similar to "persistent" but the
           directory /var/log/journal is not created if needed, so that its existence controls
           where log data goes.  "none" turns off all storage, all log data received will be
           dropped. Forwarding to other targets, such as the console, the kernel log buffer, or a
           syslog socket will still work however. Defaults to "auto".

The mentioned /run/log/journal is cleared on every reboot, hence the journal is not persistent if /var/log/journal does not exist.

Journal by default forwards logs to rsyslogd which takes care of persistency. However, this also means that querying the journal using invocations like

journalctl -u $unit

will only show data since the last reboot which cripples the journal's usefulness for selective debugging.

We need to either ensure either that

a) rsyslogd is always installed and running as part of the BL base install. We accept that journalctl is crippled and loses history or

b) /var/log/journal is created by BL configuration, for example using /etc/tmpfiles.d. We want the journal to be not crippled. We could also get rid of rsyslogd if we consider the journal the first-class facility to search and view logs, or just leave rsyslogd globally disabled.

c) /var/log/journal is created by BL configuration and log data persists in both rsyslogd /var/log/*.log etc as well as journald. Users can have both.

I find the way (a) a broken & counterintuitive configuration since basically all the systemd facilities for viewing logs that go as far as systemctl status --full $unit output are kind of broken. You can say that you want to view all postfix logs (journalctl -u postfix) but you only get a subset and have to salvage the rest from /var/log/mail.* with grep. It's also a pitfall if people decide that rsyslogd is useless because they have systemd-journald taking care of logging already and disable it, ending up without persistent logs. That's not acceptable.

As far as I can tell, Ubuntu 18.04 server does (c). Ubuntu 16.04 does (a) as does Debian 9 + 10.

(b) is typical for systemd-only distros like Arch.

(c) Uses a bit more space and has 2 log daemons write to disk.

Any opinions?


At the end of the river the sundown beams

Offline

#2 2019-11-12 14:14:40

BLizgreat!
Resident Babbler - vll!
Registered: 2015-10-03
Posts: 1,217

Re: Lithium/Buster journal persistency

Agree with you, I do c. myself at the moment and actually likely to keep it this way, have to tweak my log rotate config a bit for rsyslog. Having both and hopefully in the case of journald with a sane value for how much disk it can take up makes the most sense to me. Even for desktop nixers, I mean in terms of the resources(memory +cpu) and disk space (+ disk i/o) or extra boot time used in having both enabled isn't all that much and a Bunsenlabs user can then use whatever tutes and how-to's about logging are available to anyone online. Should they ever take an interest in learning about logging.

Playing with this myself lately and have settled on the option you're listing for c/both. Someone never knows when a user may want to expand their know-how about this subject. Also if ever desired shouldn't be much of an issue to config whichever as they see fit, disable one or other etc. From what I understand, they actually get along well running together anyway.

Last edited by BLizgreat! (2019-11-12 14:17:04)

Offline

#3 2019-11-12 14:17:10

unklar
Member
Registered: 2015-10-31
Posts: 957

Re: Lithium/Buster journal persistency

I have enjoyed the advantages of the journal for several years by getting to know siduction.
That is, in the distributions where it is not/was not set up, I set it up myself. [1]

I greet your thread and would choose c).   smile


[1] please operate "en" above

Offline

#4 2019-11-12 14:22:36

tynman
Member
Registered: 2015-10-13
Posts: 86

Re: Lithium/Buster journal persistency

I would recommend option (c). Creating the /var/log/journal directory is in my list of things to do after installing Linux-with-systemd. IMHO it should be the default setup. I'm not aware of any downside other than disk space usage. (IIRC, systemd does take measures to prevent the syslog from taking over the hard drive.)

Offline

#5 2019-11-12 14:59:28

BLizgreat!
Resident Babbler - vll!
Registered: 2015-10-03
Posts: 1,217

Re: Lithium/Buster journal persistency

Yep more babble ( you peeps know me big_smile )

Reiterate: Choosing option c.
1. Again with both enabled any/all documentation online applies for users to utilize. Related to either. The amount of system resources rsyslog uses vs giving up all the readily available documentation it has really doesn't make much sense.
2. You cannot know the experience level of a particular user who starts using Bunsenlabs gnu/Linux, could be a complete noob or a very experienced veteran. People tend to develop methods and configs which work well for them and having rsyslog already setup might be nice. Though anyone with even gnu/Linux 101 level knowledge and/or who can operate a search engine can easily enough config things to be however they wish.
3. Just saying, in all the years I've been using gnu/Linux I can't honestly think of a time when logging played a major role in solving an issue. Could be a time or two where it gave me a point in the right direction but if so, can't recall such an event. big_smile Given the fact of #3 for me, sheesh, maybe I should just go ahead and nix both these suckers, lol. Ahhhh nah, not worth it, though of course still debating with myself what's the best approach to take and how best to config these things. Shrugs ... big_smile

Now if you all will excuse me, I have a post apocalyptic ... virus(zombie kinda deal)movie going on youtube. wink

PS, Am on a zombie kick lately. However a zombie apocalypse in fact would be a geeks worst nightmare. Not so much because of all the death, gore and zombies trying to eat you but because no internet !!! Noooooooooo, ahhhhhhh !!!!!! Ya know how much chit would be involved getting a comp up and running that could possibly have access to a nice chunk of data/media available on it? Just the bare essentials dude, porn0, games, movies-music and of course cute animal vids etc etc ?!?!?! It'd be a friggin nightmare. Forget this, am going out to wrestle with the zombies, screw it !!! hmm

Last edited by BLizgreat! (2019-11-12 15:26:19)

Offline

#6 2019-11-12 18:59:37

beaker
Member
Registered: 2016-03-06
Posts: 134

Re: Lithium/Buster journal persistency

b

Online

#7 2019-11-13 01:08:18

DeepDayze
Member
From: In Linux Land
Registered: 2017-05-28
Posts: 761

Re: Lithium/Buster journal persistency

What would be a good setting to use for syslogging if your rootfs is an SSD?


Real Men Use Linux

Offline

#8 2019-11-13 03:31:51

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,097
Website

Re: Lithium/Buster journal persistency

I was leaning towards (b) but (c) has had upvotes.
I think with modern hardware either of those would be OK as long as we:
1) Make sure we document how to adjust/disable them.
2) Check systemd's (and rsyslogd's?) settings for a disk usage limit.


John
--------------------
( a boring Japan blog, idle Twitterings and GitStuff )
In case you forget, the rules.

Offline

#9 2019-11-13 07:27:04

BLizgreat!
Resident Babbler - vll!
Registered: 2015-10-03
Posts: 1,217

Re: Lithium/Buster journal persistency

DeepDayze wrote:

What would be a good setting to use for syslogging if your rootfs is an SSD?

A good setting for that imo, is to set Brain=google-extensively as your configuration of choice. big_smile Messing around, though you are getting into outside usecases. Really considering how far SSD's have come, in terms of lifespan, how many writes they can take etc. Would just leave them pretty much the same as on a rust drive(in terms of logging I mean.) This can depend on the brand and quality of said SSD but overall it's not like these logs are doing massive amounts of writes. Mentioned on a 32b/Stretch install the current /var/log directory is only 55mbs, of course you can still if desired fiddle with logrotate to ascertain the number and frequency of backups made and kept.

End of the dy and bottom-line in my view, it's gnu/Linux, practically infinite ways to config, so each end-user needs to optimize for their preferences and spec setup. All well documented to hades and gone fellow nixers. big_smile

PS, Twoion's obviously right though, journalctl is pretty much crippled without some degree of persistent logging enabled and it's also here to stay regardless. I'm just going to fiddle around with how much the thing bothers to keep around, thinking about 50-100mbs worth. Don't want the info from 400 boot's ago sitting on my massive 250gb hdd !!! tongue

Last edited by BLizgreat! (2019-11-13 08:43:18)

Offline

#10 2019-11-13 08:06:09

BLizgreat!
Resident Babbler - vll!
Registered: 2015-10-03
Posts: 1,217

Re: Lithium/Buster journal persistency

WARNING now you're stepping into the twilight-zone post ! smile

Though this is totally doable and is done by many the (network)server admins, they have however many boxes they're overseeing send logging data to a given central-server. So that it's all in one place, should they need to access it. All automated and likey just using crontabs, scripts ... rsync etc but anyways ...

If someone ( a desktop nixer) were horrendously anal about such tiny disk i/o and logging on a system. Could also easily enough be encrypted and sent out to be stored somewhere online. Google drive or whatever else. Thus would never even touch their drive. Not interested in this myself, so not going to bother checking out all the tutes/docs and how-to's related to it on the webz or the specifics of what'd be involved. Still again ... it's gnu/Nix, if you can imagine doing it on a computer, it's probably easy enough to do if someone investigates and researches. LOL .. some geek, 20yrs ago already thought of it, created software and methods for doing it and slapped it up online somewhere. big_smile


Reiteration2: Still going with keeping both. Never know what you'll want to do or dork with tomorrow. Are plenty of tools to monitor a given prog/processes disk i/o activities, may or not be watching what journalctl is doing behind the scenes. Really doubt it, as am just leaning towards giving it a sane amount of disk to use and calling it good.

Leaving twilight-zone, also Off-topicness, sowwie but couldn't resist babbling fellows. big_smile

PS, EDIT: Though may in fact have to do some more investigating about the journald thingy. Logs when weren't persistent = 8mbs, then set it up (added my user to the relevant group, created the log directory in the right place )and gave it 75mbs of disk to use. Quickly afterwards it jumped to 16mbs. Just checked "journalctl --disk-usage" and it's at 41.3mbs now. Ah oh well, still thinking 50-100mbs is fine and call it good. By default the thing is set to sync logs every 5mins. Can also be controlled but would just mean they'd be sitting in RAM until committed to disk at xyz-time-interval, shrugs.

Last edited by BLizgreat! (2019-11-13 08:17:31)

Offline

#11 2019-11-13 13:04:01

clusterF
Member
Registered: 2019-05-07
Posts: 322

Re: Lithium/Buster journal persistency

I would say c). Always good to have data redundancy.


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#12 2019-11-14 21:16:15

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,553

Re: Lithium/Buster journal persistency

I'll make an implementation that supports both (b) and (c) using systemd presets. We could have a switch in bl-welcome for example.

systemctl preset rsyslog.service # apply the BL preset (rsyslog disabled)
systemctl enable rsyslog.service # user can do whatever & override preset

At the end of the river the sundown beams

Offline

#13 2019-11-15 01:28:22

DeepDayze
Member
From: In Linux Land
Registered: 2017-05-28
Posts: 761

Re: Lithium/Buster journal persistency

twoion wrote:

I'll make an implementation that supports both (b) and (c) using systemd presets. We could have a switch in bl-welcome for example.

systemctl preset rsyslog.service # apply the BL preset (rsyslog disabled)
systemctl enable rsyslog.service # user can do whatever & override preset

That sounds like a good idea in say the advanced options within the bl-welcome script for advanced  tweaks and such.


Real Men Use Linux

Offline

#14 2019-11-15 03:47:12

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,097
Website

Re: Lithium/Buster journal persistency

There's a "system tweaks" page early in the welcome script sequence where it could be added. As long as we make it easy for new users to stay with the default.


John
--------------------
( a boring Japan blog, idle Twitterings and GitStuff )
In case you forget, the rules.

Offline

#15 2019-11-15 07:20:01

ohnonot
...again
Registered: 2015-09-29
Posts: 4,168
Website

Re: Lithium/Buster journal persistency

Redundancy (both storage and running processes) for a "lightweight" distro?
Storing duplicate logs in places that less and less people are looking for because more and more articles recommend systemd commands?

I'm for b) - and not bothering the new user with these sort of details.

Btw it is possible to view systemd journal files without systemd.

Last edited by ohnonot (2019-11-15 07:20:45)

Offline

#16 2019-11-15 14:02:19

clusterF
Member
Registered: 2019-05-07
Posts: 322

Re: Lithium/Buster journal persistency

ohnonot wrote:

Btw it is possible to view systemd journal files without systemd.

Yes systemd talons are quite deep now.


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#17 2019-11-15 22:41:27

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,553

Re: Lithium/Buster journal persistency

ohnonot wrote:

Btw it is possible to view systemd journal files without systemd.

Of course, you just need to implement a filter for the format: https://www.freedesktop.org/wiki/Softwa … nal-files/. An effort but simple and once written it works.

You can also use `strings` on the files but the output is usually not very useful.


At the end of the river the sundown beams

Offline

#18 2019-11-16 04:50:22

BLizgreat!
Resident Babbler - vll!
Registered: 2015-10-03
Posts: 1,217

Re: Lithium/Buster journal persistency

Someone definitley needs to read the man page and google on this puppy. Noticed it'd already exceeded the logging limit I'd set, as per very brief research into it. Checked "journalctl --disk-usage" ... 90+mbs, hmmmm, I'd set a 75mb limit in the file which is supposed to control this ? Errrrr, more googling and reading of man-page. Ok ... "journalctl --vacuum-size=50M" which is what it did, shrunk the thing down to 50mbs. Checked again ... exceeding the limit I'd set again. Though didn't invest much effort, looks like I'm going to have to. At least when I get around to it. In ways starting to regret enabling the thing with persistence at this point. Mentioned the entire /var/log directory with syslog, which includes all the compressed backups the thing keeps is all of 55mbs. Though journald is supposed to advance logging and keep data about things rsyslog doesn't etc blahblahblah.

Supposedly according to the docs I've seen, by default journald will use 10% of the size of the system partition it's installed on, so if / is on a 30gb partition, journald will grow up to 3gbs worth of logging. Though default, never more than 4gbs, so even if / is 50gbs, still only 4gbs worth kept. That's imo ... ridiculous, who the hell needs 4gbs of logs kept ??!?!?! Even server and network admins don't and could/will easily implement better ways to handle it, rather than journald's default crap. Ah oh well ... will get around to resolving it. For time being will just periodically check the damn thing and delete as needed until it's reconfigured. FOR ME, 50-100mbs of logs seems good enough.

Offline

#19 2019-11-16 04:59:56

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,097
Website

Re: Lithium/Buster journal persistency

twoion wrote:
ohnonot wrote:

Btw it is possible to view systemd journal files without systemd.

Of course, you just need to implement a filter for the format: https://www.freedesktop.org/wiki/Softwa … nal-files/. An effort but simple and once written it works.

Has no-one written and packaged a utility to do this yet?


John
--------------------
( a boring Japan blog, idle Twitterings and GitStuff )
In case you forget, the rules.

Offline

#20 2019-11-17 17:52:26

ohnonot
...again
Registered: 2015-09-29
Posts: 4,168
Website

Re: Lithium/Buster journal persistency

johnraff wrote:

Has no-one written and packaged a utility to do this yet?

It's really just

strings /file/in/question

and then grep for whatever (usually -i message).
Not sure why twoion thought that's difficult. Of course it's slightly more difficult than grepping a text file, but then I'm sure there was a reason why a different file format was chosen - maybe because it takes less space on disk.

Offline

#21 2019-11-17 18:10:57

BLizgreat!
Resident Babbler - vll!
Registered: 2015-10-03
Posts: 1,217

Re: Lithium/Buster journal persistency

^If talking about why kept in binary vs text. That's so it can be easily converted into whatever format whoever wants supposedly. Makes it more portable or versatile is what I've seen people saying. Still haven't really gotten around to dorking with this thing(journald), yep ... just checked and still exceeding the space limit I set for the thing to use. Though that's not going to automagically fix itself, until said dorking occurs. big_smile

At this point, considering off-loading the thing and reverting back to rsyslog, likely devote a lil more time cause it's supposed to provide much logging on things syslog doesn't. Though again, really can't recall when logs played much of a role in solving a problem for me anyway. Yep ... will poke at journald and google some more but if it keeps aggravating me, it's going to get flushed down the crapper.

Last edited by BLizgreat! (2019-11-17 18:11:50)

Offline

Board footer

Powered by FluxBB