You are not logged in.

#1 2016-10-26 08:07:50

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

[FIXED] journalctl: cannot view old boots

I've done what I thought was needed to view journalctl's records:

*) Added myself to the systemd-journal group.

*) Enabled journal archives with Storage=persistent in /etc/systemd/journald.conf

Now /var/log/journal is full of directories holding ~16MB each, but journalctl refuses to display them.

'journalctl' shows the current boot log OK (this is stored in memory, not disk).

'journalctl --disk-usage' reports "Journals take up 16.0M on disk." but that directory holds 700MB.

Here are some more trials (the current boot started at 09:43:53 JST, but there were two others before that today):

john@bunsen1:~$ journalctl --since yesterday
-- Logs begin at Wed 2016-10-26 09:43:53 JST, end at Wed 2016-10-26 16:39:02 JST. --
Oct 26 09:43:53 bunsen1 systemd-journal[196]: Runtime journal is using 8.0M (max allowed
Oct 26 09:43:53 bunsen1 systemd-journal[196]: Runtime journal is using 8.0M (max....
...
john@bunsen1:~$ journalctl --verify
PASS: /var/log/journal/d09c9b8618294949b795dacaa7a9babb/system.journal 
PASS: /var/log/journal/d09c9b8618294949b795dacaa7a9babb/user-1000.journal
## this is the newest directory in /var/log/journal
...
john@bunsen1:~$ journalctl -b d09c9b8618294949b795dacaa7a9babb
Failed to look up boot ID d09c9b8618294949b795dacaa7a9babb+0: Cannot assign requested address
## Same for any other ID
...
john@bunsen1:~$ journalctl --boot=-1
Failed to look up boot -1: Cannot assign requested address

It just seems unable to access those directories.
I wonder if some ACL attribute is missing?

john@bunsen1:~$ ls -l /var/log/journal/*/system*
-rw-r----- 1 root systemd-journal 8388608 Oct 12 18:16 /var/log/journal/0d1849318e334326bee8799026fa767f/system.journal
-rw-r----- 1 root systemd-journal 8388608 Sep 27 18:48 /var/log/journal/12cd71a3446c4100ad1e6df7679ca271/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct  5 18:48 /var/log/journal/152bce06150c4a0cb9457e8bcac88a77/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 13 18:33 /var/log/journal/1c4e6f402bd54dfea14143dcaa39a500/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 13 15:21 /var/log/journal/2300ddfebb1341908756bba469eead09/system.journal
...

Other people posting this command on the web have -rw-r-----+ in the first field, the '+' indicating that some extra attribute has been set. I don't know if the absence of that '+' in my case suggests anything?

Anyway, first off, can any other Debian Jessie users (with systemd ver. 215) report journalctl working OK for them, or not working also?

EDIT: @tknomanzr found that upgrading systemd from the backports fixed this issue. (systemd ver. 230 is available in jessie-backports)

Last edited by johnraff (2016-10-31 01:46:41)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#2 2016-10-26 10:27:12

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: [FIXED] journalctl: cannot view old boots

Can you view the old logs as root?

What is the output of:

groups

I will check this on my BL system later.


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#3 2016-10-26 12:54:56

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: [FIXED] journalctl: cannot view old boots

Head_on_a_Stick wrote:

I will check this on my BL system later.

My BL system will show all the old boots even with "Storage=auto" (the default setting) for a normal user in the systemd-journal group, /var/log/journal exists on this system but I may have created that myself.

In respect of the ACLs:

empty@TheLab:~$ ls -l /var/log/journal/*/system*
-rw-r-----+ 1 root root            16777216 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-0000000000000001-00053c5341e2d651.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-00000000000032d0-00053f50217bdf04.journal
-rw-r-----+ 1 root systemd-journal 16777216 Oct 26 14:43 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal
empty@TheLab:~$

I'm not sure what that extra "+" symbol is, best I could get is from `info ls`:

A file with any other combination of alternate access methods is
     marked with a ‘+’ character.

... which is rather cryptic hmm

I tried `rm -rf /var/log/journal` on my system and the directory was re-created at the next boot with full access for systemd-journal group members, perhaps try that?

Last edited by Head_on_a_Stick (2016-10-26 14:27:46)


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#4 2016-10-26 16:33:39

username
Member
Registered: 2016-02-13
Posts: 8

Re: [FIXED] journalctl: cannot view old boots

--file seems to nail it. (I think)

paul@debian:~$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR

paul@debian:~$ ls -l /var/log/journal/*/system* | head
-rw-r-----+ 1 root systemd-journal 4194304 Oct  7 21:15 /var/log/journal/09dd84fed83d436497e3699bd92756ea/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 19 07:20 /var/log/journal/1f0e0f01f70c4ceeba6cc70857628f9b/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct  9 18:36 /var/log/journal/35f2419b558e46d89f65aad7fe15f152/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 22 21:08 /var/log/journal/3ecdf16c82c6483a9ba5cf3630153502/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 12 20:25 /var/log/journal/4025794fc832451d8a0cb4b4d8d48fa6/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct  5 23:57 /var/log/journal/40eb35a60f654e5e963b0317bc742f19/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 14 17:55 /var/log/journal/4d62ca90dae7453892905a23c1e2a3ad/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 23 15:18 /var/log/journal/4d967415bb7449fe834b6709056ef9cd/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 17 22:27 /var/log/journal/5ff0a49ae1844cd1a3d81f6177804ba5/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 12 18:37 /var/log/journal/609efaa700a64bad8ab0baa6397d0577/system.journal

paul@debian:~$ journalctl --file /var/log/journal/4025794fc832451d8a0cb4b4d8d48fa6/system.journal 
-- Logs begin at Wed 2016-10-12 20:04:15 BST, end at Wed 2016-10-12 20:25:08 BST. --
Oct 12 20:04:15 debian systemd-journal[167]: Runtime journal is using 6.9M (max allowed 55.6M, trying to leave 83.5M free o
Oct 12 20:04:15 debian systemd-journal[167]: Runtime journal is using 6.9M (max allowed 55.6M, trying to leave 83.5M free o
Oct 12 20:04:15 debian kernel: Initializing cgroup subsys cpuset
Oct 12 20:04:15 debian kernel: Initializing cgroup subsys cpu
Oct 12 20:04:15 debian kernel: Initializing cgroup subsys cpuacct
Oct 12 20:04:15 debian kernel: Linux version 3.16.0-4-686-pae (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4

From a fedora forumhttp://forums.fedoraforum.org/showthread.php?t=305343

Hope that helps.

Offline

#5 2016-10-27 04:18:01

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

username wrote:

--file seems to nail it. (I think)
Hope that helps.

Yes, it did, thank you! smile
Using 'ls -ltr' puts the newest files at the bottom, so this is a usable workaround for checking the last couple of boots:

john@bunsen1:~$ ls -ltr /var/log/journal/*/system*
-rw-r----- 1 root systemd-journal 8388608 Sep 25 09:54 /var/log/journal/77e72c685d3a479faaee27ed729bea4a/system.journal
...
-rw-r----- 1 root systemd-journal 8388608 Oct 27 12:56 /var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/system.journal
john@bunsen1:~$ journalctl --file=/var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/system.journal
-- Logs begin at Thu 2016-10-27 10:25:51 JST, end at Thu 2016-10-27 12:56:26 JST. --
Oct 27 10:25:51 bunsen1 systemd-journal[197]: Runtime journal is using 8.0M (max allowed 76.5M, trying to leave 114.8M free of 75
Oct 27 10:25:51 bunsen1 systemd-journal[197]: Runtime journal is using 8.0M (max allowed 76.5M, trying to leave 114.8M free of 75
...

It would still be nice to be able to use the regular journalctl commands though...

Head_on_a_Stick wrote:

In respect of the ACLs:

empty@TheLab:~$ ls -l /var/log/journal/*/system*
-rw-r-----+ 1 root root            16777216 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-0000000000000001-00053c5341e2d651.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-00000000000032d0-00053f50217bdf04.journal
-rw-r-----+ 1 root systemd-journal 16777216 Oct 26 14:43 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal
empty@TheLab:~$

I'm not sure what that extra "+" symbol is, best I could get is from `info ls`:

A file with any other combination of alternate access methods is
     marked with a ‘+’ character.

... which is rather cryptic hmm

Yes. Other posts I've seen on the web also have that mysterious '+', which I lack. That could be the problem.

Also:

john@bunsen1:~$ journalctl --verify
PASS: /var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/system.journal                               
PASS: /var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/user-1000.journal 

It looks as if only the latest entry is considered usable. That was also the case yesterday, when the PASS file was different, and still present, but no longer passed.

I tried `rm -rf /var/log/journal` on my system and the directory was re-created at the next boot with full access for systemd-journal group members, perhaps try that?

I'll give that a go - well, mv not rm. yikes


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#6 2016-10-27 04:44:19

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

I'm afraid moving the journal directory didn't help.
This, after two boots:

john@bunsen1:~$ ls -ltr /var/log/journal/*/system*
-rw-r----- 1 root root            8388608 Oct 27 13:27 /var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 27 13:29 /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
john@bunsen1:~$ journalctl --verify
PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal 
PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal
john@bunsen1:~$ journalctl -b -1
Failed to look up boot -1: Cannot assign requested address

Note: the first files created are owned by root:root, not root:systemd-journal as they should be, and which happened with the second set.
After the first boot, 'journalctl --verify' showed /var/log/journal/937faef198c941b58a1f9d6d4082981f/user.journal as being passed, but not now.
The previous boot cannot be displayed, although 'journalct --file' still works.

And those '+' marks are still missing. neutral


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#7 2016-10-27 05:14:10

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

Looking at file attributes, e2fsprogs comes with lsattr:

john@bunsen1:~$ lsattr -R /var/log/journal
-------------e-- /var/log/journal/937faef198c941b58a1f9d6d4082981f

/var/log/journal/937faef198c941b58a1f9d6d4082981f:
-------------e-- /var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal
-------------e-- /var/log/journal/937faef198c941b58a1f9d6d4082981f/user-1000.journal

-------------e-- /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb

/var/log/journal/3ffd810c793f456c9f9a0107ebce54fb:
-------------e-- /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
-------------e-- /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal

And according to man chattr, " The  'e'  attribute  indicates  that the file is using extents for mapping the blocks on disk." OK...

There's also an attr package (repos) which provides getfattr:

john@bunsen1:~$ getfattr -dR /var/log/journal
getfattr: Removing leading '/' from absolute path names
# file: var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal
user.crtime_usec=0sIjKqHNE/BQA=

# file: var/log/journal/937faef198c941b58a1f9d6d4082981f/user-1000.journal
user.crtime_usec=0s0/7VHdE/BQA=

# file: var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
user.crtime_usec=0s/I2uK9E/BQA=

# file: var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal
user.crtime_usec=0sk3iNLdE/BQA=

About equally cryptic.

BUT
, if @Head_on_a_Stick or @username, either of you could run 'lsattr -R /var/log/journal', or, after installing attr, run 'getfattr -dR /var/log/journal', and post part of the output, we might be able to identify something that your files have got that mine haven't, accounting for that '+' in the output of ls.
Help much appreciated!

Last edited by johnraff (2016-10-27 05:15:35)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#8 2016-10-27 07:32:14

earlybird
ほやほや
Registered: 2015-12-16
Posts: 717
Website

Re: [FIXED] journalctl: cannot view old boots

Just forward the journal to a syslog daemon like rsyslog, problem solved...just my personal private workaround for all this journal insanity.

Offline

#9 2016-10-27 21:42:03

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: [FIXED] journalctl: cannot view old boots

Requested output:

empty@TheLab:~$ sudo lsattr -R /var/log/journal
---------------- /var/log/journal/d073d511f5ea4915955c41cdef8b8378

/var/log/journal/d073d511f5ea4915955c41cdef8b8378:
---------------- /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal
---------------- /var/log/journal/d073d511f5ea4915955c41cdef8b8378/user-1000.journal

No extants for me, apparently...

empty@TheLab:~$ sudo getfattr -dR /var/log/journal
getfattr: Removing leading '/' from absolute path names
# file: var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal
user.crtime_usec=0sWw8nV8U/BQA=

# file: var/log/journal/d073d511f5ea4915955c41cdef8b8378/user-1000.journal
user.crtime_usec=0sjxcrWMU/BQA=

“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#10 2016-10-27 22:30:53

Horizon_Brave
Operating System: Linux-Nettrix
Registered: 2015-10-18
Posts: 1,473

Re: [FIXED] journalctl: cannot view old boots

My journalctl functionality seems to work fine, hope this can help ya;

Below: No + sign

kingghidorah@kingghidorah:~$ ls -l /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system*
-rw-r----- 1 root root            8388608 Oct 25 17:05 /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system@00053fb6df8c99a3-e6fa2e05c728d252.journal~
-rw-r----- 1 root systemd-journal 8388608 Oct 27 18:17 /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system.journal
kingghidorah@kingghidorah:~$ 

Below: Attributes

sudo lsattr -R /var/log/journal/
[sudo] password for kingghidorah: 
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63

/var/log/journal/985e7e0edff6e6488183b8767b8d5b63:
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system@00053fb6df8c99a3-e6fa2e05c728d252.journal~
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/user-1000.journal
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system.journal

Below is my /etc/systemd/journald.conf

[Journal]
Storage=auto
Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
MaxLevelWall=emerg

"I have not failed, I have found 10,000 ways that will not work" -Edison

Offline

#11 2016-10-28 00:16:17

bra10n
Member
Registered: 2016-06-15
Posts: 22

Re: [FIXED] journalctl: cannot view old boots

johnraff wrote:

I'm afraid moving the journal directory didn't help.
This, after two boots:

john@bunsen1:~$ ls -ltr /var/log/journal/*/system*
-rw-r----- 1 root root            8388608 Oct 27 13:27 /var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 27 13:29 /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
john@bunsen1:~$ journalctl --verify
PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal 
PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal
john@bunsen1:~$ journalctl -b -1
Failed to look up boot -1: Cannot assign requested address

Note: the first files created are owned by root:root, not root:systemd-journal as they should be, and which happened with the second set.
After the first boot, 'journalctl --verify' showed /var/log/journal/937faef198c941b58a1f9d6d4082981f/user.journal as being passed, but not now.
The previous boot cannot be displayed, although 'journalct --file' still works.

And those '+' marks are still missing. neutral

Can't you change ownership of the original location to systemd.journal and add yourself to the systemd.journal group?

Offline

#12 2016-10-28 00:21:30

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

@earlybird thanks - if nothing else works I'll come back and ask you how to do that.

@HoaS hmm, so it looks as if attributes might be a distraction.

@H_B "My journalctl functionality seems to work fine" means you can read old journals OK with things like 'journalctl -b -1'?

An interesting difference your directory has from mine is this file: /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system@00053fb6df8c99a3-e6fa2e05c728d252.journal~
I've seen web references to these "system@something" files but just assumed they were from an earlier or later systemd version - I don't have any. (You are using systemd 215?)

You have different settings in /etc/systemd/journald.conf from mine:

[Journal]
Storage=auto
Compress=yes
ForwardToWall=yes
MaxLevelWall=emerg

Mine:

[Journal]
Storage=persistent

Did you add those settings yourself? Actually, I thought 'Storage=persistent' was necessary to keep old journals on disk.

@HoaS, @user do either of you have those system@... files in your journal directories?


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#13 2016-10-28 00:23:56

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

bra10n wrote:

Can't you change ownership of the original location to systemd.journal and add yourself to the systemd.journal group?

I'm already in the systemd.journal group (that's necessary to view any journals without sudo), and changing the group ownership on that one set of files made no difference, unfortunately.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#14 2016-10-28 00:34:25

bra10n
Member
Registered: 2016-06-15
Posts: 22

Re: [FIXED] journalctl: cannot view old boots

johnraff wrote:
bra10n wrote:

Can't you change ownership of the original location to systemd.journal and add yourself to the systemd.journal group?

I'm already in the systemd.journal group (that's necessary to view any journals without sudo), and changing the group ownership on that one set of files made no difference, unfortunately.

So could the situation be that these logs are being ditched after each log out/reboot? This might explain their *existence* but not their recovery after log out etc.

I'm wondering now if the directory needs any priveleges with regards to "persistence".

Or perhaps revert to HoaS's config with

Storage=auto

Last edited by bra10n (2016-10-28 00:40:19)

Offline

#15 2016-10-28 00:40:55

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

The old log files definitely exist, and can be viewed with 'journalctl --file /file/path' - but they can't be brought up with the usual journalctl commands for "previous boot" etc, so it's pretty kludgy.

I've already set 'Storage=persistent' - I think that's necessary to get old journals saved on disk at all.

But, it might be that some subtle setting somewhere is missing... it would be nice to find it...


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#16 2016-10-28 00:43:21

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

bra10n wrote:

Or perhaps revert to HoaS's config with

Storage=auto

That was H_B's config, but I had been under the impression "persistent" was necessary to keep old records on disk... I'll try switching to "auto" and see what happens.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#17 2016-10-28 00:51:55

bra10n
Member
Registered: 2016-06-15
Posts: 22

Re: [FIXED] journalctl: cannot view old boots

Here are the steps I always follow to get journaling up and running. Note: I have these saved in a file from a time when systemd still had a price-tag wink


mkdir /var/log/journal
chgrp systemd-journal /var/log/journal
chmod g+rwx /var/log/journal
usermod -a -G systemd-journal <your username>
systemctl reboot

I have these saved in a file for reference.

Here's a copy of my /etc/systemd/journal.config;

[Journal]
Storage=auto
Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
MaxLevelWall=emerg

Needless to say this works here. I am no means an expert mind, but I'm intriqued as to why you are having difficulties...

Edit: Sorry for the misquote above.

Last edited by bra10n (2016-10-28 00:53:05)

Offline

#18 2016-10-28 07:09:08

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: [FIXED] journalctl: cannot view old boots

johnraff wrote:

I've already set 'Storage=persistent' - I think that's necessary to get old journals saved on disk at all.

If /var/log/journal exists then the journal should be saved to disk even if "Storage=auto", at least according to journald.conf(5).

johnraff wrote:

do either of you have those system@... files in your journal directories?

Yes, see the output in this post:
https://forums.bunsenlabs.org/viewtopic … 360#p39360


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#19 2016-10-29 06:35:20

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

I've tried  "Storage=auto" but it seems to make no difference. I've just noticed that while that mysterious attribute "+" is missing on my */system.journal files, it is present on my */user-1000.journal files. Either way, 'journalctl --user -b -1' is just as useless as the system version. neutral

EDIT:

Head_on_a_Stick wrote:
johnraff wrote:

do either of you have those system@... files in your journal directories?

Yes, see the output in this post:
https://forums.bunsenlabs.org/viewtopic … 360#p39360

Ah! so you do. And I don't have any such files - maybe they perform some indexing function?

Still baffled as to why viewing of previous bootlogs seems to work OK for everyone else, but at this point I'm going to use a temporary workaround. I've written a couple of functions to put in ~/.bash_aliases that let me type 'journal' to see the last bootlog, which was what this was all about originally:

newest(){
    if [[ $1 = '-n'* ]]
    then
        local offset="${1#-n}"
        shift
    else
        local offset=0
    fi
    declare -A files=()
    for i in "$@"
    do
        files[$(stat -c %Y "$i")]="$i"
    done
    local list=($(sort -nr <(printf '%s\n' "${!files[@]}")))
    echo "${files[${list[$offset]}]}"
}

journal(){
    journalctl --file "$(newest -n1 /var/log/journal/*/system.journal)"
}

That "newest" function will output the newest file of any list it's given, so might be useful to other people -  if you give it -n1 it outputs the next newest file, and so on...

Last edited by johnraff (2016-10-29 06:37:18)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#20 2016-10-29 08:30:40

bra10n
Member
Registered: 2016-06-15
Posts: 22

Re: [FIXED] journalctl: cannot view old boots

johnraff wrote:

...changing the group ownership on that one set of files made no difference, unfortunately.

Last idea... you could check who owns 'journal'

Offline

#21 2016-10-30 03:46:06

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

^It looks OK to me:

john@bunsen1:~$ ls -ld /var/log/journal
drwxr-sr-x 7 root systemd-journal 4096 Oct 30 10:26 /var/log/journal

John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

#22 2016-10-30 07:03:29

bra10n
Member
Registered: 2016-06-15
Posts: 22

Re: [FIXED] journalctl: cannot view old boots

johnraff wrote:

^It looks OK to me:

john@bunsen1:~$ ls -ld /var/log/journal
drwxr-sr-x 7 root systemd-journal 4096 Oct 30 10:26 /var/log/journal

Here;

bra10n@Aspire-5750:~$ ls -ld /var/log/journal
drwxr-sr-x 3 root systemd-journal 4096 Oct 29 19:01 /var/log/journal

So if I'm correct, you have permissions to rwx and cannot read journal logs, while here I have wx only and everything works. Strange.

Offline

#23 2016-10-30 16:28:34

tknomanzr
BL Die Hard
From: Around the Bend
Registered: 2015-09-29
Posts: 1,029

Re: [FIXED] journalctl: cannot view old boots

I did some checking on this. I was having the same issues @johnraff, which backporting systemd appears to have resolved. Once I backported systemd, setting storage to persistent and ensuring I was part of systemd-journal gave me proper handling of multiple boot logs as shown below:

tknomanzr@wtfbox:~$ journalctl --list-boots
-1 ba9e4cae0fc14425bc35698f9e6c4d34 Sun 2016-10-30 11:20:03 CDT—Sun 2016-10-30 1
 0 f841f7b622754843b852ceac1f9be2cc Sun 2016-10-30 11:23:32 CDT—Sun 2016-10-30 1
tknomanzr@wtfbox:~$ journalctl -b -1
-- Logs begin at Sun 2016-10-30 11:20:03 CDT, end at Sun 2016-10-30 11:23:47 CDT
Oct 30 11:20:03 wtfbox systemd-journald[1714]: Runtime journal (/run/log/journal
Oct 30 11:20:03 wtfbox kernel: CPU0 microcode updated early to revision 0x38, da
Oct 30 11:20:03 wtfbox kernel: Initializing cgroup subsys cpuset
Oct 30 11:20:03 wtfbox kernel: Initializing cgroup subsys cpu
Oct 30 11:20:03 wtfbox kernel: Initializing cgroup subsys cpuacct
Oct 30 11:20:03 wtfbox kernel: Linux version 3.16.0-4-amd64 (debian-kernel@lists
Oct 30 11:20:03 wtfbox kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-a
Oct 30 11:20:03 wtfbox kernel: e820: BIOS-provided physical RAM map:
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000000009e800-0x000000000009
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000f
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000039cc
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x0000000039ccf000-0x000000003a83
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003a83e000-0x000000003acb
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003acbf000-0x000000003c08
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003c090000-0x000000003c54
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003c550000-0x000000003c55
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003c551000-0x000000003c5d
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003c5d7000-0x000000003d89
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003d898000-0x000000003dff
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x000000003dff9000-0x000000003dff
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x0000000040000000-0x000000004fff
Oct 30 11:20:03 wtfbox kernel: BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1
tknomanzr@wtfbox:~$ 

Offline

#24 2016-10-31 01:42:31

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,740
Website

Re: [FIXED] journalctl: cannot view old boots

@tknomanzr thanks for that.

john@bunsen1:~$ apt-cache policy systemd
systemd:
  Installed: 215-17+deb8u5
  Candidate: 215-17+deb8u5
  Version table:
     230-7~bpo8+2 0
        100 http://ftp.riken.jp/Linux/debian/debian/ jessie-backports/main amd64 Packages
 *** 215-17+deb8u5 0
        500 http://ftp.riken.jp/Linux/debian/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

So systemd version 230 is available in the backports. smile

Probably a pile of other unpublicized bugs will have been fixed in there, but now I've got a workaround for my immediate needs I'll stay with 215 for the moment - mainly because it would be another complicating factor when helping trying to help other BL users...

But for any other people with this issue, I'd suggest taking tk's advice and upgrading systemd to 230 from jessie-backports.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Online

Board footer

Powered by FluxBB