You are not logged in.
I've done what I thought was needed to view journalctl's records:
*) Added myself to the systemd-journal group.
*) Enabled journal archives with Storage=persistent in /etc/systemd/journald.conf
Now /var/log/journal is full of directories holding ~16MB each, but journalctl refuses to display them.
'journalctl' shows the current boot log OK (this is stored in memory, not disk).
'journalctl --disk-usage' reports "Journals take up 16.0M on disk." but that directory holds 700MB.
Here are some more trials (the current boot started at 09:43:53 JST, but there were two others before that today):
john@bunsen1:~$ journalctl --since yesterday
-- Logs begin at Wed 2016-10-26 09:43:53 JST, end at Wed 2016-10-26 16:39:02 JST. --
Oct 26 09:43:53 bunsen1 systemd-journal[196]: Runtime journal is using 8.0M (max allowed
Oct 26 09:43:53 bunsen1 systemd-journal[196]: Runtime journal is using 8.0M (max....
...
john@bunsen1:~$ journalctl --verify
PASS: /var/log/journal/d09c9b8618294949b795dacaa7a9babb/system.journal
PASS: /var/log/journal/d09c9b8618294949b795dacaa7a9babb/user-1000.journal
## this is the newest directory in /var/log/journal
...
john@bunsen1:~$ journalctl -b d09c9b8618294949b795dacaa7a9babb
Failed to look up boot ID d09c9b8618294949b795dacaa7a9babb+0: Cannot assign requested address
## Same for any other ID
...
john@bunsen1:~$ journalctl --boot=-1
Failed to look up boot -1: Cannot assign requested address
It just seems unable to access those directories.
I wonder if some ACL attribute is missing?
john@bunsen1:~$ ls -l /var/log/journal/*/system*
-rw-r----- 1 root systemd-journal 8388608 Oct 12 18:16 /var/log/journal/0d1849318e334326bee8799026fa767f/system.journal
-rw-r----- 1 root systemd-journal 8388608 Sep 27 18:48 /var/log/journal/12cd71a3446c4100ad1e6df7679ca271/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 5 18:48 /var/log/journal/152bce06150c4a0cb9457e8bcac88a77/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 13 18:33 /var/log/journal/1c4e6f402bd54dfea14143dcaa39a500/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 13 15:21 /var/log/journal/2300ddfebb1341908756bba469eead09/system.journal
...
Other people posting this command on the web have -rw-r-----+ in the first field, the '+' indicating that some extra attribute has been set. I don't know if the absence of that '+' in my case suggests anything?
Anyway, first off, can any other Debian Jessie users (with systemd ver. 215) report journalctl working OK for them, or not working also?
EDIT: @tknomanzr found that upgrading systemd from the backports fixed this issue. (systemd ver. 230 is available in jessie-backports)
Last edited by johnraff (2016-10-31 01:46:41)
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
Can you view the old logs as root?
What is the output of:
groups
I will check this on my BL system later.
Offline
I will check this on my BL system later.
My BL system will show all the old boots even with "Storage=auto" (the default setting) for a normal user in the systemd-journal group, /var/log/journal exists on this system but I may have created that myself.
In respect of the ACLs:
empty@TheLab:~$ ls -l /var/log/journal/*/system*
-rw-r-----+ 1 root root 16777216 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-0000000000000001-00053c5341e2d651.journal
-rw-r-----+ 1 root systemd-journal 8388608 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-00000000000032d0-00053f50217bdf04.journal
-rw-r-----+ 1 root systemd-journal 16777216 Oct 26 14:43 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal
empty@TheLab:~$
I'm not sure what that extra "+" symbol is, best I could get is from `info ls`:
A file with any other combination of alternate access methods is
marked with a ‘+’ character.
... which is rather cryptic
I tried `rm -rf /var/log/journal` on my system and the directory was re-created at the next boot with full access for systemd-journal group members, perhaps try that?
Last edited by Head_on_a_Stick (2016-10-26 14:27:46)
Offline
--file seems to nail it. (I think)
paul@debian:~$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR
paul@debian:~$ ls -l /var/log/journal/*/system* | head
-rw-r-----+ 1 root systemd-journal 4194304 Oct 7 21:15 /var/log/journal/09dd84fed83d436497e3699bd92756ea/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 19 07:20 /var/log/journal/1f0e0f01f70c4ceeba6cc70857628f9b/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 9 18:36 /var/log/journal/35f2419b558e46d89f65aad7fe15f152/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 22 21:08 /var/log/journal/3ecdf16c82c6483a9ba5cf3630153502/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 12 20:25 /var/log/journal/4025794fc832451d8a0cb4b4d8d48fa6/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 5 23:57 /var/log/journal/40eb35a60f654e5e963b0317bc742f19/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 14 17:55 /var/log/journal/4d62ca90dae7453892905a23c1e2a3ad/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 23 15:18 /var/log/journal/4d967415bb7449fe834b6709056ef9cd/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 17 22:27 /var/log/journal/5ff0a49ae1844cd1a3d81f6177804ba5/system.journal
-rw-r-----+ 1 root systemd-journal 4194304 Oct 12 18:37 /var/log/journal/609efaa700a64bad8ab0baa6397d0577/system.journal
paul@debian:~$ journalctl --file /var/log/journal/4025794fc832451d8a0cb4b4d8d48fa6/system.journal
-- Logs begin at Wed 2016-10-12 20:04:15 BST, end at Wed 2016-10-12 20:25:08 BST. --
Oct 12 20:04:15 debian systemd-journal[167]: Runtime journal is using 6.9M (max allowed 55.6M, trying to leave 83.5M free o
Oct 12 20:04:15 debian systemd-journal[167]: Runtime journal is using 6.9M (max allowed 55.6M, trying to leave 83.5M free o
Oct 12 20:04:15 debian kernel: Initializing cgroup subsys cpuset
Oct 12 20:04:15 debian kernel: Initializing cgroup subsys cpu
Oct 12 20:04:15 debian kernel: Initializing cgroup subsys cpuacct
Oct 12 20:04:15 debian kernel: Linux version 3.16.0-4-686-pae (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4
From a fedora forumhttp://forums.fedoraforum.org/showthread.php?t=305343
Hope that helps.
Offline
--file seems to nail it. (I think)
Hope that helps.
Yes, it did, thank you!
Using 'ls -ltr' puts the newest files at the bottom, so this is a usable workaround for checking the last couple of boots:
john@bunsen1:~$ ls -ltr /var/log/journal/*/system*
-rw-r----- 1 root systemd-journal 8388608 Sep 25 09:54 /var/log/journal/77e72c685d3a479faaee27ed729bea4a/system.journal
...
-rw-r----- 1 root systemd-journal 8388608 Oct 27 12:56 /var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/system.journal
john@bunsen1:~$ journalctl --file=/var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/system.journal
-- Logs begin at Thu 2016-10-27 10:25:51 JST, end at Thu 2016-10-27 12:56:26 JST. --
Oct 27 10:25:51 bunsen1 systemd-journal[197]: Runtime journal is using 8.0M (max allowed 76.5M, trying to leave 114.8M free of 75
Oct 27 10:25:51 bunsen1 systemd-journal[197]: Runtime journal is using 8.0M (max allowed 76.5M, trying to leave 114.8M free of 75
...
It would still be nice to be able to use the regular journalctl commands though...
In respect of the ACLs:
empty@TheLab:~$ ls -l /var/log/journal/*/system* -rw-r-----+ 1 root root 16777216 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-0000000000000001-00053c5341e2d651.journal -rw-r-----+ 1 root systemd-journal 8388608 Oct 20 19:31 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system@a15f1f965e2d4bd68fe443cb4e6fe1a9-00000000000032d0-00053f50217bdf04.journal -rw-r-----+ 1 root systemd-journal 16777216 Oct 26 14:43 /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal empty@TheLab:~$
I'm not sure what that extra "+" symbol is, best I could get is from `info ls`:
A file with any other combination of alternate access methods is
marked with a ‘+’ character.... which is rather cryptic
Yes. Other posts I've seen on the web also have that mysterious '+', which I lack. That could be the problem.
Also:
john@bunsen1:~$ journalctl --verify
PASS: /var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/system.journal
PASS: /var/log/journal/d5dda2a57fb74d1998613af343e2ee4a/user-1000.journal
It looks as if only the latest entry is considered usable. That was also the case yesterday, when the PASS file was different, and still present, but no longer passed.
I tried `rm -rf /var/log/journal` on my system and the directory was re-created at the next boot with full access for systemd-journal group members, perhaps try that?
I'll give that a go - well, mv not rm.
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
I'm afraid moving the journal directory didn't help.
This, after two boots:
john@bunsen1:~$ ls -ltr /var/log/journal/*/system*
-rw-r----- 1 root root 8388608 Oct 27 13:27 /var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal
-rw-r----- 1 root systemd-journal 8388608 Oct 27 13:29 /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
john@bunsen1:~$ journalctl --verify
PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal
john@bunsen1:~$ journalctl -b -1
Failed to look up boot -1: Cannot assign requested address
Note: the first files created are owned by root:root, not root:systemd-journal as they should be, and which happened with the second set.
After the first boot, 'journalctl --verify' showed /var/log/journal/937faef198c941b58a1f9d6d4082981f/user.journal as being passed, but not now.
The previous boot cannot be displayed, although 'journalct --file' still works.
And those '+' marks are still missing.
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
Looking at file attributes, e2fsprogs comes with lsattr:
john@bunsen1:~$ lsattr -R /var/log/journal
-------------e-- /var/log/journal/937faef198c941b58a1f9d6d4082981f
/var/log/journal/937faef198c941b58a1f9d6d4082981f:
-------------e-- /var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal
-------------e-- /var/log/journal/937faef198c941b58a1f9d6d4082981f/user-1000.journal
-------------e-- /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb
/var/log/journal/3ffd810c793f456c9f9a0107ebce54fb:
-------------e-- /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
-------------e-- /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal
And according to man chattr, " The 'e' attribute indicates that the file is using extents for mapping the blocks on disk." OK...
There's also an attr package (repos) which provides getfattr:
john@bunsen1:~$ getfattr -dR /var/log/journal
getfattr: Removing leading '/' from absolute path names
# file: var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal
user.crtime_usec=0sIjKqHNE/BQA=
# file: var/log/journal/937faef198c941b58a1f9d6d4082981f/user-1000.journal
user.crtime_usec=0s0/7VHdE/BQA=
# file: var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal
user.crtime_usec=0s/I2uK9E/BQA=
# file: var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal
user.crtime_usec=0sk3iNLdE/BQA=
About equally cryptic.
BUT, if @Head_on_a_Stick or @username, either of you could run 'lsattr -R /var/log/journal', or, after installing attr, run 'getfattr -dR /var/log/journal', and post part of the output, we might be able to identify something that your files have got that mine haven't, accounting for that '+' in the output of ls.
Help much appreciated!
Last edited by johnraff (2016-10-27 05:15:35)
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
Requested output:
empty@TheLab:~$ sudo lsattr -R /var/log/journal
---------------- /var/log/journal/d073d511f5ea4915955c41cdef8b8378
/var/log/journal/d073d511f5ea4915955c41cdef8b8378:
---------------- /var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal
---------------- /var/log/journal/d073d511f5ea4915955c41cdef8b8378/user-1000.journal
No extants for me, apparently...
empty@TheLab:~$ sudo getfattr -dR /var/log/journal
getfattr: Removing leading '/' from absolute path names
# file: var/log/journal/d073d511f5ea4915955c41cdef8b8378/system.journal
user.crtime_usec=0sWw8nV8U/BQA=
# file: var/log/journal/d073d511f5ea4915955c41cdef8b8378/user-1000.journal
user.crtime_usec=0sjxcrWMU/BQA=
Offline
My journalctl functionality seems to work fine, hope this can help ya;
Below: No + sign
kingghidorah@kingghidorah:~$ ls -l /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system*
-rw-r----- 1 root root 8388608 Oct 25 17:05 /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system@00053fb6df8c99a3-e6fa2e05c728d252.journal~
-rw-r----- 1 root systemd-journal 8388608 Oct 27 18:17 /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system.journal
kingghidorah@kingghidorah:~$
Below: Attributes
sudo lsattr -R /var/log/journal/
[sudo] password for kingghidorah:
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63
/var/log/journal/985e7e0edff6e6488183b8767b8d5b63:
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system@00053fb6df8c99a3-e6fa2e05c728d252.journal~
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/user-1000.journal
-------------e-- /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system.journal
Below is my /etc/systemd/journald.conf
[Journal]
Storage=auto
Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
MaxLevelWall=emerg
"I have not failed, I have found 10,000 ways that will not work" -Edison
Offline
I'm afraid moving the journal directory didn't help.
This, after two boots:john@bunsen1:~$ ls -ltr /var/log/journal/*/system* -rw-r----- 1 root root 8388608 Oct 27 13:27 /var/log/journal/937faef198c941b58a1f9d6d4082981f/system.journal -rw-r----- 1 root systemd-journal 8388608 Oct 27 13:29 /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal john@bunsen1:~$ journalctl --verify PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/system.journal PASS: /var/log/journal/3ffd810c793f456c9f9a0107ebce54fb/user-1000.journal john@bunsen1:~$ journalctl -b -1 Failed to look up boot -1: Cannot assign requested address
Note: the first files created are owned by root:root, not root:systemd-journal as they should be, and which happened with the second set.
After the first boot, 'journalctl --verify' showed /var/log/journal/937faef198c941b58a1f9d6d4082981f/user.journal as being passed, but not now.
The previous boot cannot be displayed, although 'journalct --file' still works.And those '+' marks are still missing.
Can't you change ownership of the original location to systemd.journal and add yourself to the systemd.journal group?
Offline
@HoaS hmm, so it looks as if attributes might be a distraction.
@H_B "My journalctl functionality seems to work fine" means you can read old journals OK with things like 'journalctl -b -1'?
An interesting difference your directory has from mine is this file: /var/log/journal/985e7e0edff6e6488183b8767b8d5b63/system@00053fb6df8c99a3-e6fa2e05c728d252.journal~
I've seen web references to these "system@something" files but just assumed they were from an earlier or later systemd version - I don't have any. (You are using systemd 215?)
You have different settings in /etc/systemd/journald.conf from mine:
[Journal]
Storage=auto
Compress=yes
ForwardToWall=yes
MaxLevelWall=emerg
Mine:
[Journal]
Storage=persistent
Did you add those settings yourself? Actually, I thought 'Storage=persistent' was necessary to keep old journals on disk.
@HoaS, @user do either of you have those system@... files in your journal directories?
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
Can't you change ownership of the original location to systemd.journal and add yourself to the systemd.journal group?
I'm already in the systemd.journal group (that's necessary to view any journals without sudo), and changing the group ownership on that one set of files made no difference, unfortunately.
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
bra10n wrote:Can't you change ownership of the original location to systemd.journal and add yourself to the systemd.journal group?
I'm already in the systemd.journal group (that's necessary to view any journals without sudo), and changing the group ownership on that one set of files made no difference, unfortunately.
So could the situation be that these logs are being ditched after each log out/reboot? This might explain their *existence* but not their recovery after log out etc.
I'm wondering now if the directory needs any priveleges with regards to "persistence".
Or perhaps revert to HoaS's config with
Storage=auto
Last edited by bra10n (2016-10-28 00:40:19)
Offline
The old log files definitely exist, and can be viewed with 'journalctl --file /file/path' - but they can't be brought up with the usual journalctl commands for "previous boot" etc, so it's pretty kludgy.
I've already set 'Storage=persistent' - I think that's necessary to get old journals saved on disk at all.
But, it might be that some subtle setting somewhere is missing... it would be nice to find it...
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
Or perhaps revert to HoaS's config with
Storage=auto
That was H_B's config, but I had been under the impression "persistent" was necessary to keep old records on disk... I'll try switching to "auto" and see what happens.
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
Here are the steps I always follow to get journaling up and running. Note: I have these saved in a file from a time when systemd still had a price-tag
mkdir /var/log/journal
chgrp systemd-journal /var/log/journal
chmod g+rwx /var/log/journal
usermod -a -G systemd-journal <your username>
systemctl reboot
I have these saved in a file for reference.
Here's a copy of my /etc/systemd/journal.config;
[Journal]
Storage=auto
Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
MaxLevelWall=emerg
Needless to say this works here. I am no means an expert mind, but I'm intriqued as to why you are having difficulties...
Edit: Sorry for the misquote above.
Last edited by bra10n (2016-10-28 00:53:05)
Offline
I've already set 'Storage=persistent' - I think that's necessary to get old journals saved on disk at all.
If /var/log/journal exists then the journal should be saved to disk even if "Storage=auto", at least according to journald.conf(5).
do either of you have those system@... files in your journal directories?
Yes, see the output in this post:
https://forums.bunsenlabs.org/viewtopic … 360#p39360
Offline
I've tried "Storage=auto" but it seems to make no difference. I've just noticed that while that mysterious attribute "+" is missing on my */system.journal files, it is present on my */user-1000.journal files. Either way, 'journalctl --user -b -1' is just as useless as the system version.
EDIT:
johnraff wrote:do either of you have those system@... files in your journal directories?
Yes, see the output in this post:
https://forums.bunsenlabs.org/viewtopic … 360#p39360
Ah! so you do. And I don't have any such files - maybe they perform some indexing function?
Still baffled as to why viewing of previous bootlogs seems to work OK for everyone else, but at this point I'm going to use a temporary workaround. I've written a couple of functions to put in ~/.bash_aliases that let me type 'journal' to see the last bootlog, which was what this was all about originally:
newest(){
if [[ $1 = '-n'* ]]
then
local offset="${1#-n}"
shift
else
local offset=0
fi
declare -A files=()
for i in "$@"
do
files[$(stat -c %Y "$i")]="$i"
done
local list=($(sort -nr <(printf '%s\n' "${!files[@]}")))
echo "${files[${list[$offset]}]}"
}
journal(){
journalctl --file "$(newest -n1 /var/log/journal/*/system.journal)"
}
That "newest" function will output the newest file of any list it's given, so might be useful to other people - if you give it -n1 it outputs the next newest file, and so on...
Last edited by johnraff (2016-10-29 06:37:18)
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline
...changing the group ownership on that one set of files made no difference, unfortunately.
Last idea... you could check who owns 'journal'
Offline
^It looks OK to me:
john@bunsen1:~$ ls -ld /var/log/journal
drwxr-sr-x 7 root systemd-journal 4096 Oct 30 10:26 /var/log/journal
...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), now on Bluesky, there's also some GitStuff )
Offline