You are not logged in.

#1 2019-06-10 18:49:52

AndrewSmart
Member
Registered: 2019-06-10
Posts: 7

[SOLVED] live issue --> persistence update issue & persistence guide

This is an old project on a flash drive, not sure what I was doing. If memory serves I had updated systemd with persistence set up and now it's FUBAR. I had to update systemd due to something else I was trying, like bluetooth or something.

I can see user:live is user:password for live install.

This is roughly the prompt I see (from memory sorry, I'll edit in a bit):

emergency mode enter root password press System-D to continue type journald -ad after logging in and view system logs:

Nothing I've tried works. "live", "user", "root", "bunsenlabs", "".

$ uname -a
Linux debian 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) i686 GNU/Linux
$ cat /etc/debian_version 
8.2

A message I see just before this emergency prompt is the "union mounting" of the persistence partition. I'll try to attach a photo of the screen but it is a POS camera.

Anyways, I love crunchbang, I've used my original 64-bit install of #! persistence for 5+ years and had just updated it (resquashing occasionally). This is my 32-bit usb stick for older machines... if this isn't recoverable I'll just install the latest bunsenlabs on it, no biggie.

Last edited by AndrewSmart (2019-07-01 19:35:56)

Offline

#2 2019-06-11 02:06:11

hhh
Meep!
Registered: 2015-09-17
Posts: 7,965
Website

Re: [SOLVED] live issue --> persistence update issue & persistence guide

My (now pretty old) persistence tutorial had the flaw of me having no clue how to upgrade the kernel. That might be the problem.

Offline

#3 2019-06-11 04:25:59

AndrewSmart
Member
Registered: 2019-06-10
Posts: 7

Re: [SOLVED] live issue --> persistence update issue & persistence guide

Ohhhh right. Thanks for the memory jog.

The solution to that problem is resquashing before shutting down. And you have to have booted with the 'toram' and 'persistence' kernel parameters so that you can write the new squashfs file to the live partition. By squashing everything that was in persistence back into live there won't be any issues with different versions of the kernel/libc on the live/persistence partitions on the next boot. The guide I have also cleans out stuff on persistence that was squashed.

EDIT: Also any apt package change that invokes update-initramfs is most conveniently done with 'toram' so that the new initrd can be written to /boot on the live partition. This is so that the live partition is able to be mounted rw instead of just ro. If this update-initramfs happens and you didn't set 'toram' it probably won't boot in persistence mode and you'll need to copy the new initrd over from persistence to live in either live mode (not persistence) or offline.

I made sure to resquash everything whenever I updated the kernel or libc, and systemd I wasn't sure if I had to or not (systemd was new to me then, and I had hunch there would be problem and there was, it went FUBAR). So that is the problem I had with this flash drive, thanks again for the memory jog.

EDIT: I'd since looked in the apt logs. Actually the problem was not systemd, it was that I updated libc from stable to testing to try to get something else to work. I intended to revert the upgrade or resquash before shutting down but had forgotten. The flash drive wouldn't boot again in persistence mode due to the libc conflict.

Now I need to read how to squash the live+persistence partitions back into the squashfs externally (not booted to said live partition). Then I expect it will boot. I don't need the live root password after all. Or I guess I'll just install a more recent version of BunsenLabs now that I think about it... idk.

I noticed the old #! forum isn't displaying my resquash guide, here it is below. I came up with it after lots and lots of trial and error, reading, and failures (I'm not sure the list of pseudo files I make are complete for example, but it boots and everything I do works!). The guide could do with a rewrite things had changed since then I added a few notes where I noticed new things:

--------------OLD CRUNCHBANG PERSISTENCE RESQUASH GUIDE---------------
I followed a different guide on this #! forum for creating the live USB in the first place. I didn't use unetbootin or syslinux (just an iso, dd, grub, and gparted really).
This guide assumes:

  1. You have two partitions on the live USB:

    1. The first containing the squashfs, vmlinuz, and initrd; this partition has the 'boot' flag.

    2. The second containing the rw persistence partition, labeled 'persistence'.

  2. Your persistence.conf file contains just "/ union", on the persistence partition.

Following this guide I had squashed the 2.3GB of used space to 550MB.  The increased space usage is due to the updating process; if you want the space back we must re-squash everything into the squashfs.  If you have a humongous USB, what you would gain from this guide is a some-what faster boot.  I wish there were a way for this to be automated; but AFAIK that is not possible at this time. If all your stuff installed since the last squash is too big for the live partition, then you must either uninstall stuff or make the live partition larger. I uninstall things like java or chrome as they're large and frequently updated, not worth squashing, then reinstall them later.

  1. First, you may have noticed if you run apt-get update, you may encounter errors if update-initramfs is triggered: http://crunchbang.org/forums/viewtopic.php?id=25013
    Follow this guide to get around that problem (so vmlinuz and initrd on the first partition can be updated): http://crunchbang.org/forums/viewtopic.php?id=38301 (old #! issue, maybe bunsenlabs not affected)
    I forgot to mention earlier that /boot must be mounted, so that the squashfs, initrd.img, and vmlinuz may be updated by update-initramfs:

    sudo mount /dev/sdb1 /live/image  #Replace /dev/sdb1 with your relevant USB partition containing /boot stuff (initrd.img, vmlinuz). With more recent debian versions than #! this /live/image now seems to be /lib/live/mount/medium, and may have to be remounted as rw so that /boot can be updated by update-initramfs.
    • In the above guide you must be booted with both the 'persistence' and 'toram' kernel parameters. 'toram' is necessary to be able to mount the /boot partition otherwise it will be in-use.

  2. Now, run this script to generate the new squashfs (edit TARGET_SQUASHFS_FILE to where desired, such as /tmp or an external HDD like I did): https://gist.github.com/AndrewSmart/90e … 08db8f1426

    • In the above guide you must be booted with the 'persistence' kernel parameter; 'toram' doesn't matter, though it would be quicker if so.

    • Two further optimizations on the above script would be to not squash /boot and /var/log; though it works fine as is.

    • In the script I chose not to squash /home.

    • I suggest you back up your original squashfs the first time you try this guide.

  3. Now copy TARGET_SQUASHFS_FILE to your first partition, over ./live/image/filesystem.squashfs I believe. Also move the filesystem.packages there if you wish to follow that convention (I believe that file is necessary if you wish to install #! to the HDD).

  4. Now most everything had been squashed into the squashfs on the first partition; but there is a problem as we have the files duplicated both there and in the persistence partition.  aufs unfortunately won't clean up the duplicates from the rw branch (I wish it would).  We have to do that manually.  Boot without the 'persistence' kernel parameter, and run this script on your persistence partition: https://gist.github.com/AndrewSmart/2f6 … 1922c4556f

  5. Now boot as you normally would with the 'persistence' kernel parameter and verify you receive no new error messages on startup or shutdown.  If all is well, remove /bak at your convenience.  Enjoy your updated & condensed live USB stick!

Last edited by AndrewSmart (2019-07-19 18:51:34)

Offline

#4 2019-06-11 13:31:02

hhh
Meep!
Registered: 2015-09-17
Posts: 7,965
Website

Re: [SOLVED] live issue --> persistence update issue & persistence guide

Thanks for posting that! I probably won't test until after Lithium is released, we have quite a bit of work to do in a short time. smile

Offline

#5 2019-06-13 21:20:20

AndrewSmart
Member
Registered: 2019-06-10
Posts: 7

Re: [SOLVED] live issue --> persistence update issue & persistence guide

No problem, thank you for the great lightweight distro.

Yes /live/image seems to be /lib/live/mount/medium now. That is an issue in my guide above. IIRC it can be remounted rw given 'toram', as it is mounted ro by script:

/dev/disk/by-uuid/c27f1e3d-77af-4023-83a2-f143268b33ab /lib/live/mount/medium ext4 ro,noatime,stripe=128 0 0

Only after it is remounted as rw then update-initramfs can update initrd.img, though I think /boot/ has to symbolically link to the right place. Without 'toram' it won't allow being remounted as rw (IIRC).

There might be other issues, like I ought do do uuid stuff in scripts instead of stuff like /dev/sdb1. I have other utility scripts laying around, like diffing the packages between live & persistence, useful when trying to shrink the filesystem.squashfs file as small as possible.

Also maybe my issue wasn't systemd, I think my memory got mixed up. I'd updated lib & initrd without squashing them into live (I can tell by looking at persistence partition). Going to try to surgically get them in there.

My /boot/grub/grub.cfg:

menuentry "#! 11 - waldorf 3.16 64bits - by UUID - Persistence" {
  #Enter the UUID of your boot partition (this is where grub and your kernel reside)
  set uuid_grub_boot=ab3920b1-2d3f-4bf8-9da1-27bc2b5c9e2f
  #Enter the UUID of the persistence partition.
  set uuid_os_root=aedab356-05bc-4e30-ba77-6317b0210c7e
  #Here we set the grub "root" variable by locating the uuid of the root partition identified above
  search --no-floppy --fs-uuid $uuid_os_root --set=root
  #Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
  search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
  #Here's the magic. We test to see if the boot and root partitions have the same UUID.
  #If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
  if [ $uuid_grub_boot == $uuid_os_root ] ; then
    set grub_boot=$grub_boot/boot
  fi

  linux ($grub_boot)/live/vmlinuz-3.16.0-4-amd64 boot=live config quiet noeject toram persistence persistence-storage=filesystem usbcore.autosuspend=-1 live-media=/dev/disk/by-uuid/$uuid_grub_boot elevator=as
  initrd ($grub_boot)/live/initrd.img-3.16.0-4-amd64
}

menuentry "#! 11 - waldorf 3.16 64bits - by UUID - Live" {
  #Enter the UUID of your boot partition (this is where grub and your kernel reside)
  set uuid_grub_boot=ab3920b1-2d3f-4bf8-9da1-27bc2b5c9e2f
  #Enter the UUID of the persistence partition.
  set uuid_os_root=aedab356-05bc-4e30-ba77-6317b0210c7e
  #Here we set the grub "root" variable by locating the uuid of the root partition identified above
  search --no-floppy --fs-uuid $uuid_os_root --set=root
  #Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
  search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
  #Here's the magic. We test to see if the boot and root partitions have the same UUID.
  #If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
  if [ $uuid_grub_boot == $uuid_os_root ] ; then
    set grub_boot=$grub_boot/boot
  fi

  linux ($grub_boot)/live/vmlinuz-3.16.0-4-amd64 boot=live config quiet noeject toram usbcore.autosuspend=-1 live-media=/dev/disk/by-uuid/$uuid_grub_boot
  initrd ($grub_boot)/live/initrd.img-3.16.0-4-amd64
}

Notes:
live-media & persistence-storage kernel parameters seemed to be necessary for me, and uuid's necessary for jumping from computer to computer.
The menuentry I copied from a forum somewhere else, I needed something supporting live & persistence booting by UUID and I modified it.

IIRC update-initramfs or some hook thereabouts was in a crunchbang live-tools .deb package. I bet the whole process of updating the kernel on a persistence usb can be scripted in live-update-initramfs of live-tools package (I know #! had a custom script here, so I expect bunsenlabs does too). Maybe have a pre-update hook require that 'toram' is among the kernel boot parameters so that initrd & vmlinuz can be updated. Maybe resquash filesystem.squashfs when things like libc and nvidia are updated. Would be neat to have those scripts handle this automagically.

It's worked great for me for 5-6+ years as my main operating system... very fast boot time and shutdown time on USB3 ports. Can jump from computer to computer and have all my software ready, any library/lab/home. Never had to deal with Windows or unfamiliar systems again, reinstalling software over & over, or even installing linux again. HDDs are just for storage now. I love this workflow I'd settled in, works well for me.

The only issue I have is sometimes usb filesystem writes hang (blocks) when I only have 2-3 gb left of free space on the 32gb stick, like no usb reads can proceed until a lengthy write is done, it may be an old aufs/kernel or whatever bug though... I'm not sure. I just have to keep at least 3gb free space on it as it gets worse the less free space there is. I think it was a regression, I didn't have it initially until I updated the kernel. Maybe it's fixed in newer kernel or different union filesystem. I think I found the bug report and bookmarked it somewhere but it was so low incidence that the devs put it on the backburner. Anyways, works great besides that.

Last edited by AndrewSmart (2019-06-13 22:43:15)

Offline

#6 2019-07-01 18:17:04

AndrewSmart
Member
Registered: 2019-06-10
Posts: 7

Re: [SOLVED] live issue --> persistence update issue & persistence guide

Ok... a couple surgical attempts didn't work with constraints I won't go into. I guess it may have worked better if I could remember what I had done to screw it up. I decided it would be less effort to get the newest bunsenlabs iso than to fix whatever I did to this USB, I can just move my /home over. My 64 bit Crunchbang (pre waldorf) persistence USB has worked great for 5-6+ years updating the kernel and everything, but won't work on my old 32 bit 900A EEEPC netbook (1 GB ram stock!).

I will add my notes here because I think there are interested people. My setup doesn't support UEFI or whatever... you can fill in those gaps, I've never dealt with UEFI. Rant on UEFI I just have to get off my chest:

Whatever UEFI is I doubt it's better than the ancient Sun's SPARC Openboot, I love OpenBoot and I miss it, seriously powerful and how personal computers SHOULD be. Being without OpenBoot on a PC is like removing the steeringwheel from your car (seriously not joking sad, there is a reason you see Solaris diehards that won't convert to Linux and this is one of them I think). Maybe removing the parking brake and OBD2 port would be better analogy than steering wheel, but not everyone knows what OBD2 port is used for so... meh. Openboot is open-source and no longer burdened by patents AFAIK... so I have doubts about over-complicated obtuse UEFI stuff and the motivation behind industry (*cough* NSA *cough*) pushing it ("gee how do we get around this secure Linux/BSD OS?")... and rootkits jumping on the bandwagon... I'm not buying any UEFI hardware... sorry rant.

I'm putting my notes here if you ever want to try this out or make Bunsenlabs support this workflow better, or something.

I used latest bunsenlabs iso (bl-Helium-4-i386.iso) but it was very big, 1.1GiB. This is not good for 'toram' and 'persistence' kernel parameters, we can do better by things like removing the browser and duplicate vmlinuz initrd.img within the squashfs. This way it will boot faster (very meaningful with USB2 ports!), take up less RAM, and use less USB space. I shrank my squashfs down to 448MiB, I could have gone smaller but I didn't care to sort through the smaller packages.

Here are the partitions on my USB, I believe the 4.0MiB unallocated space at the start is to mitigate portability problems (whatever guide I had used years ago said to do that):
googledrive hosted image of gparted screenshot, open in new tab if it doesn't display inline, shows ext4 live and persistence partitions with boot flag on live

I don't recall the grub install command to the live partition... you can figure that out.

Now we gut the iso (filesystem.squashfs on the iso) of packages we don't want. Gut packages that are large and/or frequently updated, they can go onto persistence partition (remove now, install them later).

# Temp directories
sudo mkdir /media/iso.squashfs /media/iso /media/myadditions /media/overlay
sudo mount -o loop bl-Helium-4-i386.iso /media/iso

#copy filesystem.squashfs, initrd.img, and vmlinuz off of there. Also config, System.map and filesystem.packages.xz if you want to follow convention.
sudo mount -t squashfs /tmp/filesystem.squashfs /media/iso.squashfs

#rw branch must be before ro, if out of order you will see error and /media/overlay will be ro instead of rw
#We do this to make new smaller squashfs (with stuff removed).
sudo mount -t aufs -o br=/media/myadditions=rw:/media/iso.squashfs=ro -o udba=reval none /media/overlay/

#resolv.conf and mount binds are to remove error/warning messages when removing packages with apt
sudo cp /etc/resolv.conf /media/overlay/etc/resolv.conf
sudo mount --bind /dev/ /media/overlay/dev/
sudo mount --bind /dev/pts /media/overlay/dev/pts
sudo mount --bind /proc /media/overlay/proc

#Now chroot into the squashfs so we can use apt within it to remove packages
sudo chroot --userspec root:root /media/overlay

#Run this to see packages installed ordered by size, then apt-get remove or purge stuff. Oriental fonts took up a lot of space:
dpkg-query -Wf 'Size\t\n' | sort -n | less

#I think I installed deborphan and localepurge to help clean stuff.
apt-get install deborphan localepurge

#A subset of stuff I removed (I guess I should have left firmware packages for portability, oh well, too late):
apt-get purge vlc-l10n firefox-esr filezilla
apt-get purge  p7zip-full libgoffice-0.10-10-common firmware-ti-connectivity  firmware-cavium firmware-qlogic gnumeric-common libgsf-1-114 libgsf-1-common p7zip pxlib1
apt-get purge firmware-brcm80211 aptitude-common aptitude gstreamer1.0-plugins-bad gstreamer1.0-plugins-good synaptic
apt-get purge transmission-gtk hexchat-common libevent-2.0-5 libminiupnpc10 libnatpmp1 transmission-common
apt-get purge samba-libs libavahi-glib1 libcdio-cdda1 libcdio-paranoia1 libgd3 libgdata-common libgdata22 libgoa-1.0-0b libgoa-1.0-common libgphoto2-6 libgphoto2-port12 libldb1 libmtp-common libmtp9 libnfs8 liboauth0 libpython2.7 libtalloc2 libtevent0 libwbclient0 python-talloc
apt-get purge libflite1 libass5 libatomic1 libavformat57 libbs2b0 libchromaprint1 libebur128-1 libfftw3-double3 libgme0 libopencv-core2.4v5 libopencv-imgproc2.4v5 libopenmpt0 libpgm-5.2-0 libpostproc54 librubberband2 libsodium18 libssh-gcrypt-4 libswscale4 libtbb2 libzmq5
apt-get purge gstreamer1.0-plugins-ugly liba52-0.7.4 libcdio13 libdvdnav4 libdvdread4 libmad0 libmpeg2-4 libmpg123-0 libopencore-amrnb0 libopencore-amrwb0 libsidplay1v5

#Remove stuff it tells you to
deborphan
apt-get autoremove
apt-get clean

#I recommend vim.tiny over vim, vim.tiny much smaller, also install whatever else your heart desires that is small and infrequently updated. These will be available in 'live' mode (if installed when in 'persistence' mode, it won't be available in plain 'live' mode):
apt-get install vim-tiny
ln -s /usr/bin/vi /usr/local/bin/vim

#Also, the live scripts will have a heart attack if the user named 'user' doesn't exist... I guess this setup does things out of the order the live scripts expect before they setup the 'user' account, so to fix that problem we make it here within the chroot:
adduser user

#If you boot up and it prompts you to log in and says 'Authentication Failure' and virtual terminals don't let you log in either, this is the problem, 'user' account was not set up.

#There might be other trivial bugs I forgot about fixing. Like apt may complain warnings about /etc/default/locale being not set up. That problem is only within the chroot and won't exist after booting up.

Within the chroot into the squashfs, remove stuff in /boot, we already have copies. vmlinuz and initrd.img will go into the live partition, if they are duplicated within the squashfs it does nothing but waste space.

Then I used the scripts mentioned earlier in this thread to do some things like removing apt's cache and making the filesystem.squashfs from /media/myoverlay (compressing it takes a long time). You'll have to use your brain when interpreting the script lines, like some lines will be within the chroot. Others like making the filesystem.squashfs will be outside the chroot and you're squashing /media/overlay as written in this guide.

Then put the resulting filesystem.squashfs into /boot on the live partition, along, initrd.img, vmlinuz, and the other optional things if you want to follow convention.

Cleanup:

sudo umount /media/overlay/dev/pts
sudo umount /media/overlay/dev/
sudo umount /media/overlay/proc
sudo umount /media/overlay
sudo umount /media/iso.squashfs
sudo umount /media/iso
sudo rm -r /media/myadditions
sudo rmdir /media/iso /media/iso.squashfs /media/overlay

Make sure to fill in grub.cfg with your appropriate UUIDs mentioned in earlier post.

And don't forget to make the persistence.conf file '/ union' on the persistence partition.

This is kind of an incomplete guide... but it's what I did. Feel free to make it more complete.

I think I might have messed up the permissions of some files on my install. I didn't mess up in this guide but when I first did it I processed stuff in /tmp and that sometimes causes permissions issues, with 'extended' file permissions/attributes or whatever. I made sure to always process files in /media in this guide to mitigate that possibility. I may have to redo it I guess. The keyring has issues remembering (being persistent) EDIT: Ok I installed seahorse and had to set the 'Default Keyring' as the default keyring ironically and that fixed the problem smile. Also there is a 'live' welcoming prompt box that pops up on every Bunsenlabs boot, I need to find where that is and remove that line EDIT: ok removed that line in bottom of .config/openbox/autostart. There might be other issues I don't know about, I've only used this a handful of days.

I originally settled on using Crunchbang like this after reviewing lightweight Debian distributions years ago. Blog posts praised #! for the features I wanted (e.g. follows stable Debian release, lightweight, minimal interface). I didn't like pupplylinux, and I didn't see anything else I liked. I guess there might be a more suitable Debian distro for this USB persistence setup but I don't know of any. Any Debian distro should work with squashfs persistence really... it is upstream Debian that supports this. Xubuntu too heavy and had other issues... that is what I did before vanilla Debian, then Crunchbang.

Anyways, I think you (any interested reader) can fill in the gaps from what I've written and make a complete guide/setup if you wish.

Last edited by AndrewSmart (2019-07-04 23:06:43)

Offline

Board footer

Powered by FluxBB