LVM guide (by el_koraco)

johnraff · 2020-07-28 07:06:35

Anyone remember el_koraco from the CrunchBang forums? Well back in the day I used Scrapbook (now defunct Firefox extension) to store his Monumental LVM Guide, and still refer to it when I want to set up a new system. So I thought it wouldn't hurt to have it available here. You can still find it on the #! forum but most of the content is invisible, so I'm going to ressurrect it here. I'll leave out the chit-chat, and trim things down a bit here and there, maybe do an occasional edit (especially old URLs), and perhaps add a final post on snapshots+backups.

NOTE: Any bits marked like this have been added by me.

Last edited by johnraff (2020-07-29 06:54:59)

johnraff · 2020-07-28 07:18:50

The monumental #! LVM guide by el_koraco Posted 2012/5/6

Caveat - 100% of what this thread will entail is based on manpages, the Linux Documentation Project, System Rescue CD and other sources. The point being, none of it is original and I'm not trying to reinvent the wheel, but neither #! nor Debian have easily accessible LVM documentation, so few people use it, which is a shame.

It goes without saying that you shouldn't try anything described here without backing the fuck up, then backing up the backup, and backing up the backup's backup.

The players

What is LVM, or Logical Volume Manager? It's a userspace tool making use of a kernel mechanism called device-mapper (the same mechanism used by RAID) for higher level storage. In human readable words, LVM offers you a more flexible partitioning scheme, avoiding a lot of the traditional partitioning complications. Instead of working with partitions like /dev/sda1 or /dev/sda2, it works with its own concepts, like physical volumes, volume groups and logical volumes. On server deployments, this offers advantages such as having the system treat numerous hard drives as a single one and so on, but that's not what this thread is about.

Here, we'll go into using LVM on a single hard drive, on your bog standard personal machine. Now, LVM is primarily designed for server systems (it's the default partitioning scheme for Red Hat and clones and Red Hat Unusable), and people (mostly people from the Ubuntu forums) like to mention how there are few benefits to using LVM on a personal machine. Nothing could be further from the truth, as we'll see. To start off with, some concepts:

Physical Volume - as the name suggests, this is the physical storage used by LVM. It can be one hard drive, numerous hard drives, one partition, several partitions, whatever you want. Physical volulmes are applied to;

Volume Group - this is the sum of Physical Volumes. LVM works with your Volume Groups, which it uses for;

Logical Volume - these are the "partitions" inside Volume Groups, or "inside LVM" so to speak. This is where /home and / and /tmp and /var and the gang go;

Let's get partitioning

So to sum it up, we'll go into partitioning your hard drive during the #! or Debian install, assign your entire hard drive minus a /boot partition (a Physical Volume) to a Volume Group inside which we'll make Logical Volumes. Head hurt yet? No problem, it shall be revealed. Now, making a separate /boot partition is largely a legacy thing, cuz Grub One wasn't able to boot from LVM, since LVM is a userspace tool, not a kernel mechanism. Grub Two can freely boot from an LVM volume, so you don't need the /boot partition anymore, but I've found that having a /boot partition is good for many reasons (ditching Grub for extlinux is a good reason).

Debian does the best default LVM partitioning, but it's still suboptimal, because it assigns the entire LVM space to logical partitions, which is not necessary. LVM enables you to resize partitions on the fly, inside a running system, so you really only need as much space for a logical partition as you would normally use. Say, you can make / 3GB, and add more space to it later if necessary. Nevermind, let's get into partitioning, things will become clearer then. I'm assuming a text install, because that's what I normally do. The graphical install is the same, but with mouse clicks instead of key presses. Backup all your data to an external medium, and start the install. This entire section is covered with graphics here.

Go baby!

Once you get to the partitioning section, choose Manual. Delete all the partitions. Now, select the free space and make a 200 MB partiton, assign it to /boot and make it ext2 (you don't need journaling on /boot). Select the rest of the free space, in the "Use as" section, select "Physical volume for LVM", done setting up the partition. Now, go above, to the

Configure the Logical Volume Manager

and

Create Volume Group

Name it what you will, I named mine "debian", so this is what I'll be using in this thread. The installer will then offer you a choice of two partitions to put the VG on. Choose /dev/sda2 or /dev/sda5, whatever is offered. Now, we'll make /, /home and /swap logical volumes. In the LVM setup menu, there'll be the option to

Create Logical Volume

Select that, the installer will offer you the volume group "debian". Select it (Enter or mouse click). Logical Volume name is "root" (without the caps, of course). Logical Volume size? Make it 5GB. Just type in 5GB in the installer. Now, repeat the next two steps for home and swap (name them home and swap). I made home 30 GB, you can make it smaller or bigger, depending on how much data you'll put on there. Say, go for below 75 percent occupacy in home and make swap the size of your RAM, or don't make swap at all, as you see fit. I'll make my swap 1.5 GB. Once you've made logical volumes for home and swap, select

Finish

You're back at the main installer menu, but now you'll see more options. On the very top, there will be LVM entries. Something like

"LVM VG, debian, LV home - 30 GB Linux device-mapper (linear)
> #1 30 GB

Select that one, you'll get the standard partitioning options. Use as ext4, mountpoint /home, Done setting up the partition. Next

"LVM VG, debian, LV root - 5 GB Linux device-mapper (linear)
> #2 5 GB

Select, use as ext4, mountpoint /, Done setting up. Finally,

"LVM VG, debian, LV swap - 1,5 GB Linux device-mapper (linear)
> #3 1,5 GB

Select, use as swap area, done. Procced with the install, install GRUB to the MBR for now, reboot into your new system, and test some stuff. You'll no longer have all those /dev/sda entries to fiddle with, but your new and vastly improved scheme:

fdisk -l

Disk /dev/sda: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006f7f2

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *       38883       38914      243712   83  Linux
/dev/sda2               1       38883   312326145    5  Extended
/dev/sda5               1       38883   312326144   8e  Linux LVM

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/debian-root
                      4.7G  2.3G  2.2G  52% /
tmpfs                 974M  4.0K  974M   1% /lib/init/rw
udev                  972M  244K  971M   1% /dev
tmpfs                 974M     0  974M   0% /dev/shm
/dev/sda1             224M   18M  195M   9% /boot
/dev/mapper/debian-home
                       31G  966M   29G   4% /home

Next up, we'll deal with resizing (growing and shrinking) logical volumes, snapshoting, making new volumes, mounting them, making backups and generally having fun.

johnraff · 2020-07-28 07:58:21

Making a logical volume designed specifically for videos and making it user owned and autostarted

As we've said in the first post, one of the main benefits of LVM is that the standard partitioning restrictions don't apply. You can make tens, hundreds, thousands of logical volumes, and LVM will not complain in the slightest.

So, since videos usually take a good chunk of space, let's make a logical volume specifically for them. You can do the same for music, documents, images, whatever. The additional coolness is that if you do a reinstall with an LVM-aware installer, like the Debian one (or the Ubuntu debian installer, known as "Alternate install CD", or Anaconda, or AIF), you can totally leave out the logical volumes and assign them to the system later on. Also, since videos are usually large chunks of data, let's, just for fun, use XFS, the former Irix filesystem designed to work with a lot of large chunks of data. First off,

sudo apt-get install xfsprogs xfsdump
mkdir -p videos

Now, we'll start working with the LVM tools lvcreate and lvdisplay. Open up a root terminal and start
working:

elkoraco@mrdeb: sudo -i
root@mrdeb: lvdisplay
 --- Logical volume ---
  LV Name                /dev/debian/root
  VG Name                debian
  LV UUID                LTACtF-y34L-MZeo-QQ9E-HXCc-LYBB-RAOWTc
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                4.66 GiB
  Current LE             1192
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0
   
  --- Logical volume ---
  LV Name                /dev/debian/home
  VG Name                debian
  LV UUID                EgJC5M-3caQ-dyc5-fp7Y-6MYs-22cW-MD0C1e
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                31.00 GiB
  Current LE             7936
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:2
   
  --- Logical volume ---
  LV Name                /dev/debian/swap
  VG Name                debian
  LV UUID                wG10aT-r9CH-ckB3-NYNA-nh0n-Tsi7-u3bGI0
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                1.39 GiB
  Current LE             357
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

As you see here, the LVM tools have slightly different naming scheme than the kernel. Device mapper gets its name because it maps block devices to one another. So the kernel and other tools see your logical devices as /dev/mapper/debian-whatever. Since LVM is obviously aware of volume groups and logical volumes, it applies the /dev/group/name naming scheme. So in our case, LVs are named /dev/debian/home and so on.

NOTE If you look in the /dev directory, at least on current Debian, you'll find that /dev/mapper/debian-home and /dev/debian/home (or whatever names you're using) are both symlinks pointing to a top-level file, maybe named dm-1 or whatever. So the two naming schemes are interchangeable. One point: although hyphens are OK in volume group and logical volume names, the central hyphen in /dev/mapper/vg/lv then gets doubled to keep things unambiguous. So maybe avoid hyphens in names. Also:

man lvm wrote:

A directory bearing the name of each Volume Group is created under /dev when any of its Logical Volumes are activated. Each active Logical Volume is accessible from this directory as a symbolic link leading to a device node.
Links or nodes in /dev/mapper are intended only for internal use and the precise format and escaping might change between releases and distributions. Other software and scripts should use the /dev/VolumeGroupName/LogicalVolume‐Name format to reduce the chance of needing amendment when the software is updated. Should you need to process the node names in /dev/mapper, you may use dmsetup splitname to separate out the original VG, LV and internal layer names.

We're about to make our first LV:

root@mrdeb: root@mrdeb ~ # lvcreate -L50G -n video /dev/debian
  Logical volume "video" created

Let us take a look at the commands and options given. lvcreate is used to create an LV. The -L50G option tells LVM to create a 50GB logical volume. The -n option tells LVM how we want to name the new LV. If we didn't give this option, the LV would get an ugly automatic name, which we don't want! Finally, we're telling LVM to make the LV on the debian volume group. (See 'man lvcreate' for details.) Let's run lvdisplay again, and we'll see a new entry:

root@mrdeb: lvdisplay 
....
 --- Logical volume ---
  LV Name                /dev/debian/video
  VG Name                debian
  LV UUID                Soe6bQ-C5Hb-nZU9-Lv5i-oUZB-DFWm-2yzbqq
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:4

Now, a volume in and of itself is of little use. We need a filesystem on it. So, we'll be making an XFS volume now (we'll go into ext4 systems later, but since it's a tutorial, why not expand):

root@mrdeb: mkfs.xfs /dev/debian/video
meta-data=/dev/debian/video      isize=256    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

And goddamn, mxfs.xfs is fast! Now, let's mount the LV in our home folder and give the ownership to the user:

root@mrdeb ~ # mount /dev/mapper/debian-video /home/elkoraco/videos
root@mrdeb ~ # chown elkoraco:elkoraco /home/elkoraco/videos/

If we go to the videos folder now, Thunar will tell us we have 49,something GB space. The last thing to do is mount the LV to /homeelkoraco/videos in fstab (the file that tells the kernel how to mount specific partitions and volumes), so we open /etc/fstab as root (or with gksudo or sudo) and add this:

/dev/mapper/debian-video /home/elkoraco/videos           xfs    noatime         0       2

Take note, you don't use UUIDs with LVM, but the /dev/mapper naming scheme. That's it! Easy as pie! You can freely resize this directory later on, which we wil get into in the next chapter.

NOTE While I've found the /dev/vg/lv naming scheme usable in mount commands, this user at least thought the mapper scheme better for fstab.

johnraff · 2020-07-28 08:07:01

Growing a logical volume

Now, before we get into the nitty-gritty of resizing logical volumes, let's go back to the install process. We made the root partition only 5 GB (the default Debian value is 10 GB). Why? In a properly set-up LVM scheme, there is never a need to shrink partitions (I may talk about how this is done, but I'm not sure, as resizing filesystems can be dangerous). Consider this example. You have a hard disk like mine, of 320 GB. During the install, you make / 10GB, swap 4GB and the rest goes to home. Suddenly, you find yourself in a situation in which you need another partition. Doesn't matter why. What do you do? Well, burn a live distro that comes with Gparted, boot it, run Gparted, resize your /home, pray to God the system didn't get borked as a result and make a new partition.

With a good LVM setup, this never happens. We assigned 319,75GB to the LVM Physical disk, and of that 319,75GB, we made root 5GB, /home 30GB and swap 1,5 GB. That leaves us with 283,25 GB of space to use as we see fit. The free space is there, and LVM is ready to happily let you use it as you want. So we made a 50GB LV for videos, which leaves us with 253 GB. So, say we start running out of room on /home and /home/elkoraco/videos - we'll just add more space to the partitions, and resize the filesystems on them onfuckingline in mere seconds. We have both ext4 and XFS, so we will learn how to resize them both. First, for /home (I'm assuming a root terminal, which you open with sudo -i. You can execute every command with sudo if you don't wanna open a root terminal, I don't use sudo for admin purposes, it's just how I roll these days):

NOTE lvextend, lvreduce and lvresize (which covers both) now accept a --resizefs option, which (for many file systems - please check yours) will do the file system resizing for you at the same time, so you don't have to use those resize2fs commands or worry about getting the file system size just right.

root@mrdeb: lvextend -L+1GB /dev/debian/home
  Extending logical volume home to 32.00 GiB
  Logical volume home successfully resized

The operation is done in no time, as most LVM operations are. Let's take a look at the options: lvextend is used to extend a logical volume. Again, we're using the -L option for indicating a linear volume. Next to that, we're giving the option on how to extend. +1GB means extend it by one GB. Finally, we've told LVM which LV to extend. Take note, the +1GB isn't the only available stanza, which we'll show during extending /home/elkoraco/videos. For now, wee need to resize the ext4 filesystem on /home:

root@mrdeb: resize2fs -p /dev/debian/home
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/debian/home is mounted on /home; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 2
Performing an on-line resize of /dev/debian/home to 8388608 (4k) blocks.
The filesystem on /dev/debian/home is now 8388608 blocks long.

We've given resize2fs (it's part of e2fsprogs, and isntalled by default) the command to resize the filesystem. Since we've given it no options, it resized to the maximum available free space. The -p option is just for resize2fs to show you the progress. Check out Thunar, /home is now a Gig bigger. Let's extend /home/elkoraco/videos:

root@mrdeb: lvextend -L53GB /dev/debian/video
  Extending logical volume video to 53.00 GiB
  Logical volume video successfully resized

The same thing, only this time we used a different stanza. Instead of giving the +1GB option, we told LVM the exact size of the LV we want after the resize. It used to be 50GB, now it's 53. I actually wanted to make it 51, but tyoped . So, we need to resize the filesystem once again. While online resizing is a fresh and welcome addition to the ext family of filesystems, XFS can only be resized while online and mounted, and the XFS website even recommends the use of this filesystem on LVM, due to stuff being easier. Luckily for us, /home/elkoraco/videos is already mounted via fstab. So, to resize:

root@mrdeb ~ # xfs_growfs -d /dev/debian/video
meta-data=/dev/mapper/debian-video isize=256    agcount=4, agsize=3276800 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=13107200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 13107200 to 13893632

You need to pass the -d option to xfs_growfs, so that it knows to take up all available space. Also, xfs_growfs is instant. All in all, growing the LVs and resizing the filesystems on them took less than a minute (a live distro would hardly begin booting by that time), and it is completely bullet-proof safe.

Next up, snapshots and backups - LVM at its best.

johnraff · 2020-07-28 08:23:01

Snapshots and backups

It was all fun and games up until now, but now we're gonna see the true power of LVM. Backing up the root partition. Most people don't do it. Why not? Well, for one, they don't expect things to go sour until they do. Or else they find the process too complicated. Either way, most people back up just their data and reinstall the system. By using LVM and following a few simple techniques, you'll never have to worry about a borked system again, because you'll have a reliable backup ready to go.

We'll be using a nifty little tool called fsarchiver (File System Archiver). It's in the repos, so you just need to apt-get it. However, some caveats apply when doing root partition backups. Most importantly, the root partition shouldn't be mounted. It can be, but you risk stuff changing on the partition while it's being backed up and having the backup result in an unconsistent state. So, since root can't be unmounted while it's in use, doing a root backup means booting a Live CD, mounting your backup partition somewhere and doing the backup. Once again, LVM comes to the rescue with its über-awesome snapshot feature.

(This paragraph slightly edited by jr)
You may have heard about snapshots while listening to stuff about ZFS or btrfs. These filesystems combine a filesystem and logical volume manager in one. It will take at least until the post-Wheezy Stable Debian for btrfs to become default, so we'll use LVM. An LVM snapshot is simply a view of a logical volume frozen in time. The resulting snapshot can be made much smaller, as it doesn't contain any data by itself, merely references to the data on the original volume. As stuff changes on the original volume, the hardlinks in the snapshot are changed to files, in the state they were when the snapshot was made, so the data shown to the outside world never changes, although the snapshot gradually occupies more space as the original moves on. You can think of it as something like forking a new branch in git? Here, we will make a backup logical volume, make an ext4 filesystem on it, mount it, snapshot our / volume, and then use fsarchiver to make a backup of the snapshot. It may sound complicated, but it couldn't be easier. As usual, open up a root terminal:

root@mrdeb: mkdir -p /mnt/backup
root@mrdeb ~ # lvcreate -L2G -n backup /dev/debian
  Logical volume "backup" created

As you see, we've passed the option to make the backup volume 2GB. fsarchiver is efficient in compression, so we don't need much more. (That was in 2012! My backup partition is 100G) We could either make it bigger to begin with or expand it later, as per the last post. Going further, let's make an ext4 filesystem on the volume, before we mount it:

root@mrdeb ~ # mkfs.ext4 /dev/debian/backup
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Now, we'll mount it on /mnt/backup:

root@mrdeb: mount /dev/debian/backup /mnt/backup

It's time to snap the root volume:

root@mrdeb ~ # lvcreate -L1G -s -n snap /dev/debian/root
  Logical volume "snap" created

Take note that our root volume is 5GB, and the snap only 1GB. We could have made it even smaller, snapshots require very little space initially. As the original changes, the snapshot will grow in order to preserve the former state. If it happens that there's not enough space on the snapshot LV to record further changes, the snap volume will just disappear, or become unusable.

A summary of the options is as follows: lvcreate is known, the -L1G is also known, but now there's this -s option. The -s tells LVM to make a snapshot. We follow that with -n for a name, and then follow the order: snapshot name - volume to be snapped. So we told LVM to snap root, make the snapshot 1GB and name the snapshot volume snap. I keep talking about the 1GB value, however, if we look at lvdisplay, we'll see something different:

root@mrdeb: lvdisplay 
--- Logical volume ---
  LV Name                /dev/debian/root
  VG Name                debian
  LV UUID                LTACtF-y34L-MZeo-QQ9E-HXCc-LYBB-RAOWTc
  LV Write Access        read/write
  LV snapshot status     source of
                         /dev/debian/snap [active]
  LV Status              available
  # open                 1
  LV Size                4.66 GiB
  
 --- Logical volume ---
  LV Name                /dev/debian/snap
  VG Name                debian
  LV UUID                psjsBS-XIel-klcr-nbzI-gFlb-wIXu-Q6qqRB
  LV Write Access        read/write
  LV snapshot status     active destination for /dev/debian/root
  LV Status              available
  # open                 0
  LV Size                4.66 GiB
  Current LE             1192
  COW-table size         1.00 GiB
  COW-table LE           512
  Allocated to snapshot  0.13% 
  Snapshot chunk size    4.00 KiB
  Segments               1

As you see, /dev/debian/root is listed as the source for /dev/debian/snap. But, /dev/debian/snap shows that it's 4,66 GB big, not 1GB. Did LVM eat your disk? No, LVM doesn't really care how big you made the snapshot, it only shows the relation to the source of the snapshot. Now, we'll use fsarchiver on the live system.

root@mrdeb ~ # fsarchiver savefs /mnt/backup/lvmbackup.fsa /dev/debian/snap

The syntax is thus: /mountpoint/name.fsa /partition/to/backup. So in this case, we're backing up /dev/debian/snap to the /mnt/backup mountpoint where we've put our backup LV, and we're calling the new backup lvmbackup.fsa. The process will take anywhere from 2 minutes to half an hour, depending on the power of your CPU. In case you need to restore, you'll need a live CD (I recommend grml, but any rescue CD - Parted Magic, System Rescue Cd, Knoppix... - will do):

root@grml: mkdir -p /mnt/restore
root@grml: modprobe dm-mod
root@grml: lvscan
root@grml: lvchange -ay
root@grml: mount /dev/debian/backup /mnt/restore
root@grml: fsarchiver restfs /mnt/restore/lvmbackup.fsa id=0,dest=/dev/debian/root

Now, snapshots incur overhead to the system, as every change to the source volume will be written to the source volume and to the snapshot. So, once you've completed the backup, remove the snapshot:

root@mrdeb ~ # lvremove /dev/debian/snap
Do you really want to remove active logical volume snap? [y/n]: y
  Logical volume "snap" successfully removed

---
And this is the end of el_koraco's Monumental LVM Guide. Any further posts are from me (johnraff).

johnraff · 2020-07-28 09:02:35

(Now posting as johnraff again.)
bit more on backups

The smart daemon has been emaiing me about unreadable sectors on my ssd for a while now, for a long time "2" but this week "3". Knowing from experience that ssd's can catastrophically fail without warning, I've got even more interested in backups the last few days.

el_k's guide was spot-on, and confirmed by general searching, that an LVM snapshot is the way to get a stable source for backups, especially if it's of your root system.

fsarchiver
All seems to be according to the docs, but:

1) Different compression settings don't seem to make such a huge difference to the archive size, though I haven't yet tried the highest (-z 9).
eg backing up a root partition holding ~18GB of data:
-z 7 -j 3 (7th compression level, 3 CPU cores) made a 10.6GB archive in ~12min.
-z 1 (lowest compression, only one core) gave 12.3GB in ~2.8min.
So unless you really need the absolutely smallest archives, I'd go for the speed of -z 1
But my testing was on Debian Stretch. On Buster, fsarchiver supports zstd compression which looks much faster, so maybe reconsider the above.

2) There doesn't seem to be any "from file" option for excludes, so my full command line was:

sudo fsarchiver savefs --overwrite -z 1 /mnt/backup/fsarchiver/root.fsa /dev/ssd/rootsnap -e '/bulk/*' -e '/data/*' -e '/dev/*' -e '/media/*' -e '/mnt/*' -e '/lost+found' -e '/proc/*' -e '/run/*' -e '/sys/*' -e '/tmp/*'

which is a bit unwieldy.

3) Unlike with rsync, you can't "refresh" an archive, so each one has to be made anew, though you can overwrite existing archive files.

rsync
fsarchiver is great for making a backup you can load straight back into the drive (if you mess up your system) or into a new one after that catastrophic failure, but my machine has a big 2TB hard disk as well as the 128GB sdd. The hdd seems OK for the moment, but backing that up will mean buying another 2TB disk. Anyway, there's plenty of space there for emergency backups of the sdd, so right now I'm working on a script to rsync the data from the ssd into the hdd on every shutdown. Once the rsync mirror is created it only takes a few seconds to refresh it. This isn't so much about protecting myself against my own messups (they do occur though) but about having an insurance when the ssd fails, so I think a single mirror with rsync's --del option is enough. Right?

The script isn't made yet but the basic commands it will run (plus error checks of course) are something like this (my ssd is volume group "ssd" with logical volumes "helium" and "data", hard disk' vg is "hdd"):

# just once:
sudo lvcreate -L 100G -n backup /dev/hdd
sudo mkfs.ext4 /dev/hdd/backup
sudo mkdir -p /mnt/backup
sudo mount /dev/hdd/backup /mnt/backup
sudo mkdir -p /mnt/backup/rsync
sudo mkdir -p /mnt/backup/fsarchiver

# every shutdown:

sudo lvcreate -L 5G --snapshot -n datasnap /dev/ssd/data
sudo mount /dev/ssd/datasnap /mnt/snapshot
sudo mount /dev/hdd/backup /mnt/backup
sudo mkdir -p /mnt/backup/rsync/data_bkp
sudo rsync -aAX --del --exclude-from=$HOME/.config/user/backup_data_excludes /mnt/snapshot/ /mnt/backup/rsync/data_bkp/
sudo umount /mnt/snapshot
sudo lvremove /dev/ssd/datasnap

sudo lvcreate -L 5G --snapshot -n rootsnap /dev/ssd/helium
sudo mount /dev/ssd/rootsnap /mnt/snapshot
sudo mount /dev/hdd/backup /mnt/backup
sudo mkdir -p /mnt/backup/rsync/root_bkp
sudo rsync -aAX --del --exclude-from=$HOME/.config/user/backup_root_excludes /mnt/snapshot/ /mnt/backup/rsync/root_bkp/
# backup_root_excludes excludes /home so it's done separately
sudo mkdir -p /mnt/backup/rsync/home_bkp
sudo rsync -aAX --del --exclude-from=$HOME/.config/user/backup_home_excludes /mnt/snapshot/home/ /mnt/backup/rsync/home_bkp/
sudo umount /mnt/snapshot
sudo lvremove /dev/ssd/rootsnap

I'm hoping this won't slow down my shutdown too much.

Last edited by johnraff (2020-07-29 07:01:25)

#1 2020-07-28 07:06:35

LVM guide (by el_koraco)

#2 2020-07-28 07:18:50

Re: LVM guide (by el_koraco)

#3 2020-07-28 07:58:21

Re: LVM guide (by el_koraco)

#4 2020-07-28 08:07:01

Re: LVM guide (by el_koraco)

#5 2020-07-28 08:23:01

Re: LVM guide (by el_koraco)

#6 2020-07-28 09:02:35

Re: LVM guide (by el_koraco)

Board footer