rsync backup hangs on / and /home backup [SOLVED]]

christopherisnow · 2021-02-24 11:14:53

Trying to backup / and /home from a fresh Lithium install on a new ThinkPad x270 with rsync to an external drive formatted with EXFAT. It hangs in a few places.

1. Backing up /home always hangs on a maildir file (or another one if I exclude the first culprit) in `.local/share/mail/account/cur/`. Excluding `.local/share/mail/` produces a clean backup. I've found a workaround by making a tarball of that directory before syncing, but it would be nice to know why I can't copy it as is. The current command is

rsync -av --progress --delete --exclude '.local/share/mail' /home/christopher /media/christopher/easystore/bak_new/christopher

2. Backing up / at first hangs at `usr/share/man/man3/XML::LibXML::Parser.3pm.gz`, but when I exclude that specific file, it hangs on some other file in `usr/share/man/man3/`. Excluding `usr/share/man/man3/` entirely causes rsync to hang on `var/lib/dpkg/info/libepoxy0:amd64.symbols`. Excluding that file or its directory just causes a hang somehwere new, and so on ad nauseum. This is particularly confusing since I just installed Lithium last week. Here the command is

rsync -av / --exclude={"/dev/*","/home/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","usr/share/man/man3/*","var/lib/dpkg/info/libepoxy0:amd64.symbols"} /media/christopher/easystore/bak_new/root --delete

I'm wondering if this is due to corrupt files, a faulty command, or hardware. Since I just bought the laptop secondhand, it would be good to know ASAP if the latter is the problem so I can take it back!

Last edited by christopherisnow (2021-02-28 15:10:32)

tknomanzr · 2021-02-24 17:02:17

You can use smartctl out of the smartmontools package to check the drive's health.

christopherisnow · 2021-02-24 19:43:13

The smartctl readout seems fine, so I'll assume for the moment that's not the problem.

After some tedious step-by-step testing, it's pretty clear the usr/share/man/man3 issues have something to do with Perl, perhaps package conflicts. A selection of some offending files:

usr/share/man/man3/DateTime::Locale::kw.3pm.gz
usr/share/man/man3/DateTime::Locale::luo.3pm.gz
usr/share/man/man3/Locale::RecodeData::EBCDIC_UK.3pm.gz
usr/share/man/man3/Map.Make.3o.gz
usr/share/man/man3/Locale*
usr/share/man/man3/Net::DNS::RR::AAAA.3pm.gz
usr/share/man/man3/Net::DNS::Mailbox.3pm.gz
usr/share/man/man3/Net::DNS::Nameserver.3pm.gz

Any idea what that's about?

The other problematic directory is var/lib/dpkg/info/. Here a sampling of nontransferable files:

var/lib/dpkg/info/libdns1104:amd64.symbols
var/lib/dpkg/info/libdvdread4:amd64.symbols
var/lib/dpkg/info/libdw1:amd64.list
var/lib/dpkg/info/libdw1:amd64.md5sums
var/lib/dpkg/info/libdw1:amd64.symbols
var/lib/dpkg/info/libdynaloader-functions-perl.md5sums
var/lib/dpkg/info/libe-book-0.1-1:amd64.list
var/lib/dpkg/info/libe-book-0.1-1:amd64.md5sums

I tried reinstalling a few of those packages (libdns1104, libdvdread4), but it didn't help.

When I exclude var/lib/dpkg/info/* and usr/share/man/man3/* everything copies properly.

Do these issues have anything in common?

Last edited by christopherisnow (2021-02-24 19:45:40)

tknomanzr · 2021-02-24 19:55:47

christopherisnow wrote:

The smartctl readout seems fine, so I'll assume for the moment that's not the problem.
After some tedious step-by-step testing, it's pretty clear the usr/share/man/man3 issues have something to do with Perl, perhaps package conflicts. A selection of some offending files:
usr/share/man/man3/DateTime::Locale::kw.3pm.gz
usr/share/man/man3/DateTime::Locale::luo.3pm.gz
usr/share/man/man3/Locale::RecodeData::EBCDIC_UK.3pm.gz
usr/share/man/man3/Map.Make.3o.gz
usr/share/man/man3/Locale*
usr/share/man/man3/Net::DNS::RR::AAAA.3pm.gz
usr/share/man/man3/Net::DNS::Mailbox.3pm.gz
usr/share/man/man3/Net::DNS::Nameserver.3pm.gz
Any idea what that's about?
The other problematic directory is var/lib/dpkg/info/. Here a sampling of nontransferable files:
var/lib/dpkg/info/libdns1104:amd64.symbols
var/lib/dpkg/info/libdvdread4:amd64.symbols
var/lib/dpkg/info/libdw1:amd64.list
var/lib/dpkg/info/libdw1:amd64.md5sums
var/lib/dpkg/info/libdw1:amd64.symbols
var/lib/dpkg/info/libdynaloader-functions-perl.md5sums
var/lib/dpkg/info/libe-book-0.1-1:amd64.list
var/lib/dpkg/info/libe-book-0.1-1:amd64.md5sums
I tried reinstalling a few of those packages (libdns1104, libdvdread4), but it didn't help.
When I exclude var/lib/dpkg/info/* and usr/share/man/man3/* everything copies properly.
Do these issues have anything in common?

Those dpkg entries are concerning because I believe they will lead to issues later on if you ever need to use that backup. Have you tried reinstalling those packages?

christopherisnow · 2021-02-24 20:04:37

Those dpkg entries are concerning because I believe they will lead to issues later on if you ever need to use that backup. Have you tried reinstalling those packages?

I tried reinstalling libdns1104 and libdvdread4, but it didn't fix the problem. I still can't copy them:

rsync -av /var/lib/dpkg/info/libdvdread4:amd64.symbols /media/christopher/easystore/bak_new/
sending incremental file list
libdvdread4:amd64.symbols
rsync: mkstemp "/media/christopher/easystore/bak_new/.libdvdread4:amd64.symbols.yC4n0T" failed: No such file or directory (2)

sent 1,729 bytes  received 165 bytes  3,788.00 bytes/sec
total size is 1,624  speedup is 0.86
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]

I'm flummoxed how I could break something so quickly -- just installed Lithium last week!

Last edited by christopherisnow (2021-02-24 20:05:48)

christopherisnow · 2021-02-25 23:57:14

Both issues solved... sort of.

1. It seems archiving Maildirs is the common way of backing them up in any case. So I go with the flow and can now copy /home to an external hard drive.

2. It was definitely a Perl conflict run amok. After manually deleting a bunch of modules installed with apt, the man3 issues resolved themselves. Some of the packages creating the problem in /var/lib/dpkg/info/ could be reinstalled, thus preventing the rsync hang... but some not.

I ended up just reinstalling Lithium and will be more conservative about Perl modules (which I rarely need anyway) in the future. It didn't seem worth the tedium.

It *would* be nice to know what might have happened to /var/lib/dpkg/info/*, if anyone reconizes this mishap.

christopherisnow · 2021-02-26 16:11:02

I reinstalled Lithium and Perl problems are gone. But after installing some standard packages with apt, texlive2020, skypeforlinux, and a few off-the-shelf .debs I'm still hanging on files in /var/lib/dpkg/info/:

libdbus-1-3:amd64.symbols   RI
libepoxy0:amd64.symbols 
libexif12:amd64.symbols
libexiv2-14:amd64.symbols
libexo-1-0:amd64.list       RI
libexo-1-0:amd64.md5sums    RI
libexo-1-0:amd64.symbols    RI
libexo-helpers.list         RI
libdns1104:amd64.symbols

RI marks packages I upgraded or reinstalled that no longer hang. There are plenty more.

What on earth could cause this problem on a clean install? Can I safely exclude the .symbols packages from a backup?

sleekmason · 2021-02-26 18:50:44

I've been looking at doing this as well.

A quick search pulled up a couple of possibilities concerning rsync and exfat.

basically, one says to use the -t option for correct times, and the other says that exfat cannot use the -a option at all, which I notice you are using.

I have no idea if this is still the case or not, but you looked stuck:)

https://superuser.com/questions/763366/ … e-on-exfat
https://stackoverflow.com/questions/326 … xfat-fails

In any event, it seem using exfat has caused a problem or two.

christopherisnow · 2021-02-26 21:38:58

Thanks for these links about exfat. I actually don't have problems backing up /home (or so it seems) or anything in / except /var, even with the -a option, so I wonder what makes /var so special here. This article discusses /var backup problems with rsync, but doesn't explain them: https://unix.stackexchange.com/question … 596#636596

I'm beginning to think it's academic, though, and I'd be better off backing up / as a tarball or similar. I don't need easy access to the backup contents anyway (which I *would* like for /home), except to restore.

Perhaps I need to set up a dedicated backup partition in in exfat and/or do a bit more research about backups in general. There's a lot of opinions out there, but if anyone has a simple favorite script for backing up a laptop onto a portable drive as described above, I'd be happy to read it.

sleekmason · 2021-02-26 22:04:34

Have you used luckybackup before? I tried it years ago with success. Uses rsync.
* Edit - Sounds like you want a script rather than a full package though. Well, just in case.

http://luckybackup.sourceforge.net/
https://packages.debian.org/buster/utils/luckybackup

sudo apt install luckybackup

Last edited by sleekmason (2021-02-26 22:22:15)

christopherisnow · 2021-02-28 15:09:24

Actually I prefer to stick with tried and true command line programs like rsync and tar. Just needed to brush up on some details.

I settled on a non-incremental backup of / with tar, where the two most recent versions are kept and anything older gets deleted.

tar -czpf $TARGET/root_$DATE.tar.gz --exclude={"/dev/*","/home/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found/*"} /

if [ `find $TARGET -name 'root*.tar.gz' | wc -l ` -gt 2 ]; then
    cd $TARGET && rm "$(ls -t | tail -1)"
fi

/home gets backed up with rsync. I feel more comfortable being able to access files in a portable drive on the fly without having to extract a tarball.

# /home
rsync -av -H --progress --delete --exclude={'.local/share/mail/*','.local/share/Trash/*'} $SOURCE $TARGET/

The Maildirs in .local/share/mail get backed up as tarballs in their normal parent directory. I also backup the empty subdirectory structure there so mbsync knows where to put everything when syncing IMAP accounts from remote servers.

# Archive Maildirs
cd $SOURCE/.local/share/mail
find . -maxdepth 10 -mindepth 1 -type d -not -name '.*' -exec rsync -av * --include '*/' --exclude '*' ./* $TARGET/christopher/.local/share/mail/ \;
find . -maxdepth 1 -mindepth 1 -type d -not -name '.*' -exec tar -czpf $TARGET/christopher/.local/share/mail/{}.tar.gz {}  \;
rsync -av .notmuch /media/christopher/easystore/bak_new/christopher/.local/share/mail/

Most important takeaway here is that EXFAT turns out to be the wrong drive format for the job on many levels. It doesn't doesn't recognize permissions or symlinks. I'm pretty sure it's also the cause of my woes with /var. I'll create a new ext4 partition on my external drive just for the backup.

DeepDayze · 2021-02-28 16:35:52

Can't go wrong with LuckyBackup and you can run it as user to backup what you want in your $HOME or run as superuser to backup the rootfs.

sleekmason · 2021-02-28 19:26:20

^^ Yep.

christopherisnow wrote:

Actually I prefer to stick with tried and true command line programs like rsync and tar. Just needed to brush up on some details.
<snip>
Most important takeaway here is that EXFAT turns out to be the wrong drive format for the job on many levels. It doesn't doesn't recognize permissions or symlinks. I'm pretty sure it's also the cause of my woes with /var. I'll create a new ext4 partition on my external drive just for the backup.

So it was indeed Exfat that was causing the problem after all. Good to know. Glad you found a solution that works for you.

DeepDayze · 2021-02-28 19:40:16

sleekmason wrote:

^^ Yep.
christopherisnow wrote:
Actually I prefer to stick with tried and true command line programs like rsync and tar. Just needed to brush up on some details.
<snip>
Most important takeaway here is that EXFAT turns out to be the wrong drive format for the job on many levels. It doesn't doesn't recognize permissions or symlinks. I'm pretty sure it's also the cause of my woes with /var. I'll create a new ext4 partition on my external drive just for the backup.
So it was indeed Exfat that was causing the problem after all. Good to know. Glad you found a solution that works for you.

Not a good idea to use a filesystem like exfat, FAT32 or even NTFS for your Linux rsync backups anyway IMO. Ext2/3/4 are the way to go.

#1 2021-02-24 11:14:53

rsync backup hangs on / and /home backup [SOLVED]]

#2 2021-02-24 17:02:17

Re: rsync backup hangs on / and /home backup [SOLVED]]

#3 2021-02-24 19:43:13

Re: rsync backup hangs on / and /home backup [SOLVED]]

#4 2021-02-24 19:55:47

Re: rsync backup hangs on / and /home backup [SOLVED]]

#5 2021-02-24 20:04:37

Re: rsync backup hangs on / and /home backup [SOLVED]]

#6 2021-02-25 23:57:14

Re: rsync backup hangs on / and /home backup [SOLVED]]

#7 2021-02-26 16:11:02

Re: rsync backup hangs on / and /home backup [SOLVED]]

#8 2021-02-26 18:50:44

Re: rsync backup hangs on / and /home backup [SOLVED]]

#9 2021-02-26 21:38:58

Re: rsync backup hangs on / and /home backup [SOLVED]]

#10 2021-02-26 22:04:34

Re: rsync backup hangs on / and /home backup [SOLVED]]

#11 2021-02-28 15:09:24

Re: rsync backup hangs on / and /home backup [SOLVED]]

#12 2021-02-28 16:35:52

Re: rsync backup hangs on / and /home backup [SOLVED]]

#13 2021-02-28 19:26:20

Re: rsync backup hangs on / and /home backup [SOLVED]]

#14 2021-02-28 19:40:16

Re: rsync backup hangs on / and /home backup [SOLVED]]

Board footer