You are not logged in.

#1 2019-09-30 13:54:42

achim
Member
Registered: 2019-08-23
Posts: 16

Error messages when copying with rsync

Hello everybody,

I let my phone read out by means of a self-created script. The process now takes several minutes because rsync detected problems with 2 jpg files. I have the message written to an Err file, which then displays the following content:

rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A002%2C009%5D/Phone/DCIM/Camera/20190614_104730-neu.jpg": Input/output error (5)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A002%2C009%5D/Phone/DCIM/Camera/20190614_104730.jpg": Input/output error (5)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A002%2C009%5D/Phone/DCIM/Camera/20190614_104730-neu.jpg": Input/output error (5)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A002%2C009%5D/Phone/DCIM/Camera/20190614_104730.jpg": Input/output error (5)
ERROR: 20190614_104730-neu.jpg failed verification -- update discarded.
ERROR: 20190614_104730.jpg failed verification -- update discarded.
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]

The code for reading looks like this:

#!/bin/sh

quelle=/run/user/1000/gvfs/mtp:host=%5Busb%3A002%2C009%5D/Phone/DCIM/Camera/

ziel=$HOME/Bilder/Import/
heute=$(date +%Y-%m-%d)

rsync -aq --delete "${quelle}"  "${ziel}${heute}/" --link-dest="${ziel}last/" 2>rsync.err
ln -nsf "${ziel}${heute}" "${ziel}last"

exit 0

Since rsync is busy for a long time to read out the faulty files, I would like to change the code so that the affected files are skipped as soon as detected and a message is written to the log file. I gladly accept every suggestion.

A hard disk error is excluded, because the process by deleting the files extremely fast expires.

My system: Linux Mint 18.3 Cinnamon 64 bit


Best regards

achim

Offline

#2 2019-09-30 16:56:38

damo
....moderator....
Registered: 2015-08-20
Posts: 5,208

Re: Error messages when copying with rsync

Is it always the same files? If so, why not just use

--exclude=PATTERN       exclude files matching PATTERN

Be Excellent to Each Other...

FORUM RULES and posting guidelines «» Help page for forum post formatting
Artwork on DeviantArt  «» BunsenLabs on DeviantArt

Offline

#3 2019-09-30 17:40:43

achim
Member
Registered: 2019-08-23
Posts: 16

Re: Error messages when copying with rsync

These are different, damaged image files. Since I do not know the reason for the damage, I always have to reckon with the mistake. The option --exclude is unfortunately not available here.
The faulty files can be copied to the hard disk with copy and paste after the script has finished.

Offline

#4 2019-09-30 17:59:35

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,543

Re: Error messages when copying with rsync

Hmmm.

The faulty files can be copied to the hard disk with copy and paste after the script has finished.

Could you take a checksum of the files on the source medium and then after copy-pasting on the target medium? Are they different or identical?

Also, what happens when you do use the inefficient mode

rsync --whole-file --append

for copying? By default, rsync depends on being able to read subregions of source files for its xfer algorithm a lot; I can imagine that MTP in this case with these specific files or the gvfs MTP implementation messes some things up.

Edit: If the source is an Android phone, perhaps take the checksum using some file manager application to make sure MTP does not get in the way.


New Horizons

Offline

#5 2019-09-30 18:53:06

damo
....moderator....
Registered: 2015-08-20
Posts: 5,208

Re: Error messages when copying with rsync

Another way may be to open the offending files in an image editor, and do Save/Export As. It may fix the format?


Be Excellent to Each Other...

FORUM RULES and posting guidelines «» Help page for forum post formatting
Artwork on DeviantArt  «» BunsenLabs on DeviantArt

Offline

#6 2019-10-01 02:00:34

achim
Member
Registered: 2019-08-23
Posts: 16

Re: Error messages when copying with rsync

@twoion
Unfortunately, I did not succeed with your suggestions.

@damo
With your suggestion it was possible to repair the files on the hard disk!

However, I am still looking for the possibility that after detecting the error to the next file is changed. Like this: If error then next smile

achim

Offline

#7 2019-10-03 08:38:36

ohnonot
...again
Registered: 2015-09-29
Posts: 4,135
Website

Re: Error messages when copying with rsync

Always the same files?
Why not just delete them from the phone?

Offline

#8 2019-10-03 11:33:28

achim
Member
Registered: 2019-08-23
Posts: 16

Re: Error messages when copying with rsync

Yes, they are always the same files and they can be repaired on the hard drive after the hint from damo. I do not want to delete it.
My goal is (as already written in message no. 6) in the script to include the possibility that when the error is detected, the file is simply skipped. And maybe at the end it will be displayed as "defective".

Offline

#9 2019-10-05 09:12:43

ohnonot
...again
Registered: 2015-09-29
Posts: 4,135
Website

Re: Error messages when copying with rsync

^ Maybe rsync has an option for it.
'--keep-going-on-error' or sth like that.
If not it will be difficult to script, since you basically have to check for every file if it's defective, before rsyncing it.

But IMO the chance of this happening (again) is so low that a manual is sufficient.

Offline

#10 2019-10-05 09:32:54

clusterF
Member
Registered: 2019-05-07
Posts: 260

Re: Error messages when copying with rsync

^ that sounds about right.

man rsync...

--ignore-errors         delete even if there are I/O errors
        -e, --rsh=COMMAND           specify the remote shell to use
            --rsync-path=PROGRAM    specify the rsync to run on remote machine
            --existing              skip creating new files on receiver
            --ignore-existing       skip updating files that exist on receiver
            --remove-source-files   sender removes synchronized files (non-dir)
            --del                   an alias for --delete-during
            --delete                delete extraneous files from dest dirs
            --delete-before         receiver deletes before xfer, not during
            --delete-during         receiver deletes during the transfer
            --delete-delay          find deletions during, delete after
            --delete-after          receiver deletes after transfer, not during
            --delete-excluded       also delete excluded files from dest dirs
            --ignore-missing-args   ignore missing source args without error
            --delete-missing-args   delete missing source args from destination
            --ignore-errors         delete even if there are I/O errors
            --force                 force deletion of dirs even if not empty
            --max-delete=NUM        don't delete more than NUM files
            --max-size=SIZE         don't transfer any file larger than SIZE
            --min-size=SIZE         don't transfer any file smaller than SIZE
            --partial               keep partially transferred files
            --partial-dir=DIR       put a partially transferred file into DIR
            --delay-updates         put all updated files into place at end

--delete

If  the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. This is to prevent temporary filesystem failures (such as NFS errors) on the sending side from causing a massive deletion of files on the destination. You can override this with the --ignore-errors option.

Last edited by clusterF (2019-10-05 09:42:34)


"Thou Wanderer, Father of all! Thou Wanderer, Spirit of all!"

Offline

#11 2019-10-05 10:49:15

achim
Member
Registered: 2019-08-23
Posts: 16

Re: Error messages when copying with rsync

Thank you, clusterF. I think that way the script will work without problems.

achim

Offline

Board footer

Powered by FluxBB