[WIP] Sort the Downloads folder (sort.sh)

clusterF · 2019-12-08 08:54:51

Not sure what to call this script yet but so far it is working. Maybe there is already a bash script out there for this.

Before i go any further i just wanted some feedback on how to proceed further, is there any danger of a mass mv of various files together, so far ive only tested .txt and .mp3 files but im wondering how to go about mv multiple file extensions at once? Maybe rsync could be a better option. Im also thinking maybe a bash array for the extensions but im relatively new to bash scripting and have no idea on arrays.

My idea is thus...

1. cd to Downloads
2. move various file extensions for example, .mov .webm .mp3 .flac .docx .odt .txt .pdf etc to designated folders.

#!/usr/bin/env bash
#set -e

dls () {
        cd $HOME/Downloads
}

mvtxt () {
        mv *.txt $HOME/Documents
}

mvmp3 () {
        mv *.mp3 $HOME/Music
}

if      dls; then
        mvtxt
        mvmp3
fi
        exit 1

edit: please read entire thread as modifications abound.....

Last edited by clusterF (2019-12-09 10:25:58)

damo · 2019-12-08 09:22:50

One approach is to do something like (pseudocode-ish)

for f in Downloads/*;do
  ext=$(basename "$f")
  ext="${ext#*.}"
  case e in "$ext"
    zip|tar  )  mv "$f" ~/somewhere;;
    mov|avi|mp4 )  mv "$f" $(xdg-user-dir VIDEOS);;
    jpg|png|tiff ) mv "$f" $(xdg-user-dir PICTURES);;
  esac
done

This needs better extraction of extensions BTW - it doesn't account for multiple dots for example.

clusterF · 2019-12-08 10:09:13

^ thanks for that.

rsync commands might be better to protect destination files through the backup flag, mv can do this as well but the man page says does not accept arguments where as rsync does.

such as.

rsync -bvh --progress --remove-source-files *.txt $HOME/Documents


rsync -bvh --progress --remove-source-files *.mp3 $HOME/Music

Last edited by clusterF (2019-12-08 10:12:43)

clusterF · 2019-12-08 12:06:24

researched bash arrays a little bit and came up with below. Bit weird behavior, forgets the .txt file but every other file ext is ok and moves to the designated folders. Staying with mv command for now as i dont need to pass any other backup arguments at this stage.

edit: weird behavior vanished after retry with newly created files.

#!/usr/bin/env bash

arr=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
arr1=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )

dls () {
        cd $HOME/Downloads
}

docs () {
        mv -b ${arr[@]} $HOME/Documents 2> /dev/null
}

music () {
        mv -b ${arr1[@]} $HOME/Music 2> /dev/null
}

if      dls; then
        docs
        music
fi
        exit 1

edited/updated.

Tested and working so far, below code snippets in aid of populating the download folders with files. Now need to somehow test with larger files.

~/Downloads:$ touch test.doc test.docx test.ebook test.log test.md test.msg test.odt test.org test.pages test.pdf test.rtf test.rst test.tex test.txt test.wpd test.wps 

~/Downloads:$ touch test.aac test.aiff test.ape test.au test.flac test.gsm test.it test.m3u test.m4a test.mid test.mod test.mp3 test.mpa test.pls test.ra test.s3m test.sid test.wav test.wma test.xm

Last edited by clusterF (2019-12-08 13:53:21)

johnraff · 2019-12-08 14:23:36

You should quote the array expansions or any file names with spaces will break the script.
ie "${arr[@]}" instead of ${arr[@]}

Why are you throwing away error messages with '2> /dev/null'? Normally, there shouldn't be any.

Also 'exit 1' at the end means your script always reports an error to any process that calls it. In your case this is probably never noticed, but I don't think you need an 'exit' command at the end at all. It will exit naturally once all the commands have been executed, and the return value will be that of the last command (usually 0).

clusterF · 2019-12-08 14:46:59

^ thanks for your input johnraff

Im not sure they are error messages.

mv: cannot stat '*.doc': No such file or directory

johnraff · 2019-12-08 15:19:07

Try:

shopt -s nullglob
# array definitions
arr=( *.doc *.docx... etc
shopt -u nullglob

See: https://www.endpoint.com/blog/2016/12/1 … b-failglob

Last edited by johnraff (2019-12-09 01:19:09)

deleted0 · 2019-12-08 17:46:25

Is this any help?

https://distrowatch.com/weekly.php?issue=20191021#qa

8bit

Naik · 2019-12-08 21:42:45

Hey,

I tried the approach with rsync and think this a great idea it works very well.

#!/usr/bin/env bash
# sort the download folder usin rsync
# inspired by ClusterF

# filetypes to sort
arr=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
arr1=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm *.ogg )
arr2=( *.jpg *.JPG *.png *.xfc *.bmp *.gif )

# change this to use it to sort any folder you like
dls () {
        cd $HOME/Downloads
}

# how to process and where to put the various filetypes
docs () {
        rsync -bvh --progress --remove-source-files ${arr[@]} $HOME/Documents 2> /dev/null
}

music () {
        rsync -bvh --progress --remove-source-files -b ${arr1[@]} $HOME/Music 2> /dev/null
}

iso () {
		rsync -bvh --progress --remove-source-files -b *.iso $HOME/Downloads/iso 2> /dev/null
}

mc () {
		rsync -bvh --progress --remove-source-files -b *.jar $HOME/.minecraft/mods 2> /dev/null
}

pix () {
		rsync -bvh --progress --remove-source-files -b ${arr2[@]} $HOME/Pictures/unsorted/ 2> /dev/null
}

deb () {
		rsync -bvh --progress --remove-source-files -b *.deb $HOME/bin 2> /dev/null
}

# ..and go
if      dls; then
        docs
        music
        iso
        mc
        pix
        deb
fi
        exit 1

Would you mind me handing this over to the folks following the great yad hacking thread?
I think it would be great to have a little gui making it possible to choose which file-types to process.

naik --greetz

clusterF · 2019-12-08 22:08:31

Thanks again johnraff, i will look into this again after i finish work.

@ eight.bit.al, thanks ill check it out.

@ Naik, thanks and of course do what you like with it, i would be interested to see something made in yad for it.

clusterF · 2019-12-09 10:24:03

ok so the "shopt -s nullglob" & "shopt -u nullglob" stop the script from working johnraff. Not sure why as i dont get any sort of error messages in the terminal but the script does not complete and the files stay in Downloads.

back to using rsync to move files as ive now created a backup directory and redirected backups from destination to %HOME/.sort.sh_backups/, ,this is just in case of overwrites.

also better naming and some comments thanks to naik.

still a WIP....

#!/usr/bin/env bash
set -e


# filetypes to sort
doc_array=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
music_array=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )


# change this to use it to sort any folder you like
sortfolder () {
        cd $HOME/Downloads
}

# how to process and where to put the various filetypes
docs () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${doc_array[@]} $HOME/Documents 2> /dev/null
}

music () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${music_array[@]} $HOME/Music 2> /dev/null
}

if      sortfolder; then
	mkdir -p $HOME/.sort.sh_backups/       
	docs
        music
fi

brontosaurusrex · 2019-12-09 20:09:44

clusterF wrote:

--backup-dir=

This will backup files with same name that would otherwise be overwritten? (I was never a friend with man rsync)

johnraff · 2019-12-10 03:16:27

clusterF wrote:

ok so the "shopt -s nullglob" & "shopt -u nullglob" stop the script from working johnraff. Not sure why as i dont get any sort of error messages in the terminal but the script does not complete and the files stay in Downloads.

I see it now. Your array elements with asterisks are trying to look for files that match the glob at the point when you define the array ie before moving to the download folder, probably in ~/. If there are no files that match, then the asterisk will be preserved as-is, and can be used later with mv. However, when you set nullglob any non-matching globs are replaced with null, so when you try to move files nothing will happen.

What would work (if you still want to play with this) would be to fill the arrays after you arrive in Downloads, so maybe make a new function:

fillarrays () {
        shopt -s nullglob
        arr=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
        arr1=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )
        shopt -u nullglob
}

And the end of your script could be:

if      dls; then
        fillarrays
        docs
        music
fi

clusterF · 2019-12-10 11:56:47

@ brotosaurusrex, as far as im aware it will. So if i have a file of the same name + extension by coincidence in Documents that is getting rsynced from the source directory Downloads it will back it up into a backup directory of your choosing instead of backing it up in the destination directory or overwriting the destination file of the same name and extension.

Thanks johnraff, that is a much better solution and can confirm working.

updated script....

#!/usr/bin/env bash
set -e

fillarrays () {
        shopt -s nullglob
        doc_array=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
        music_array=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )
        shopt -u nullglob
}

# change this to use it to sort any folder you like
sortfolder () {
        cd $HOME/Downloads
}

# how to process and where to put the various filetypes
docs () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${doc_array[@]} $HOME/Documents
}

music () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${music_array[@]} $HOME/Music
}

if      sortfolder; then
	mkdir -p $HOME/.sort.sh_backups/
	fillarrays
	docs
        music
fi

Last edited by clusterF (2019-12-10 11:59:06)

clusterF · 2019-12-10 13:21:40

This program from dyne seems like a better solution to this script.

https://github.com/dyne/harvest

clusterF · 2019-12-10 14:46:16

^ that sounds interesting. So having the browser aware of where certain file extensions should belong auto magically instead of just choosing Downloads or save to. Would have to be very integrated into the desktop i would imagine.

damo · 2019-12-10 15:07:02

Having stuff arrive in Downloads is handy to keep everything in one place though. I copy out what I want to keep, and clear it out occasionally.

I absolutely don't want files to automagically be sent somewhere and have to hunt for them.

johnraff · 2019-12-11 02:17:50

^Also having Downloads as a staging area makes it quick and easy to run a virus scan before moving things elsewhere.

clusterF · 2019-12-11 10:01:57

@ damo, would make more sense if browsers had a better user experience where it knows what kind of file type you are downloading and prompts you if you would like to put say a .txt file Documents or .jpg in Pictures and so on. Auto magically probably prone to some sort of bugginess for sure.

Anyhow i have created a bash menu, it was a good learning experience this and will add more extensions in due course.

#!/usr/bin/env bash
# bash menu to move/sort various file extensions from $HOME/Downloads to common $HOME/Directories.
set -e

# get the file extensions
fillarrays () {
shopt -s nullglob
doc_array=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
music_array=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )
video_array=( *.3g2 *.3gp *.aaf *.asf *.avchd *.avi *.drc *.flv *.m2v *.m4p *.m4v *.mkv *.mng *.mov *.mp2 *.mp4 *.mpe *.mpeg *.mpg *.mpv *.mxf *.nsv *.ogg *.ogv *.ogm *.qt *.rm *.rmvb *.roq *.srt *.svi *.vob *.webm *.wmv *.yuv )
pictures_array=( *.3dm *.3ds *.max *.bmp *.dds *.gif *.jpg *.jpeg *.png *.psd *.xcf *.tga *.thm *.tif *.tiff *.yuv *.ai *.eps *.ps *.svg *.dwg *.dxf *.gpx *.kml *.kmz *.webp )
shopt -u nullglob
}

# use rsync to move extensions and back them up (if necessary) in case of over writing in destination directory
docs () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Documents/ ${doc_array[@]} $HOME/Documents
}

music () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Music/ ${music_array[@]} $HOME/Music
}

videos () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Videos/ ${video_array[@]} $HOME/Videos
}

pictures () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Pictures/ ${pictures_array[@]} $HOME/Pictures
}

#the source directory to move extensions
sortfolder () {
        cd $HOME/Downloads
}

# create the menu
echo "Bash Menu To Clean Up The Downloads Folder"
PS3='Please enter a number of your choice: '
options=("Documents" "Music" "Videos" "Pictures" "Quit")
select opt in "${options[@]}"
do
    case $opt in
        "Documents")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Documents/       
				docs
		    fi
            ;;
        "Music")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Music/       
				music
		    fi
            ;;
        "Videos")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Videos/       
				videos
		    fi
            ;;
        "Pictures")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Pictures/       
				pictures
		    fi
            ;;
        "Quit")
            break
            ;;
        *) echo "invalid option $REPLY";;
    esac
done

Last edited by clusterF (2019-12-11 10:10:05)

johnraff · 2019-12-12 06:27:06

One point is that especially on Unix-based systems, the file extension is pretty much meaningless. Any kind of file can be given any extension. Web servers usually send the MIME-type, but that can also be wrong sometimes. On your system the command 'file' will actually look inside the file and try to figure it out. Maybe a script could use the output of 'file --mime-type --brief $file' ...

#1 2019-12-08 08:54:51

[WIP] Sort the Downloads folder (sort.sh)

#2 2019-12-08 09:22:50

Re: [WIP] Sort the Downloads folder (sort.sh)

#3 2019-12-08 10:09:13

Re: [WIP] Sort the Downloads folder (sort.sh)

#4 2019-12-08 12:06:24

Re: [WIP] Sort the Downloads folder (sort.sh)

#5 2019-12-08 14:23:36

Re: [WIP] Sort the Downloads folder (sort.sh)

#6 2019-12-08 14:46:59

Re: [WIP] Sort the Downloads folder (sort.sh)

#7 2019-12-08 15:19:07

Re: [WIP] Sort the Downloads folder (sort.sh)

#8 2019-12-08 17:46:25

Re: [WIP] Sort the Downloads folder (sort.sh)

#9 2019-12-08 21:42:45

Re: [WIP] Sort the Downloads folder (sort.sh)

#10 2019-12-08 22:08:31

Re: [WIP] Sort the Downloads folder (sort.sh)

#11 2019-12-09 10:24:03

Re: [WIP] Sort the Downloads folder (sort.sh)

#12 2019-12-09 20:09:44

Re: [WIP] Sort the Downloads folder (sort.sh)

#13 2019-12-10 03:16:27

Re: [WIP] Sort the Downloads folder (sort.sh)

#14 2019-12-10 11:56:47

Re: [WIP] Sort the Downloads folder (sort.sh)

#15 2019-12-10 13:21:40

Re: [WIP] Sort the Downloads folder (sort.sh)

#16 2019-12-10 14:46:16

Re: [WIP] Sort the Downloads folder (sort.sh)

#17 2019-12-10 15:07:02

Re: [WIP] Sort the Downloads folder (sort.sh)

#18 2019-12-11 02:17:50

Re: [WIP] Sort the Downloads folder (sort.sh)

#19 2019-12-11 10:01:57

Re: [WIP] Sort the Downloads folder (sort.sh)

#20 2019-12-12 06:27:06

Re: [WIP] Sort the Downloads folder (sort.sh)

Board footer