You are not logged in.

#1 2019-12-08 08:54:51

clusterF
Member
Registered: 2019-05-07
Posts: 458

[WIP] Sort the Downloads folder (sort.sh)

Not sure what to call this script yet but so far it is working. Maybe there is already a bash script out there for this.

Before i go any further i just wanted some feedback on how to proceed further, is there any danger of a mass mv of various files together, so far ive only tested .txt and .mp3 files but im wondering how to go about mv multiple file extensions at once? Maybe rsync could be a better option. Im also thinking maybe a bash array for the extensions but im relatively new to bash scripting and have no idea on arrays.

My idea is thus...

1. cd to Downloads
2. move various file extensions for example, .mov .webm .mp3 .flac .docx .odt .txt .pdf etc to designated folders.

#!/usr/bin/env bash
#set -e

dls () {
        cd $HOME/Downloads
}

mvtxt () {
        mv *.txt $HOME/Documents
}

mvmp3 () {
        mv *.mp3 $HOME/Music
}

if      dls; then
        mvtxt
        mvmp3
fi
        exit 1

edit: please read entire thread as modifications abound.....

Last edited by clusterF (2019-12-09 10:25:58)


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#2 2019-12-08 09:22:50

damo
....moderator....
Registered: 2015-08-20
Posts: 5,419

Re: [WIP] Sort the Downloads folder (sort.sh)

One approach is to do something like (pseudocode-ish)

for f in Downloads/*;do
  ext=$(basename "$f")
  ext="${ext#*.}"
  case e in "$ext"
    zip|tar  )  mv "$f" ~/somewhere;;
    mov|avi|mp4 )  mv "$f" $(xdg-user-dir VIDEOS);;
    jpg|png|tiff ) mv "$f" $(xdg-user-dir PICTURES);;
  esac
done

This needs better extraction of extensions BTW - it doesn't account for multiple dots for example.


Be Excellent to Each Other...

FORUM RULES and posting guidelines «» Help page for forum post formatting
Artwork on DeviantArt  «» BunsenLabs on DeviantArt

Offline

#3 2019-12-08 10:09:13

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

^ thanks for that.

rsync commands might be better to protect destination files through the backup flag, mv can do this as well but the man page says does not accept arguments where as rsync does.

such as.

rsync -bvh --progress --remove-source-files *.txt $HOME/Documents


rsync -bvh --progress --remove-source-files *.mp3 $HOME/Music

Last edited by clusterF (2019-12-08 10:12:43)


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#4 2019-12-08 12:06:24

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

researched bash arrays a little bit and came up with below. Bit weird behavior, forgets the .txt file but every other file ext is ok and moves to the designated folders. Staying with mv command for now as i dont need to pass any other backup arguments at this stage.

edit: weird behavior vanished after retry with newly created files.

#!/usr/bin/env bash

arr=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
arr1=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )

dls () {
        cd $HOME/Downloads
}

docs () {
        mv -b ${arr[@]} $HOME/Documents 2> /dev/null
}

music () {
        mv -b ${arr1[@]} $HOME/Music 2> /dev/null
}

if      dls; then
        docs
        music
fi
        exit 1

edited/updated.

Tested and working so far, below code snippets in aid of populating the download folders with files. Now need to somehow test with larger files.

~/Downloads:$ touch test.doc test.docx test.ebook test.log test.md test.msg test.odt test.org test.pages test.pdf test.rtf test.rst test.tex test.txt test.wpd test.wps 

~/Downloads:$ touch test.aac test.aiff test.ape test.au test.flac test.gsm test.it test.m3u test.m4a test.mid test.mod test.mp3 test.mpa test.pls test.ra test.s3m test.sid test.wav test.wma test.xm

Last edited by clusterF (2019-12-08 13:53:21)


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#5 2019-12-08 14:23:36

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,194
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

You should quote the array expansions or any file names with spaces will break the script.
ie "${arr[@]}" instead of ${arr[@]}

Why are you throwing away error messages with '2> /dev/null'? Normally, there shouldn't be any.

Also 'exit 1' at the end means your script always reports an error to any process that calls it. In your case this is probably never noticed, but I don't think you need an 'exit' command at the end at all. It will exit naturally once all the commands have been executed, and the return value will be that of the last command (usually 0).


John

...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Offline

#6 2019-12-08 14:46:59

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

^ thanks for your input johnraff

Im not sure they are error messages.

mv: cannot stat '*.doc': No such file or directory

"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#7 2019-12-08 15:19:07

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,194
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

Try:

shopt -s nullglob
# array definitions
arr=( *.doc *.docx... etc
shopt -u nullglob

See: https://www.endpoint.com/blog/2016/12/1 … b-failglob

Last edited by johnraff (2019-12-09 01:19:09)


John

...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Offline

#8 2019-12-08 17:46:25

eight.bit.al
Member
From: The top of the World
Registered: 2015-10-01
Posts: 379

Re: [WIP] Sort the Downloads folder (sort.sh)


A mind is a terrible thing not to change.
The problem lies not with new ideas, but in escaping the old ones.

Offline

#9 2019-12-08 21:42:45

Naik
Member
From: Lipsia
Registered: 2015-10-03
Posts: 196

Re: [WIP] Sort the Downloads folder (sort.sh)

Hey,

I tried the approach with rsync and think this a great idea it works very well.

#!/usr/bin/env bash
# sort the download folder usin rsync
# inspired by ClusterF

# filetypes to sort
arr=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
arr1=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm *.ogg )
arr2=( *.jpg *.JPG *.png *.xfc *.bmp *.gif )

# change this to use it to sort any folder you like
dls () {
        cd $HOME/Downloads
}

# how to process and where to put the various filetypes
docs () {
        rsync -bvh --progress --remove-source-files ${arr[@]} $HOME/Documents 2> /dev/null
}

music () {
        rsync -bvh --progress --remove-source-files -b ${arr1[@]} $HOME/Music 2> /dev/null
}

iso () {
		rsync -bvh --progress --remove-source-files -b *.iso $HOME/Downloads/iso 2> /dev/null
}

mc () {
		rsync -bvh --progress --remove-source-files -b *.jar $HOME/.minecraft/mods 2> /dev/null
}

pix () {
		rsync -bvh --progress --remove-source-files -b ${arr2[@]} $HOME/Pictures/unsorted/ 2> /dev/null
}

deb () {
		rsync -bvh --progress --remove-source-files -b *.deb $HOME/bin 2> /dev/null
}

# ..and go
if      dls; then
        docs
        music
        iso
        mc
        pix
        deb
fi
        exit 1

Would you mind me handing this over to the folks following the great yad hacking thread?
I think it would be great to have a little gui making it possible to choose which file-types to process.

naik --greetz


"Kaum macht [Mensch]* es richtig, funktioniert es sofort!"
BL-Kitchen on GitHub

Offline

#10 2019-12-08 22:08:31

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

Thanks again johnraff, i will look into this again after i finish work.

@ eight.bit.al, thanks ill check it out.

@ Naik, thanks and of course do what you like with it, i would be interested to see something made in yad for it.


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#11 2019-12-09 10:24:03

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

ok so the "shopt -s nullglob" & "shopt -u nullglob" stop the script from working johnraff. Not sure why as i dont get any sort of error messages in the terminal but the script does not complete and the files stay in Downloads.

back to using rsync to move files as ive now created a backup directory and redirected backups from destination to %HOME/.sort.sh_backups/, ,this is just in case of overwrites.

also better naming and some comments thanks to naik.

still a WIP....

#!/usr/bin/env bash
set -e


# filetypes to sort
doc_array=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
music_array=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )


# change this to use it to sort any folder you like
sortfolder () {
        cd $HOME/Downloads
}

# how to process and where to put the various filetypes
docs () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${doc_array[@]} $HOME/Documents 2> /dev/null
}

music () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${music_array[@]} $HOME/Music 2> /dev/null
}

if      sortfolder; then
	mkdir -p $HOME/.sort.sh_backups/       
	docs
        music
fi

"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#12 2019-12-09 20:09:44

brontosaurusrex
Middle Office
Registered: 2015-09-29
Posts: 1,966
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

clusterF wrote:

--backup-dir=

This will backup files with same name that would otherwise be overwritten? (I was never a friend with man rsync)

Online

#13 2019-12-10 03:16:27

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,194
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

clusterF wrote:

ok so the "shopt -s nullglob" & "shopt -u nullglob" stop the script from working johnraff. Not sure why as i dont get any sort of error messages in the terminal but the script does not complete and the files stay in Downloads.

I see it now. Your array elements with asterisks are trying to look for files that match the glob at the point when you define the array ie before moving to the download folder, probably in ~/. If there are no files that match, then the asterisk will be preserved as-is, and can be used later with mv. However, when you set nullglob any non-matching globs are replaced with null, so when you try to move files nothing will happen.

What would work (if you still want to play with this) would be to fill the arrays after you arrive in Downloads, so maybe make a new function:

fillarrays () {
        shopt -s nullglob
        arr=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
        arr1=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )
        shopt -u nullglob
}

And the end of your script could be:

if      dls; then
        fillarrays
        docs
        music
fi

John

...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Offline

#14 2019-12-10 11:56:47

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

@ brotosaurusrex, as far as im aware it will. So if i have a file of the same name + extension by coincidence in Documents that is getting rsynced from the source directory Downloads it will back it up into a backup directory of your choosing instead of backing it up in the destination directory or overwriting the destination file of the same name and extension.

Thanks johnraff, that is a much better solution and can confirm working.

updated script....

#!/usr/bin/env bash
set -e

fillarrays () {
        shopt -s nullglob
        doc_array=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
        music_array=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )
        shopt -u nullglob
}

# change this to use it to sort any folder you like
sortfolder () {
        cd $HOME/Downloads
}

# how to process and where to put the various filetypes
docs () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${doc_array[@]} $HOME/Documents
}

music () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/ ${music_array[@]} $HOME/Music
}

if      sortfolder; then
	mkdir -p $HOME/.sort.sh_backups/
	fillarrays
	docs
        music
fi

Last edited by clusterF (2019-12-10 11:59:06)


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#15 2019-12-10 13:21:40

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

This program from dyne seems like a better solution to this script.

https://github.com/dyne/harvest


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#16 2019-12-10 14:22:24

earlybird
ほやほや
Registered: 2015-12-16
Posts: 738
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

clusterF wrote:

This program from dyne seems like a better solution to this script.

https://github.com/dyne/harvest

That one goes by extensions only. I personally had a script based on file --mime-type (that is, based on libmagic recognition) but stopped using it since the auto-organization is absolutely not smart enough.

KDE (and GNOME) have a semantic desktop based on file tags.

It would be for example wonderful if there was a service (for example, based on libinotify and awareness in the desktop environment of the fact which application is saving a file) if a file that was downloaded using a browser and a corresponding tag "download" getting applied automatically. In addition, for sorting, the DE would also apply tags like mime type (is an image, is a document...) and in the end, all files would be browsable in "views" on a hierarchy of a given search result of a search over tags.

Like GMail with labels, and notmuch tags for email.

Offline

#17 2019-12-10 14:46:16

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

^ that sounds interesting. So having the browser aware of where certain file extensions should belong auto magically instead of just choosing Downloads or save to. Would have to be very integrated into the desktop i would imagine.


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#18 2019-12-10 15:07:02

damo
....moderator....
Registered: 2015-08-20
Posts: 5,419

Re: [WIP] Sort the Downloads folder (sort.sh)

Having stuff arrive in Downloads is handy to keep everything in one place though. I copy out what I want to keep, and clear it out occasionally.

I absolutely don't want files to automagically be sent somewhere and have to hunt for them.


Be Excellent to Each Other...

FORUM RULES and posting guidelines «» Help page for forum post formatting
Artwork on DeviantArt  «» BunsenLabs on DeviantArt

Offline

#19 2019-12-11 02:17:50

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,194
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

^Also having Downloads as a staging area makes it quick and easy to run a virus scan before moving things elsewhere.


John

...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Offline

#20 2019-12-11 10:01:57

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

@ damo, would make more sense if browsers had a better user experience where it knows what kind of file type you are downloading and prompts you if you would like to put say a .txt file Documents or .jpg in Pictures and so on. Auto magically probably prone to some sort of bugginess for sure.

Anyhow i have created a bash menu, it was a good learning experience this and will add more extensions in due course.

#!/usr/bin/env bash
# bash menu to move/sort various file extensions from $HOME/Downloads to common $HOME/Directories.
set -e

# get the file extensions
fillarrays () {
shopt -s nullglob
doc_array=( *.doc *.docx *.ebook *.log *.md *.msg *.odt *.org *.pages *.pdf *.rtf *.rst *.tex *.txt *.wpd *.wps ) 
music_array=( *.aac *.aiff *.ape *.au *.flac *.gsm *.it *.m3u *.m4a *.mid *.mod *.mp3 *.mpa *.pls *.ra *.s3m *.sid *.wav *.wma *.xm )
video_array=( *.3g2 *.3gp *.aaf *.asf *.avchd *.avi *.drc *.flv *.m2v *.m4p *.m4v *.mkv *.mng *.mov *.mp2 *.mp4 *.mpe *.mpeg *.mpg *.mpv *.mxf *.nsv *.ogg *.ogv *.ogm *.qt *.rm *.rmvb *.roq *.srt *.svi *.vob *.webm *.wmv *.yuv )
pictures_array=( *.3dm *.3ds *.max *.bmp *.dds *.gif *.jpg *.jpeg *.png *.psd *.xcf *.tga *.thm *.tif *.tiff *.yuv *.ai *.eps *.ps *.svg *.dwg *.dxf *.gpx *.kml *.kmz *.webp )
shopt -u nullglob
}

# use rsync to move extensions and back them up (if necessary) in case of over writing in destination directory
docs () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Documents/ ${doc_array[@]} $HOME/Documents
}

music () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Music/ ${music_array[@]} $HOME/Music
}

videos () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Videos/ ${video_array[@]} $HOME/Videos
}

pictures () {
        rsync -bvh --progress --remove-source-files --backup-dir=$HOME/.sort.sh_backups/Pictures/ ${pictures_array[@]} $HOME/Pictures
}

#the source directory to move extensions
sortfolder () {
        cd $HOME/Downloads
}

# create the menu
echo "Bash Menu To Clean Up The Downloads Folder"
PS3='Please enter a number of your choice: '
options=("Documents" "Music" "Videos" "Pictures" "Quit")
select opt in "${options[@]}"
do
    case $opt in
        "Documents")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Documents/       
				docs
		    fi
            ;;
        "Music")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Music/       
				music
		    fi
            ;;
        "Videos")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Videos/       
				videos
		    fi
            ;;
        "Pictures")
            if  sortfolder; then
				fillarrays
				mkdir -p $HOME/.sort.sh_backups/Pictures/       
				pictures
		    fi
            ;;
        "Quit")
            break
            ;;
        *) echo "invalid option $REPLY";;
    esac
done

Last edited by clusterF (2019-12-11 10:10:05)


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#21 2019-12-12 06:27:06

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,194
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

One point is that especially on Unix-based systems, the file extension is pretty much meaningless. Any kind of file can be given any extension. Web servers usually send the MIME-type, but that can also be wrong sometimes. On your system the command 'file' will actually look inside the file and try to figure it out. Maybe a script could use the output of 'file --mime-type --brief $file' ...


John

...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Offline

#22 2019-12-12 09:37:23

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

^ that is probably a better approach to this for sure johnraff, i will have to look into that.

Just playing around with your suggestion and this one liner finds all text files in a folder.

file --mime * | grep "text"

edit: this one a bit better.

file --mime-type * | grep "text" | awk '{print $1}' ORS=' ' | tr -d {[:]}

Bit more progress using mv

#!/usr/bin/env bash
set -e

find_files=$(file --mime-type $HOME/Downloads/* | grep text | awk '{print $1}' ORS=' ' | tr -d {[:]})

move_files () {
        mv $find_files $HOME/Documents/

}

move_files

much simpler it seems so far.

Last edited by clusterF (2019-12-12 12:14:26)


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#23 2019-12-12 14:13:34

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

^ until one runs into file names with spaces....?


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

#24 2019-12-13 01:45:01

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,194
Website

Re: [WIP] Sort the Downloads folder (sort.sh)

^Yes that.

Also, 'file --mime-type --brief' (as I suggested above) might be easier for a script to use, possibly no need to call grep.


John

...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Offline

#25 2019-12-13 10:27:40

clusterF
Member
Registered: 2019-05-07
Posts: 458

Re: [WIP] Sort the Downloads folder (sort.sh)

johnraff wrote:

^Yes that.

Also, 'file --mime-type --brief' (as I suggested above) might be easier for a script to use, possibly no need to call grep.

youtube-dl likes to put spaces in the file names when it downloads, so i probably need to stop that from happening somehow as normally i dont allow spaces in file names. edit: seems there is a flag "--restrict-filenames" in youtube-dl for us unix users which will put underscores where white spaces are.

if i use the --brief flag i dont get the filename just the file attributes eg; for .txt it will "text/plain"

file --mime-type --brief $HOME/Downloads/FILE | grep text | awk '{print $1}' ORS=' ' | tr -d {[:]}

text/plain

and if i leave out "grep text" it just grabs all filenames and attributes in the folder.

Last edited by clusterF (2019-12-13 10:48:34)


"Common sense is like deodorant, those who need it the most never use it."

git: clusterF

Offline

Board footer

Powered by FluxBB