Handy command-line stuff for terminals or scripts

ohnonot · 2018-05-23 07:15:32

misko_2083 wrote:

I use arrays too much.

posix compliance is overrated.
i mean, python isn't posix compliant either.
arrays are the only thing that make me, a poor bash hacker, feel like a real coder.

johnraff · 2018-09-28 08:47:17

Split a long string by multiple regex matches

Most of the familiar utilities - sed, grep & friends - deal with data a line at a time, but more and more these days line breaks are becoming irrelevant, and parsing something like a config file harder, at least with shell commands. Just now I was playing with a bash function to parse a conky 10 style config file. Running down it with read works as long as the writer formatted it nicely with each entry on its own line, but in fact conky is quite happy with something like:

 --[[ conky configuration
 Index » Scripts, Tutorials & Tips » Show us your conky
 ]] conky.config
  = { imlib_cache_size = 0,
--  Window Settings
    own_window = true ,own_window_type = 'normal',
    own_window_hints = 'undecorated,above,skip_taskbar,skip_pager,sticky',
-- (more follows)

So you see the 'conky.config = {' is split over two lines, and follows right after the comment block ]], the first config entry is right after the config opening '{' and two entries are on the same line. You can't work down this line by line, so I tried squashing all the config entries into a single line:

imlib_cache_size = 0,own_window = true,own_window_type = 'normal',own_window_transparent = true,own_window_hints = 'undecorated,above,skip_taskbar,skip_pager,sticky',color0 = "B0E0E6",gap_x = 20,gap_y = 45,alignment = 'top_right',double_buffer = true,

With that, at first it looks as if you could just split it into individual entries using the ',' as a delimiter, but what about those commas inside quotes like own_window_hints = 'undecorated,above,skip_taskbar,skip_pager,sticky',?

You either need to learn lua or Python, or persistently stick to Bash and use a regular expression. This works and the Bash array BASH_REMATCH helps. It's a pity Bash won't work down the string putting every occurrence of a regex into the array, but only the first match, but there's a way out - get the first match, then chop that bit off the front of the string and try the regex again. This way you can get every occurrence, and using inner brackets ( ) you can pick out sub-elements of the match without any further processing.

So in this particular case:

line="imlib_cache_size = 0,own_window = true,own_window_type = 'normal',own_window_transparent = true,own_window_hints = 'undecorated,above,skip_taskbar,skip_pager,sticky',color0 = \"B0E0E6\",gap_x = 20,gap_y = 45,alignment = 'top_right',double_buffer = true,"
regex="([[:alnum:]_]+)[[:blank:]]*=[[:blank:]]*([[:alnum:]]+|'[^']+'|\"[^\"]+\")"
while [[ $line =~ $regex ]] ; do echo "name: ${BASH_REMATCH[1]}"; echo "value: ${BASH_REMATCH[2]}"; line=${line#*${BASH_REMATCH[0]}}; done
name: imlib_cache_size
value: 0
name: own_window
value: true
name: own_window_type
value: 'normal'
name: own_window_transparent
value: true
name: own_window_hints
value: 'undecorated,above,skip_taskbar,skip_pager,sticky'
name: color0
value: "B0E0E6"
name: gap_x
value: 20
name: gap_y
value: 45
name: alignment
value: 'top_right'
name: double_buffer
value: true

I was quite happy to get this working, and it might come in useful again some day.

Original idea here: https://unix.stackexchange.com/a/251031

BTW Don't try to use this to parse html or XML or all the wrath of the interwebs will be visited upon you.

johnraff · 2019-02-16 05:44:57

Dynamically named variables

Ran into this the other day.
Sometimes in a script you'd like to create a variable whose name is decided in the script, not set by you. Like:
'varname=$(some process); value=$(other process)'
You can't just do:

$varname=$value

(try it)
But luckily it's not that hard - either declare or printf will work:

declare $varname=$value
printf -v "$varname" '%s' "$value"

---
Another way you might sometimes prefer is to create an associative array, with the variable name going in as the key:

declare -A variables
variables[$varname]=$value

iMBeCil · 2019-02-16 14:30:51

^Thanks johnraff ... Have to try it, and it works, even the variable substitution strategy on the left side works:

#!/bin/bash
#
# This is 'test' script.

varname="CoolVarName"
value="CoolValue"

# Note 'my' in new variable name:
declare "my$varname"=$value
# OR:
#printf -v "my$varname" '%s' "$value" 

declare -p | grep Cool

This gives:

$ ./test
declare -- _="myCoolVarName=CoolValue"
declare -- myCoolVarName="CoolValue"
declare -- value="CoolValue"
declare -- varname="CoolVarName"

It shows that new declared variable really is 'myCoolVarName'.

Good to know ... although I think people writing such code are not liked too much I presume that readability-wise it is much more sane to use associative array approach.

Last edited by iMBeCil (2019-02-16 14:31:15)

ohnonot · 2019-03-28 06:51:18

^ and a pipemenu that makes use of the "list vms" and "startvm" commands:

#!/bin/sh

# OpenBox "VirtualBox" pipe-menu
# Outputs a menu roughly akin to:
#
#    VirtualBox
#    ----------
#    <Virtual Machine>
#    <Virtual Machine>
#    <Virtual Machine>
#    ...
#
# Usage:
#
# 1. Copy this file somewhere on your path and make it executable
# 2. Add the following line somewhere to your /.config/openbox/menu.xml
#
#    <menu id="vms" label="Virtual Machines" execute="$HOME/.config/openbox/pipemenus/vb-pipemenu" />
#
# 3. Reconfigure openbox

# output the initial menu
cat <<EOF
<openbox_pipe_menu>
    <item label="VirtualBox">
        <action name="Execute">
            <command>
                virtualbox
            </command>
        </action>
    </item>
EOF

# seperate the main command from the virtuals
echo "    <separator/>"

# output the list of virtual machines
IFS='"'
VBoxManage list -s vms | while read empty vm checksum
do
    label="${vm//_/-}"
    cat <<EOF
    <item label="$label">
        <action name="Execute">
            <command>
                vboxmanage startvm "$vm"
            </command>
        </action>
    </item>
EOF
done;

# and finally...
echo "</openbox_pipe_menu>"

Last edited by ohnonot (2019-03-28 06:52:59)

beaker · 2019-04-03 18:07:35

I just ran into a really handy tip for displaying saved Network Manager wireless passwords.

In a terminal

sudo grep psk= /etc/NetworkManager/system-connections/*

ohnonot · 2019-04-03 18:11:53

beaker wrote:

I just ran into a really handy tip for displaying saved Network Manager wireless passwords.

do i sense a hint of sarcasm there?
:devil:

Last edited by ohnonot (2019-04-03 18:12:21)

ohnonot · 2019-04-13 20:24:17

for a long time i've been using a little script to download music with youtube-dl, straight into my Music folder.
been trying to perfect it.

youtube-dl can extract a lot of info from the download, and generate a suitable download path and filename for one or multiple files.
ideally for me, that would be:

music_base_directory/artist/album/title.ext

but unfortunately there's no way to get that info in a straightforward manner from all the different sites.

youtube is not artist-centric at all
soundcloud or bandcamp treat album/artist differently from each other

so my script still isn't perfect, but sufficiently so that i can automate the task of just right-clicking a link and download everything (complete albums, playlists, even one artists complete works), and still find the music afterwards.

so:

for soundcloud or bandcamp it is fully automatic
for youtube it asks to provide the name of a folder (typically the artist)
everything else is treated like youtube

to make it better i would have to download metadata first, make some intelligent decisions, and only then download the actual media, and i did not want to do that (who knows when the site decides to throttle or block you, better not overdo it).

the script takes one link at a time, e.g.

youtube-dl-music https://www.youtube.com/watch?v=iYBz_xvUa_8

you should change the base directory of where you want downloads to go.

Last edited by ohnonot (2021-07-08 09:31:24)

hhh · 2019-04-14 05:39:08

@ohnonot, looking forward to trying this on lithium, thanks for the great work!

johnraff · 2020-02-10 02:40:52

tree

Everyone knows this already no doubt, but I just ran into it today. What's more, already had it installed.
Just run 'tree' to see a pretty display of directories and contents, starting from wherever you are.
Very nice, and much quicker than opening a file manager.

(The package is also called "tree" and it came as a dependency of inxi so Lithium users will have it already.)

damo · 2020-02-13 05:54:00

From man bash CONDITIONAL EXPRESSIONS

-t fd  True if file descriptor fd is open and refers to a terminal.

So I wanted to display a script's help when run from the menu or a keybind, by echoing it to notify-send. I found this gem:

# test if 'script --help' was run in terminal, or by menu/gui/keybind
if [[ -t 1 ]];then     # see man bash CONDITIONAL EXPRESSIONS
    echo -e "${USAGE}"    # echo to terminal
else
    notify-send "$(echo -e ${USAGE})"  # show in notification
fi

or in @johnraff's preferred coding style:

[[ -t 1 ]] && echo -e "${USAGE}" || notify-send "$(echo -e ${USAGE})"

Result

johnraff · 2020-02-13 07:37:55

damo wrote:

or in @johnraff's preferred coding style:
[[ -t 1 ]] && echo -e "${USAGE}" || notify-send "$(echo -e ${USAGE})"

Code whatever way you like, and I'll do the same.

Actually, I generally prefer this && that one-liners mainly for very simple choices like error checking. In a case like the above where two different actions are to be taken - and might have other stuff added in the future - I'd quite likely go for the if this; then that syntax too.

But who knows, when I think no-one's watching...

ohnonot · 2020-02-13 18:28:17

1. && binds stronger than ||, so it's like

(A && B) || C

2. if you have only one && it's fairly easy, but if you want to chain 3 or more conditionals, you have to be absolutely sure they will return either 0 or 1 (true or false) the way you want them to:

[ -r $file ] && cat $file && exit 1

Can you be 100% sure that cat will always return 0 whenever $file is readable? If not, it wouldn't exit the script and all sorts of mischief could happen.

johnraff · 2020-02-14 02:56:22

ohnonot wrote:

1. && binds stronger than ||, so it's like
(A && B) || C

You can think about it that way, but shell conditionals finally made sense to me when I realized (read somewhere) that they don't work like IF constructions in most other languages. In every case the return value of a command is being inspected. Nothing more or less. Even [ is a command! (Equivalent to test, though [[ is subtly different.)

So && means "if the last executed command returns 0 then execute the following command."
and || means "if the last executed command returns 1 then execute the following command".

So you can figure out what happens in every case with A && B || C:
A succeeds > B executes
B fails > C executes
A fails > (B does not execute) C executes

Brackets unnecessary.

if you have only one && it's fairly easy, but if you want to chain 3 or more conditionals, you have to be absolutely sure they will return either 0 or 1 (true or false) the way you want them to:
[ -r $file ] && cat $file && exit 1
Can you be 100% sure that cat will always return 0 whenever $file is readable? If not, it wouldn't exit the script and all sorts of mischief could happen.

Agree absolutely, and what I posted above more-or-less.
I generally use only A && B or A || B constructs. Sometimes condition && action || error message but only if failure of condition also needs to trigger an error.
Otherwise you can use curly brackets to keep the error test only for action:

condition && { action || error_message ;}

Anything more complicated than that, the IF constructions are much easier to keep under control.

Last edited by johnraff (2020-02-14 03:58:23)

johnraff · 2020-02-14 04:05:18

BTW I was reminded that even conditionals are commands a few days ago when trying to use the Bash array PIPESTATUS. One test on eg ${PIPESTATUS[0]} and the whole array is rewritten with the return value of that test, replacing the pipe you wanted to look at.

The fix is to copy PIPESTATUS into a regular array immediately after running the pipe, then you can test its contents at leisure. https://www.shellscript.sh/tips/pipestatus/

ohnonot · 2020-02-14 06:47:29

^ Nice one about PIPESTATUS.

johnraff wrote:

Brackets unnecessary.

Those were only for explanation, not actual usage.

Tell me, how does this work:

A || B && C

???

johnraff · 2020-02-14 06:57:28

^
A returns 0:
B not run
C runs (the last command ran returned 0)

A returns 1:
B runs
B returns 0:
C runs
B returns 1:
C not run

---
About brackets: I meant that they were not necessary (or even helpful) for understanding the logic flow.

johnraff · 2020-02-22 03:57:33

Forking

An ever-popular subject. The net is full of people asking how to fork off processes in the background from a shell script. As everyone knows, basically you just add a & after the command. It gets more complicated if you want the process to go on running after your terminal has closed, or the calling script has exited.

Google, and you'll find:

nohup command &

command & 
disown

setsid command &

# and subshells etc
( nohup command & )
( nohup command & ) &
( setsid command & )

in increasingly desperate attempts to get that pesky process sent off entirely by itself. Take a few minutes to try them in a terminal, eg

mousepad &

of course, close the terminal and mousepad dies too.

mousepad &
disown

This time mousepad survives closing the terminal. This is strong too:

setsid mousepad

But in scripts, things are different.
This is where I was stuck, and almost giving up. In a script, nohup, disown and setsid no longer work the way they did from a terminal. If a script contains eg

nohup tint2 &

tint2 might stay running if you trigger the script from a launcher or menu and it then exits, but if you run it in a terminal and hit Ctrl+C to close the script while its still running, the SIGINT or SIGTERM signal will be passed on to tint2 which stops even though it's supposed to be backgrounded.

It's because (I just learned) Job Control is not enabled in scripts by default - it's mainly intended for interactive shells. But, you can enable it by putting

set -m

at the top of your script, or - maybe safer - just around the part where you launch the background process:

set -m
do launch-in-background stuff
set +m

Now disown and friends will work the same way they do in a terminal.

Some pages that came up:
https://unix.stackexchange.com/question … ses-within
https://unix.stackexchange.com/question … set-m-does
https://medium.com/@copyconstruct/bash- … 36da3e4aa7
https://mywiki.wooledge.org/BashGuide/JobControl

rbh · 2020-02-22 09:31:59

But aint it easier to use tools like screen or tmux?

malm · 2020-02-22 09:33:26

Awesome. Didn’t know that.

#81 2018-05-23 07:15:32

Re: Handy command-line stuff for terminals or scripts

#82 2018-09-28 08:47:17

Re: Handy command-line stuff for terminals or scripts

#83 2019-02-16 05:44:57

Re: Handy command-line stuff for terminals or scripts

#84 2019-02-16 14:30:51

Re: Handy command-line stuff for terminals or scripts

#85 2019-03-28 06:51:18

Re: Handy command-line stuff for terminals or scripts

#86 2019-04-03 18:07:35

Re: Handy command-line stuff for terminals or scripts

#87 2019-04-03 18:11:53

Re: Handy command-line stuff for terminals or scripts

#88 2019-04-13 20:24:17

Re: Handy command-line stuff for terminals or scripts

#89 2019-04-14 05:39:08

Re: Handy command-line stuff for terminals or scripts

#90 2020-02-10 02:40:52

Re: Handy command-line stuff for terminals or scripts

#91 2020-02-13 05:54:00

Re: Handy command-line stuff for terminals or scripts

#92 2020-02-13 07:37:55

Re: Handy command-line stuff for terminals or scripts

#93 2020-02-13 18:28:17

Re: Handy command-line stuff for terminals or scripts

#94 2020-02-14 02:56:22

Re: Handy command-line stuff for terminals or scripts

#95 2020-02-14 04:05:18

Re: Handy command-line stuff for terminals or scripts

#96 2020-02-14 06:47:29

Re: Handy command-line stuff for terminals or scripts

#97 2020-02-14 06:57:28

Re: Handy command-line stuff for terminals or scripts

#98 2020-02-22 03:57:33

Re: Handy command-line stuff for terminals or scripts

#99 2020-02-22 09:31:59

Re: Handy command-line stuff for terminals or scripts

#100 2020-02-22 09:33:26

Re: Handy command-line stuff for terminals or scripts

Board footer