You are not logged in.

#101 2020-02-14 06:47:29

ohnonot
...again
Registered: 2015-09-29
Posts: 4,632
Website

Re: Handy command-line stuff for terminals or scripts

^ Nice one about PIPESTATUS.

johnraff wrote:

Brackets unnecessary.

Those were only for explanation, not actual usage.

Tell me, how does this work:

A || B && C

???


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#102 2020-02-14 06:57:28

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,951
Website

Re: Handy command-line stuff for terminals or scripts

^
A returns 0:
    B not run
    C runs (the last command ran returned 0)

A returns 1:
    B runs
    B returns 0:
        C runs
    B returns 1:
        C not run

---
About brackets: I meant that they were not necessary (or even helpful) for understanding the logic flow.


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#103 2020-02-22 03:57:33

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,951
Website

Re: Handy command-line stuff for terminals or scripts

Forking

An ever-popular subject. The net is full of people asking how to fork off processes in the background from a shell script. As everyone knows, basically you just add a & after the command. It gets more complicated if you want the process to go on running after your terminal has closed, or the calling script has exited.

Google, and you'll find:

nohup command &

command & 
disown

setsid command &

# and subshells etc
( nohup command & )
( nohup command & ) &
( setsid command & )

in increasingly desperate attempts to get that pesky process sent off entirely by itself. Take a few minutes to try them in a terminal, eg

mousepad &

of course, close the terminal and mousepad dies too.

mousepad &
disown

This time mousepad survives closing the terminal. This is strong too:

setsid mousepad

But in scripts, things are different.
This is where I was stuck, and almost giving up. In a script, nohup, disown and setsid no longer work the way they did from a terminal. If a script contains eg

nohup tint2 &

tint2 might stay running if you trigger the script from a launcher or menu and it then exits, but if you run it in a terminal and hit Ctrl+C to close the script while its still running, the SIGINT or SIGTERM signal will be passed on to tint2 which stops even though it's supposed to be backgrounded. neutral

It's because (I just learned) Job Control is not enabled in scripts by default - it's mainly intended for interactive shells. But, you can enable it by putting

set -m

at the top of your script, or - maybe safer - just around the part where you launch the background process:

set -m
do launch-in-background stuff
set +m

Now disown and friends will work the same way they do in a terminal.

Some pages that came up:
https://unix.stackexchange.com/question … ses-within
https://unix.stackexchange.com/question … set-m-does
https://medium.com/@copyconstruct/bash- … 36da3e4aa7
https://mywiki.wooledge.org/BashGuide/JobControl


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#104 2020-02-22 09:31:59

rbh
Member
From: Sweden/Vasterbotten/Rusfors
Registered: 2016-08-11
Posts: 447

Re: Handy command-line stuff for terminals or scripts

But aint it easier to use tools like screen or tmux?


// Regards rbh

Online

#105 2020-02-22 09:33:26

malm
jgmenu developer
Registered: 2016-10-13
Posts: 602
Website

Re: Handy command-line stuff for terminals or scripts

Awesome. Didn’t know that.

Offline

#106 2020-02-22 09:46:52

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,951
Website

Re: Handy command-line stuff for terminals or scripts

rbh wrote:

But aint it easier to use tools like screen or tmux?

Not necessarily, depending on how familiar you are with those tools, and how complex the task is. If you just want to make sure conky isn't killed or something like that, the shell solution is more appropriate IMO.


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#107 2020-02-22 10:39:30

clusterF
Member
Registered: 2019-05-07
Posts: 539

Re: Handy command-line stuff for terminals or scripts

Nice tips there johnraff, i have used disown for quite awhile when testing things out, especially conky.

Offline

#108 2020-02-24 07:14:46

ohnonot
...again
Registered: 2015-09-29
Posts: 4,632
Website

Re: Handy command-line stuff for terminals or scripts

johnraff wrote:

Job Control is not enabled in scripts by default - it's mainly intended for interactive shells. But, you can enable it by putting

set -m

at the top of your script, or - maybe safer - just around the part where you launch the background process:

set -m
do launch-in-background stuff
set +m

Now disown and friends will work the same way they do in a terminal.

Thank you so much!
I, too, stumbled around that.
Next time I need it, I know what to do.


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#109 2020-03-16 05:11:14

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,951
Website

Re: Handy command-line stuff for terminals or scripts

Make that $i local

This bit me last week. There was a loop, like

for (( i=1; i<10; i++ ))
do
    something with $i
    function somefile
done

And somefile was being sent to a function like:

function(){
    for i in "$@"
    do some file thing with "$i"
done

So after the first call to function() the loop got all messed up, because $i had been changed from an integer to a filepath. neutral
Easy fix, at the top of function()

local i

so the i inside function is kept separate from the outside loop. If the loop was a function too, best to do the same there.

Variables inside functions are global by default in Bash, so any you don't want to share elsewhere should be declared local. Well known, but it's easy to forget those little i's inside loops.  yikes


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#110 2020-03-16 07:00:45

ohnonot
...again
Registered: 2015-09-29
Posts: 4,632
Website

Re: Handy command-line stuff for terminals or scripts

Wouldn't it be easier to (always try to) use unique variable names?
for file in "$@"
etc.


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#111 2020-03-16 08:04:34

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,951
Website

Re: Handy command-line stuff for terminals or scripts

^I think it's always advisable to make any variables local inside functions that won't be referred to anywhere else. "File" is a pretty generic name too, that might be used somewhere else. In a short script where you can easily remember all the variable names it's less important of course, but in anything big, where maybe other snippets are being sourced from files... it's good to keep the environment as clean as possible.


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#112 2020-04-03 14:02:53

misko_2083
Member
Registered: 2016-05-24
Posts: 346

Re: Handy command-line stuff for terminals or scripts

Thunar FM can run as a daemon, that way thunar starts up faster.
When  /usr/bin/thunar is executed while the daemon is running, it's process will send some data to the daemon and exit.

But the daemon doesn't know it got data from Thunar.
Any process can send a dbus message to the thunar daemon and open a new thunar window.

gdbus call --session  --dest org.xfce.Thunar --object-path /org/xfce/FileManager --method org.xfce.FileManager.LaunchFiles "/home/misko"  "['/home/misko']" ":0.0" ""

But that's not the only method.
If the daemon can open a dialog to create a new file at some path.

gdbus call --session  --dest org.xfce.Thunar --object-path /org/xfce/FileManager --method org.xfce.FileManager.CreateFile "/home/misko/Documents"  "empty" ":0.0" ""

It can copy, move  files, open a dialog to display file properties, rename file, open bulk rename,show and empty trash
All even without opening a thunar window.
To list all of the methods:

gdbus introspect --session --dest org.xfce.Thunar --object-path /org/xfce/FileManager

Gdbus arguments are:
   s - string "String1"
   as - array of strings in format "['String1', 'String2']"
   b - boolean "true"  "false"
   a{sv} - arguments  "{'String': <'variant_value'>, 'String2': <'variant_value'>}"


Што ни оштровиди ум сагледати не може - љубав превазилази.

Offline

#113 2020-04-04 04:01:26

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 6,951
Website

Re: Handy command-line stuff for terminals or scripts

^That's very interesting.
I'm curious, though, what would be the advantage of using a gdbus call to copy move or rename a file, rather than cp or mv?


...elevator in the Brain Hotel, broken down but just as well...
( a boring Japan blog (currently paused), idle Twitterings and GitStuff )

Introduction to the Bunsenlabs Lithium Desktop

Offline

#114 2020-05-05 18:49:26

ohnonot
...again
Registered: 2015-09-29
Posts: 4,632
Website

Re: Handy command-line stuff for terminals or scripts

ffmpeg

Transcoding videos to x265 a.k.a. HEVC. The time ratio for a very decent 2-pass transcode is ~12 to 1 on my 4-thread mediocre intel cpu, meaning 10min of video takes roughly 2h to transcode and comes out ~100MB at 1280x720. There's no discernible quality loss compared to an up to 5 times larger x264-encoded video.
So I'm putting a few in the queue before going to work...

After trawling the net for days & trial and erroring I came up with these commands:

ffmpeg_cmd="ffmpeg -hide_banner -analyzeduration 2147483647 -probesize 2147483647"
endoptions="-max_muxing_queue_size 9999"
# about -max_muxing_queue_size:
# stackoverflow.com/questions/49686244/ffmpeg-too-many-packets-buffered-for-output-stream-01

vcodec=x265

# all bitrates in kbps !!!
bitrate=1500
preset=medium # the default

file="$1"

outfile="${file%.*}.${bitrate}k.$vcodec.$preset.mkv"

params="ctu=32:max-tu-size=16:crf=20.0:tu-intra-depth=2:tu-inter-depth=2:rdpenalty=2:me=3:subme=5:\
merange=44:b-intra=1:amp=0:ref=5:weightb=1:keyint=360:min-keyint=1:bframes=8:aq-mode=1:aq-strength=1.0:\
rd=5:psy-rd=1.5:psy-rdoq=5.0:rdoq-level=1:sao=0:open-gop=0:rc-lookahead=80:scenecut=40:max-merge=4:\
qcomp=0.8:strong-intra-smoothing=0:deblock=-2:qg-size=16:pbratio=1.2:vbv-bufsize=1000:vbv-maxrate=$bitrate"
# 1st pass
$ffmpeg_cmd -y -i "$file" -an -c:v libx265 -preset "$preset" -x265-params "pass=1:$params" $endoptions -f matroska /dev/null
# 2nd pass
$ffmpeg_cmd -i "$file" -c:a copy -c:v libx265 -preset "$preset" -x265-params "pass=2:$params" $endoptions "$outfile"

PS: don't play supertuxkart at the same time! Crashed my machine twice.
Otherwise the system remains usable! Transcoding right now.


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#115 2020-05-05 20:20:13

Martin
Member
From: Stockholm, Sweden
Registered: 2015-10-01
Posts: 434
Website

Re: Handy command-line stuff for terminals or scripts

Thanks, I will try it.

/Martin


"Problems worthy of attack
prove their worth by hitting back."
Piet Hein

Offline

#116 2020-05-07 07:23:09

ohnonot
...again
Registered: 2015-09-29
Posts: 4,632
Website

Re: Handy command-line stuff for terminals or scripts

^^ addition to my previous post:
The bitrate is not an actual fixed bitrate when doing 2-pass transcoding; additionally x265 makes fixed bitrates obsolete, or so I half understand from reading stuff online; it's a maximum bitrate, the algorithm tries to stay below that at all times.

Even so, it makes sense to adjust the bitrate for lower resolution video. I believe 1000 would be the next step down, have had good results with that even at 1280x720.

Yesterday I also transcoded a 1920x1080 video - counting the pixels, it's almost exactly twice as much as 1280x720. It didn't take twice as long to transcode, but almost. Nevertheless, bitrate=1500 was sufficient.

Anyhow, I made it a script that can take several files, lets the CPU cool down between files, notifies you when finished:
https://notabug.org/ohnonot/stuff/src/master/transcode or https://framagit.org/ohnonot/stuff/-/bl … /transcode


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

#117 2020-06-14 20:58:04

Martin
Member
From: Stockholm, Sweden
Registered: 2015-10-01
Posts: 434
Website

Re: Handy command-line stuff for terminals or scripts

Today I finally tested your script and as far as I can see it works. But, should it take this long to transcode? Silly me started to test on "Dr Mabuse der Speiler" which is 4.5 hours long. The transcoding has gone on for ten hours now and has only covered ~11% of the distance. Four cores working full time. This means it would take between 3.5 and 4 days of full CPU load to transcode 4.5 hours of film. Am I doing something wrong?

/Martin


"Problems worthy of attack
prove their worth by hitting back."
Piet Hein

Offline

#118 2020-06-15 19:06:11

ohnonot
...again
Registered: 2015-09-29
Posts: 4,632
Website

Re: Handy command-line stuff for terminals or scripts

^ x265?
Yes. A 4.5h film in high resolution on a consumer grade intel - not good. Days ahead.

I didn't get back to this anymore because I realised that it's impossible to create a one-size-fits-all script.

I later continued transcoding my stuff to x264 instead, which gives extremely good results with carefully chosen parameters and 2-pass encoding, with maybe only slightly larger filesize, and much faster.
I think only professionals with dedicated machines can afford to fully reap the benefits of encoding x265.

Here's the x264 version of the script:

#!/bin/bash

# this was only the starting point:
# videoblerg.wordpress.com/2017/11/10/ffmpeg-and-how-to-use-it-wrong/comment-page-1/

# script dependencies
for cmd in ffmpeg ffprobe; do
    which "$cmd" >/dev/null || exit 1
done

file="$1"
vcodec=x264
pix_fmt=yuv420p

read -r -p "Enter video bitrate: " vbitrate
#~ vbitrate=1500
abitrate=128
preset=veryslow
# presets: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow
tune=film
# tunes: film,animation, grain, stillimage, psnr, ssim, fastdecode, zerolatency

bufsize=$((vbitrate+abitrate))
maxrate=$((vbitrate + vbitrate/3))

ffmpeg_cmd="ffmpeg -nostdin -hide_banner -analyzeduration 2147483647 -probesize 2147483647"
endopts="-max_muxing_queue_size 9999"

#~ -g 48 maximum keyframe interval? good for streaming, not needed in our use case.
#~ -x264opts no-scenecut see: video.stackexchange.com/a/24684 - despite additional (minimal) overhead, scenecut seems to be important

outfile="${file%.*}.$vcodec.$pix_fmt.v$vbitrate.a$abitrate.$preset.$tune.mp4"

echo ffmpeg -i "$file" -pix_fmt "$pix_fmt" -vsync 1 -vcodec lib$vcodec -b:v: ${vbitrate}k -bufsize ${bufsize}k -maxrate ${maxrate}k -preset $preset -profile:v high -tune $tune -pass 1 -acodec aac -b:a ${abitrate}k -ac 2 -ar 48000 -af "aresample=async=1:min_hard_comp=0.100000:first_pts=0" -f mp4 -y /dev/null
$ffmpeg_cmd -i "$file" -pix_fmt "$pix_fmt" -vsync 1 -vcodec lib$vcodec -b:v: ${vbitrate}k -bufsize ${bufsize}k -maxrate ${maxrate}k -preset $preset -profile:v high -tune $tune -pass 1 -acodec aac -b:a ${abitrate}k -ac 2 -ar 48000 -af "aresample=async=1:min_hard_comp=0.100000:first_pts=0" $endopts -f mp4 -y /dev/null

echo ffmpeg -i "$file" -pix_fmt "$pix_fmt" -vsync 1 -vcodec lib$vcodec -b:v: ${vbitrate}k -bufsize ${bufsize}k -maxrate ${maxrate}k -preset $preset -profile:v high -tune $tune -pass 2 -acodec aac -b:a ${abitrate}k -ac 2 -ar 48000 -af "aresample=async=1:min_hard_comp=0.100000:first_pts=0" -f mp4 "${file%.*}.testy.$vbitrate.$preset.mkv"
$ffmpeg_cmd -i "$file" -pix_fmt "$pix_fmt" -vsync 1 -vcodec lib$vcodec -b:v: ${vbitrate}k -bufsize ${bufsize}k -maxrate ${maxrate}k -preset $preset -profile:v high -tune $tune -pass 2 -acodec aac -b:a ${abitrate}k -ac 2 -ar 48000 -af "aresample=async=1:min_hard_comp=0.100000:first_pts=0" $endopts -f mp4 "$outfile"

I cut it down, removing some pointless stuff. It should work, but please tell me if it doesn't.

The biggest question is the target bitrate. It will take some fiddling to find out what preserves quality while still reducing filesize (if that is your goal).

You might save some time by leaving the audio as it is (-b:a copy), if it's in a lossy format already.

Last edited by ohnonot (2020-06-15 19:09:14)


BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
---
Thank you for posting direct image links!

Offline

Board footer

Powered by FluxBB