[Solved] advice on string parsing for a pipemenu writen in bash

ansem_of_nexus · 2018-05-26 23:10:10

Let me start with the guidelines that I set for this project

has to be usable from an average minimal install
compact as much as possible both in code and controls
reduce server calls as much as possible

I have completed the pipemenu minus one thing, and that means that I am approaching the end of my abilities, so I am turning to the forums. I don't know what things in a script will make it faster or slower, but I am assuming that the parsing method will have a real affect on it, so here is the current method:

t_perc=$(echo $t | awk -F'Percent Done:' '{print $2}' | awk -F'ETA:' '{print $1}' | xargs)
s_gulimit_current=$(echo $s | awk -F'Upload speed limit:' '{print $2}' | cut -d'(' -f -1 | xargs)

The first one is the more often style all awks, but I do have a handful of the ones with cut in them. Again an assumption but I would think that if bash has a parsing method I can use the script with run faster and use less resources. The problems are that I am parsing of a variable that has the whole print out of "transmission-remote -l" in it, and there is no newline characters to make it super easy, on top of that the information is in a table format which has tons of white spaces to trim off, both proceeding and trailing.

Now I might just maybe being a perfectionist, or wrong about their being a faster method. If there is no better way to do this then I will leave it at that. Also if you want to see the parsing and how it's used in the pipemenu then go here (github link)

Last edited by ansem_of_nexus (2018-06-10 20:16:07)

johnraff · 2018-05-27 01:56:16

Can you post a snippet of the actual data you want to parse, and what you want to retrieve from it?

BTW the problem with '&' mentioned at the top of the script is very likely because it must be escaped in xml as &

There are some openbox pipemeunu functions in /usr/lib/bunsen/common/bl-include.cfg which you might refer to for escaping xml special characters. (For pipemenus running on BunsenLabs systems you could just source that file and use them directly.)

Last edited by johnraff (2018-05-27 02:01:44)

ansem_of_nexus · 2018-05-27 03:27:02

you were correct about the escaping the '&', I had attempted to do that before, but I didn't put the ';' at the end so it didn't work. As for the data I am parsing from this is some of it.

  State: Idle
  Location: /share/OS-images/
  Percent Done: 100%
  ETA: 0 seconds (0 seconds)
  Download Speed: 0 kB/s
  Upload Speed: 0 kB/s
  Have: 1.95 GB (1.95 GB verified)
  Availability: 100%
  Total size: 1.95 GB (1.95 GB wanted)
  Downloaded: 1.96 GB
  Uploaded: 29.46 GB
  Ratio: 15.0
  Corrupt DL: None
  Peers: connected to 0, uploading to 0, downloading from 0

In this section I need most of the values from the fields. state, location, percent, eta, download, upload, availability, total size, and ratio.

johnraff · 2018-05-27 03:45:58

ansem_of_nexus wrote:

The problems are that I am parsing of a variable that has the whole print out of "transmission-remote -l" in it, and there is no newline characters

But the data you posted contains newlines.

ansem_of_nexus · 2018-05-27 04:41:47

johnraff wrote:

ansem_of_nexus wrote:
The problems are that I am parsing of a variable that has the whole print out of "transmission-remote -l" in it, and there is no newline characters
But the data you posted contains newlines.

yes it does right now, but when it goes in the bash variable it loses all of the newlines, like this

  State: Idle  Location: /share/OS-images/  Percent Done: 100%  ETA: 0 seconds (0 seconds)  Download Speed: 0 kB/s  Upload Speed: 0 kB/s  Have: 1.95 GB (1.95 GB verified)  Availability: 100%  Total size: 1.95 GB (1.95 GB wanted)  Downloaded: 1.96 GB  Uploaded: 29.46 GB  Ratio: 15.0  Corrupt DL: None  Peers: connected to 0, uploading to 0, downloading from 0

johnraff · 2018-05-27 04:59:34

Is there any marker between the Value and the following Name?
Do you want to extract the total content of whatever follows the colon?

Have you lost the newlines in your processing, or were they absent from the beginning?
Did you put them in manually in the above forum post?

ansem_of_nexus · 2018-05-27 05:22:08

johnraff wrote:

Is there any marker between the Value and the following Name?
Do you want to extract the total content of whatever follows the colon?
Have you lost the newlines in your processing, or were they absent from the beginning?
Did you put them in manually in the above forum post?

so the process is that I set a variable with the output of a command that normally puts out the information with newlines, but when I echo out the variable and try to parse it, all the newlines have disappeared. As for the content extraction, yes everything after the colon, minus any proceeding or trailing spaces. I don't think that the newlines disappeared by my actions, I am using echo to call the variable for parsing, maybe I should be using printf or something. I don't know anything about printf though.

ohnonot · 2018-05-27 06:04:27

So if the question is

Shell Scripting For Speed

here's my general advice:

Make a choice of which shell you use, that's the "#/bin/..." at the top:

dash: very small & fast but limited.
you will have to use more external commands to accomplish things.
bash: bulkier, takes longer to load, maybe slightly slower, but: you can accomplish much more with shell builtins. arrays, string manipulation... often the call to awk or sed is not required anymore
sh: the default shell, depends on your setup. best code for portability (POSIX) when you sue this one. which is practically the same as coding for dash.

Be aware of what external commands are - each call to an external command slows your script down:

the command needs to be read from hard drive
if it's inside a command substitution or at either end of a pipe, it starts a new subshell. see here and here etc.
it is often overkill for the task required

Be equally aware of what the internal commands (or builtins) of the shell in question are.
Quite many commands (e.g. echo) are both internal and external. A hint: if you have a terminal open that runs the shell in question, and you can use

help somecommand

, then it's a builtin, and that's what the shell will default to.

So, reading the shell's man page might help.
A realistic task with e.g. dash, but not with bash.

Disclaimer: I like to use bash. Knowing of its capabilities, it allows me to make do with almost no piping to sed, awk etc. I reckon that in the end this is much faster.
Just imagine you have one of those hideous multi-pipe oneliners inside a loop that parses through a long file, or needs to be executed in short intervals - that's many, many subshells opened, reading external commands, and closed in rapid succession. If i can replacer all of that with bash variables and string manipulation, it will save a lot of resources.

I recommend everyone who chooses bash to read up on string manipulation. There's quite a few resources out there, and evtl. you will end up on this page with the obligatory disclaimer that it's outdated. Maybe this one for a quick reference, or this one.

And you definitely need to check out Greg's Wiki.

ohnonot · 2018-05-27 06:10:04

...and about your script in particular, in no particular order:

why use a function to output some string? assign the string to a variable instead
echo can use multiline strings; even better for many use cases: cat <<EOF etc.
according to my previous post, the tpull and spull functions will definitely slow your script down a LOT - and why xargs? i don't get it.

Last edited by ohnonot (2018-05-27 06:12:53)

johnraff · 2018-05-27 08:54:12

echo $t

That's what ate your newlines anyway. An unquoted variable.

echo "$t"

would have kept them.

But as ohnonot says, you seem to be making the task more complicated than it need be.
For output with a collection of data on individual lines, while awk is suited for this, it's not necessary. I would use read.

For example (assuming there are no colons anywhere except after the Name, data is in variable $input):

while read -r line
do
    Name="${line%:*}"
    Value="${line#*:}"
    do something with "$Name" and "$Value"
done <<<"$input"

Although you might be able to feed the data in directly on stdin (last line):

done < <( process )

johnraff · 2018-06-07 05:42:50

Moved to "Basic Help & Support".

brontosaurusrex · 2018-06-07 07:20:58

Quick escaper (https://stackoverflow.com/questions/128 … ml-in-bash)

echo "what/is/this&" | sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g; s/"/\&quot;/g; s/'"'"'/\&#39;/g'

should return

what/is/this&amp;

Last edited by brontosaurusrex (2018-06-07 07:21:20)

ansem_of_nexus · 2018-06-10 21:40:01

Thanks johnraff for moving this thread, sorry about getting it in the wrong category.

Took a while to get this finished, IRL has been taking up more of my time then I would like. But I was able to get it rewritten and used the bash parsing, as well as the 'while read', and am much happier about the speed at which the pipemenu runs. While I was working on it I did see a few more areas where I think I can streamline the code (change the 'spull' into a 'while read'), and there is another feature that I want to add to the pipemenu. So because I was able to get the pipemenu into a working/usable state, I wanted to set this thread to [Solved] because the topic is handled.
The new version

I also wanted to touch on what brontosaurusrex said:

sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g; s/"/\&quot;/g; s/'"'"'/\&#39;/g'

Thanks for letting me know that there are more characters that need to be escape (I probably should have looked this up already), I haven't taken care of them let, but I will.

#1 2018-05-26 23:10:10

[Solved] advice on string parsing for a pipemenu writen in bash

#2 2018-05-27 01:56:16

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#3 2018-05-27 03:27:02

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#4 2018-05-27 03:45:58

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#5 2018-05-27 04:41:47

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#6 2018-05-27 04:59:34

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#7 2018-05-27 05:22:08

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#8 2018-05-27 06:04:27

Re: [Solved] advice on string parsing for a pipemenu writen in bash

Shell Scripting For Speed

#9 2018-05-27 06:10:04

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#10 2018-05-27 08:54:12

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#11 2018-06-07 05:42:50

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#12 2018-06-07 07:20:58

Re: [Solved] advice on string parsing for a pipemenu writen in bash

#13 2018-06-10 21:40:01

Re: [Solved] advice on string parsing for a pipemenu writen in bash

Board footer