You are not logged in.

#1 2019-02-06 08:49:13

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

bash script function to parse config files

The easiest way to import some config variables from a file into a shell script is just to source it

cat file.cfg
var1=string
var2='other string'

. file.cfg

Now those variables will be set in the script. It's not very safe though - especially if the script is being run by root - because any code in file.cfg will just be executed in the script. Such code might just arrive by a mistake as well as from an attacker.

Web searching indicates bash doesn't have built-in config parsing like python or other more sophisticated languages, but I finally (took longer than expected) made a function you can put in scripts. It will parse a file with entries like this test file:

# comment
top_level = yes

[first]
var1=string
var2 = second
var3='string # with hash'

[second]
var1 = un # french
var2 = deux
# var3 = troix
var4 = fourth # var5 = fifth
var5 = two words #
var#6=sixth
var7 = # empty variable

Empty lines and comments ignored, spaces around '=' optional, hashmarks inside quotes (' or ") OK.
By default, all entries go in an associative array 'config', but you can pass another one as the second argument, and if you add a [section] header that defines the array name for entries that follow. If any array referred to already exists it will be added to, or existing entries overwritten if the file changes them. New arrays will be created as needed and added to the script environment.

Here's what it created with the above file, and no pre-existing arrays:

john@helium:/data/john/projects/yad-exit$ unset config first second
john@helium:/data/john/projects/yad-exit$ parse_config test.cfg
john@helium:/data/john/projects/yad-exit$ declare -p config first second
declare -A config=([top_level]="yes" )
declare -A first=([var1]="string" [var3]="string # with hash" [var2]="second" )
declare -A second=([var1]="un" [var2]="deux" [var5]="two words" [var4]="fourth" [var7]="" )

And if config already exists:

john@helium:/data/john/projects/yad-exit$ unset config first second
john@helium:/data/john/projects/yad-exit$ declare -A config
john@helium:/data/john/projects/yad-exit$ config[Initial Key]='some setting'
john@helium:/data/john/projects/yad-exit$ parse_config test.cfg
john@helium:/data/john/projects/yad-exit$ declare -p config first second
declare -A config=(["Initial Key"]="some setting" [top_level]="yes" )
declare -A first=([var1]="string" [var3]="string # with hash" [var2]="second" )
declare -A second=([var1]="un" [var2]="deux" [var5]="two words" [var4]="fourth" [var7]="" )

And using 'settings' instead of 'config' as default array name:

john@helium:/data/john/projects/yad-exit$ unset config first second
john@helium:/data/john/projects/yad-exit$ parse_config test.cfg settings
john@helium:/data/john/projects/yad-exit$ declare -p settings first second
declare -A settings=([top_level]="yes" )
declare -A first=([var1]="string" [var3]="string # with hash" [var2]="second" )
declare -A second=([var1]="un" [var2]="deux" [var5]="two words" [var4]="fourth" [var7]="" )


Limitations:

*) File is parsed line by line, so no multi-line strings are supported, unless you encode them as \n or something first.
*) No backslash escaping. If you want a literal ' or " then enclose it in the opposite type:
var = "string's got an apostrophe"

If anyone spots a snag or has a suggestion for improvement, welcome!
I was thinking this might be worth dropping in /usr/lib/bunsen/common/bl-includes
(Of course you would then have to source that file, but that's safer because it's a locked-down system file.)

Anyway, here (now updated to fix glitches that came up below):

# These functions need bash.

# Usage: parse_config <file> [<default array name>]

# If no default array name is given, it defaults to 'config'.
# If there are [section] headers in file, following entries will be
#  put in array of that name.

# Config arrays may exist already and will appended to or overwritten.
# If preexisting array is not associative, function exits with error.
# New arrays will be created as needed, and remain in the environment.
parse_config(){
    [[ -f $1 ]] || { echo "$1 is not a file." >&2;return 1;}
    if [[ -n $2 ]]
    then
        local -n config_array=$2
    else
        local -n config_array=config
    fi
    declare -Ag ${!config_array} || return 1
    local line key value section_regex entry_regex
    section_regex="^[[:blank:]]*\[([[:alpha:]_][[:alnum:]_]*)\][[:blank:]]*(#.*)?$"
    entry_regex="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\"|[^#[:blank:]]+)[[:blank:]]*(#.*)*$"
    while read -r line
    do
        [[ -n $line ]] || continue
        [[ $line =~ $section_regex ]] && {
            local -n config_array=${BASH_REMATCH[1]}
            declare -Ag ${!config_array} || return 1
            continue
        }
        [[ $line =~ $entry_regex ]] || continue
        key=${BASH_REMATCH[1]}
        value=${BASH_REMATCH[2]#[\'\"]} # strip quotes
        value=${value%[\'\"]}
        config_array["${key}"]="${value}"
    done < "$1"
}

# Usage: parse_config_vars <file>
# No arrays, just read variables individually.
# Preexisting variables will be overwritten.

parse_config_vars(){
    [[ -f $1 ]] || { echo "$1 is not a file." >&2;return 1;}
    local line key value entry_regex
    entry_regex="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\"|[^#[:blank:]]+)[[:blank:]]*(#.*)*$"
    while read -r line
    do
        [[ -n $line ]] || continue
        [[ $line =~ $entry_regex ]] || continue
        key=${BASH_REMATCH[1]}
        value=${BASH_REMATCH[2]#[\'\"]} # strip quotes
        value=${value%[\'\"]}
        declare -g "${key}"="${value}"
    done < "$1"
}

Last edited by johnraff (2019-03-08 07:26:17)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#2 2019-02-06 20:48:10

brontosaurusrex
Middle Office
Registered: 2015-09-29
Posts: 1,799
Website

Re: bash script function to parse config files

Not exactly sure what are you planning, but I'd probably just use a higher-level language (python) parser for ini files, last example on this page https://serverfault.com/questions/34566 … -variables looks promising (assuming the said parser is actually smart enough to provide a sequrita pillow for my bash stuff ..., should be tested).

Offline

#3 2019-02-07 06:30:35

ohnonot
...again
Registered: 2015-09-29
Posts: 3,842
Website

Re: bash script function to parse config files

^ i think the function is good enough if it safely avoids command execution (which i believe was the point). if it is safe in general (big if).
it's beautiful to have a pure solution. bash is the most bloated of them shells, so at least one should make full use of it.

Online

#4 2019-02-07 08:56:51

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

^ ^^Right, I was trying to stick with bash builtins only, just for the feelgood factor I suppose. Sed, awk and grep are pretty fast, though I don't think I was up for invoking python just to parse an ini file. Scouring the interwebs the same Stackoverflow pages come up, and I did run into that huge bash ini parser on github, but I thought it was a bit over the top. With a couple of smallish limitations I think my function works alright, and ought to be fairly safe, though it would be nice to have a second opinion on that.

If you don't need to put the configs into arrays but just set individual variables then this smaller function seems to do the job:

# Usage: parse_config_vars <file>
# No arrays, just read variables individually.
# Preexisting variables will be overwritten.
parse_config_vars(){
    [[ -f $1 ]] || { echo "$1 is not a file." >&2;return 1;}
    local line key value entry_regex
    entry_regex="^[[:blank:]]*([[:alnum:]_]+)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\"|[^#]+)"
    while read -r line
    do
        [[ -n $line ]] || continue
        [[ $line =~ $entry_regex ]] || continue
        key=${BASH_REMATCH[1]}
        value=${BASH_REMATCH[2]#[\'\"]} # strip quotes
        value=${value%[\'\"]}
        value=${value%${value##*[![:blank:]]}} # strip trailing spaces
        declare -g "${key}"="${value}"
    done < "$1"
}

If you can describe the kind of line you're looking for as a regular expression, and enclose the bits you want to extract in ( ) , then bash has this nice array ${BASH_REMATCH} which holds all the matches and makes the rest of the code quite easy.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#5 2019-02-07 11:29:35

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 643

Re: bash script function to parse config files

Thanks johnraff for the script. I learned a lot form it.

And this can certainly be useful to setups with limited memory where for example python is not installed.


Postpone all your duties; if you die, you won't have to do them ..

Offline

#6 2019-02-07 20:13:51

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,467

Re: bash script function to parse config files

Funny you should bring this problem up, John.

4 years ago I wrote a command line tool for examining INI files from scripts, using a proper parser: https://github.com/2ion/ini. No idea if it still works today, but the gist is that you would 'query' the contents of INI files:

ini --print section:key file.ini

You would dump the keys present in a file with

ini --list-keys file.ini

or test for  the presence of one specific key using

ini --exists section:key file.ini

you could also list all keys or values matching a specific regex.

I just tried and it works with your test file just fine. Though I think var6 behaves differently (it's libiniparser, so everything what that parser thinks is right applies). Personally, I'm more a fan of TOML and YAML nowadays as far as config files are concerned (https://github.com/toml-lang/toml for the one, the other is self-explaining) or libconfig.

I won't sign a statement that says it doesn't have bugs smile Haven't looked at the code since, well, 4 years.

It's written in C. You can build it on current Debian as follows (I think) and try for yourself:

git clone https://github.com/2ion/ini.git
cd ini
sudo apt-get install build-essential autoconf automake libiniparser-dev
autoreconf -si

Now you need to edit src/ini.c and replace

#include <iniparser.h>

with

#include <iniparser/iniparser.h>

(the thing was written and compiled on Arch).

Now you can finally

./configure && make

You'll find the binary in src/ini. Show the help:

./src/ini -h

No idea if this is useful or not!


A silent kite against the blue, blue sky

Offline

#7 2019-02-07 20:19:04

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,467

Re: bash script function to parse config files

twoion wrote:

Funny you should bring this problem up, John.

4 years ago I wrote a command line tool for examining INI files from scripts, using a proper parser: https://github.com/2ion/ini. No idea if it still works today, but the gist is that you would 'query' the contents of INI files:

ini --print section:key file.ini

You would dump the keys present in a file with

ini --list-keys file.ini

or test for  the presence of one specific key using

ini --exists section:key file.ini

you could also list all keys or values matching a specific regex.

I just tried and it works with your test file just fine. Though I think var6 behaves differently (it's libiniparser, so everything what that parser thinks is right applies). Personally, I'm more a fan of TOML and YAML nowadays as far as config files are concerned (https://github.com/toml-lang/toml for the one, the other is self-explaining) or libconfig.

I won't sign a statement that says it doesn't have bugs smile Haven't looked at the code since, well, 4 years.

It's written in C. You can build it on current Debian as follows (I think) and try for yourself:

git clone https://github.com/2ion/ini.git
cd ini
sudo apt-get install build-essential autoconf automake libiniparser-dev
autoreconf -si

Now you need to edit src/ini.c and replace

#include <iniparser.h>

with

#include <iniparser/iniparser.h>

(the thing was written and compiled on Arch).

Now you can finally

./configure && make

You'll find the binary in src/ini. Show the help:

./src/ini -h

No idea if this is useful or not!

Edit: Just noticed the code uses variable-length C arrays which is very bad big_smile I used to like them a lot. So the code would need code review before being put into production in any case smile Though I learned a lot over the past 4 years and think I might be able to write better code smile


A silent kite against the blue, blue sky

Offline

#8 2019-02-07 20:42:39

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 643

Re: bash script function to parse config files

^Certainly, 'goto end;' in 'switch' should be replaced with 'break;' As it is, it looks kind of ugly devil

Sorry, I couldn't resist smile


Postpone all your duties; if you die, you won't have to do them ..

Offline

#9 2019-02-07 22:22:33

malm
jgmenu developer
Registered: 2016-10-13
Posts: 451
Website

Re: bash script function to parse config files

^^ Nice one 2ion. Cool project.

But, yes... those dreaded C99 features.

Offline

#10 2019-02-08 04:32:43

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

There is a Debian package cfget but it's almost abandoned, and didn't do exactly what I wanted, eg doesn't ignore comments, no arrays.

@twoion if you ever feel like refreshing iniparser maybe we could think about packaging it? It offers a lot more functionality than my bash function, and probably would run just as fast.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#11 2019-02-08 08:20:25

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,467

Re: bash script function to parse config files

iMBeCil wrote:

^Certainly, 'goto end;' in 'switch' should be replaced with 'break;' As it is, it looks kind of ugly devil

Sorry, I couldn't resist smile

I think it's goto end to break both the switch statement and the while loop.


A silent kite against the blue, blue sky

Offline

#12 2019-02-08 10:40:24

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 643

Re: bash script function to parse config files

^Ooops, my bad ops you are right. Sorry.


Postpone all your duties; if you die, you won't have to do them ..

Offline

#13 2019-02-08 10:46:52

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,467

Re: bash script function to parse config files

johnraff wrote:

There is a Debian package cfget but it's almost abandoned, and didn't do exactly what I wanted, eg doesn't ignore comments, no arrays.

@twoion if you ever feel like refreshing iniparser maybe we could think about packaging it? It offers a lot more functionality than my bash function, and probably would run just as fast.

Sure, just need to replace variable length arrays with some calloc/malloc and add some return value checks (always check your return values in C big_smile) and then we're good to go.


A silent kite against the blue, blue sky

Offline

#14 2019-02-10 22:44:20

ohnonot
...again
Registered: 2015-09-29
Posts: 3,842
Website

Re: bash script function to parse config files

johnraff wrote:

If you don't need to put the configs into arrays but just set individual variables then this smaller function seems to do the job:

# Usage: parse_config_vars <file>
# No arrays, just read variables individually.
# Preexisting variables will be overwritten.
parse_config_vars(){
    [[ -f $1 ]] || { echo "$1 is not a file." >&2;return 1;}
    local line key value entry_regex
    entry_regex="^[[:blank:]]*([[:alnum:]_]+)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\"|[^#]+)"
    while read -r line
    do
        [[ -n $line ]] || continue
        [[ $line =~ $entry_regex ]] || continue
        key=${BASH_REMATCH[1]}
        value=${BASH_REMATCH[2]#[\'\"]} # strip quotes
        value=${value%[\'\"]}
        value=${value%${value##*[![:blank:]]}} # strip trailing spaces
        declare -g "${key}"="${value}"
    done < "$1"
}

I just took this for my weather conky, thanks!
it seems there's a problem when there's a comment on the same line, after the key = value pair.
the comment gets removed, but usually leaves a space behind, and the line that removes trailing quotes won't work anymore.
I think this is better (i also used more quoting than you!) - at least it works for me so far (famous last words):

parse_config_vars(){
    local line key value entry_regex
    entry_regex="^[[:blank:]]*([[:alnum:]_]+)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\"|[^#]+)"
    while read -r line
    do
        [[ -n "$line" ]] || continue
        [[ "$line" =~ $entry_regex ]] || continue
        key="${BASH_REMATCH[1]}"
        value="${BASH_REMATCH[2]#[\'\"]}" # strip quotes front
        value="${value%${value##*[![:blank:]]}}" # strip trailing spaces
        value="${value%[\'\"]}" # strip trailing quotes
        echo "Parsing config: ${key} = ${value}"
        declare -g "${key}"="${value}"
    done < "$1"
}

(i stripped out the file-existing-check because i already had that outside the function)

Last edited by ohnonot (2019-02-10 22:45:31)

Online

#15 2019-02-12 10:00:50

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

@ohnonot you're correct.

I was fooled by https://regex101.com/ into thinking that the regex would eagerly match the first alternative in the group ('[^']+'|\"[^\"]+\"|[^#]+) so pick out exactly what was between quotes, only falling back to the "not comment" alternative if there was no quoted string. This is what regex101 showed, but unfortunately this PCRE behaviour is not what bash uses, but the POSIX standard, which is to go through all the alternatives and come back with the longest match.
Described: https://www.regular-expressions.info/alternation.html and https://www.regular-expressions.info/posix.html

Your version looks as if it should work, but I'm keen to precisely pick out what's in the quotes rather than tidy up afterwards. Work in progress...

(btw The quotes you added around variables are all optional, I think.)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#16 2019-02-13 05:02:38

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

off-topic

Interesting that variables can be defined from each other without quoting, and spaces, line breaks etc are preserved:

john@helium:~$ var='first line
> second line'
john@helium:~$ echo "$var"
first line
second line
john@helium:~$ foo=$var
john@helium:~$ echo "$foo"
first line
second line

John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#17 2019-02-13 06:27:19

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

I've found one workaround.

Looking for the "value" part:
Check quite strictly for the quoted version, strip the quotes, and no need to strip trailing spaces.
Then if that fails, check for a loose match of anything up to a # but not with a trailing space.
So it needs two regex tests, and adds a bit more logic, but might be more robust??
Quotes regex:

entry_regex_quotes="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\")[[:blank:]]*(#.*)*$"

and loose version:

entry_regex_loose="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*([^#]+[^#[:blank:]])*"

I also enforced the rule that variable names can't start with a digit, ie ([[:alpha:]_][[:alnum:]_]*)

So anything inside quotes - single or double, but matching - is used, but if no such pattern is found, then anything after the = and before a possible # is used as "value".

If a line has:

var9 = 'single quotes' "double" # comment

then var9's value will be set to

'single quotes' "double"

ie including the quotes, because this is loose matching, not quote matching, but with leading and trailing spaces not included.

Does this seem like the most generally useful behaviour?
If config file writers were forced to use quotes then only the strict matching could be used, and a line like

var8="quoted string" X

would be rejected. Otherwise with loose matching allowed the whole RHS would be taken.

Anyway, the current function, doing strict, then loose, matching, looks like this:

parse_config_vars(){
    [[ -f $1 ]] || { echo "$1 is not a file." >&2;return 1;}
    local line key value entry_regex
    entry_regex_quotes="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\")[[:blank:]]*(#.*)*$"
    entry_regex_loose="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*([^#]+[^#[:blank:]])*"
    while read -r line
    do
        [[ -n $line ]] || continue
        if [[ $line =~ $entry_regex_quotes ]]
        then
            value=${BASH_REMATCH[2]#[\'\"]} # strip quotes
            value=${value%[\'\"]}
        elif [[ $line =~ $entry_regex_loose ]]
        then
            value=${BASH_REMATCH[2]}
        else
            continue
        fi
        key=${BASH_REMATCH[1]}
        declare -g "${key}"="${value}"
    done < "$1"
}

The idea is to have something that scripts could source from a library file, so should be usable just by calling the function, with no need to edit the code to suit some particular script.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#18 2019-02-13 09:40:49

ohnonot
...again
Registered: 2015-09-29
Posts: 3,842
Website

Re: bash script function to parse config files

about the quoting: i guess you're right.
i'll do some more shellchecking evtl.

johnraff wrote:

The idea is to have something that scripts could source from a library file, so should be usable just by calling the function, with no need to edit the code to suit some particular script.

that would be great!
I dropped in the new version, but now variables that consist only of one character:

this=0
that=1

are parsed empty:

this=
that=

sorry, i really suck at regex, I have no clue what's going on...

that said, my previous version is still working.
i can format the comments that are on the same line weirdly, and it still strips them out properly:

this="bla"    # (that was 3 spaces and 1 tab) #comment

Online

#19 2019-02-13 19:59:45

THX1138
Member
Registered: 2019-01-14
Posts: 191

Re: bash script function to parse config files

ohnonot wrote:

sorry, i really suck at regex, I have no clue what's going on...

I really love regex, I cant claim to be any kind of expert at it but I know a good way for you to become a lot more comfortable with it.
This website, each page explains a feature of regex and then asks you to solve a regex puzzle. After solving a few of the puzzles it makes you realise that it's ok to suck at regex, because with regex theres a thousand ways of achieving anything you want to do. I write a lot of my spam filters for my email smart host in regex but I still test them first on an online regex tester.
learn regex
test your regex


The telephone is an antiquity - you never know who is calling, there is no image, it is an outmoded product which constantly disrupts work (Ralf Hutter (Kraftwerk)) ps: my wife knows how much I dislike being disrupted at Work - Ralf Hutter hit the nail on the head there

Offline

#20 2019-02-14 06:40:31

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

ohnonot wrote:

I dropped in the new version, but now variables that consist only of one character:

this=0
that=1

are parsed empty:

this=
that=

Good catch! (It really helps no end having more eyes on this stuff as it's being de-bugged.)
Luckily the fix was easy. Here's the new version of the second regex:

entry_regex_loose="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*([^#]*[^#[:blank:]])*"

The difference is that the expression for what was picked out for "value" was insisting on at least two characters - now the first one is optional. (The last one cannot be a space - this gets rid of trailing spaces.)
This bit:
([^#]+[^#[:blank:]])*
changed to:
([^#]*[^#[:blank:]])*
The final asterisk makes the whole thing optional, allowing empty variables to be set, still ignoring spaces or comments.

---
This all begs the question, though, of whether such a "loose" parsing is in fact desirable. For example, it also means that either of these:

"in quotes" X 
"double" 'single'

would go as-is into the variable's value. They'd fail the first entry_regex_quotes test, but get slurped up by entry_regex_loose.

An alternative would be to insist that if the RHS of an assignment is unquoted then it has to be a single word only, ie no spaces or hashmarks. Might that be more congenial?
It would mean this:

entry_regex_loose="^[[:blank:]]*([[:alpha:]_][[:alnum:]_]*)[[:blank:]]*=[[:blank:]]*([^#[:blank:]]+)*[[:blank:]]*(#.*)*$"

Any strange lines would just be ignored.

A library function, as well as being useful, should also behave the way most users would expect, right? Not quite sure yet exactly what that would be.

---
BTW +1 to THX1138's recommended regex tester. I find it extremely useful, even though having to remember that bash, sed and grep use POSIX regex which is a bit simpler than the PCRE they offer by default. (Grep does have a -P option for PCRE though.)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#21 2019-02-15 21:06:48

ohnonot
...again
Registered: 2015-09-29
Posts: 3,842
Website

Re: bash script function to parse config files

i tested the new one once (!) and it seems to work.

for me the good thing about this function is that i could use it as a drop-in replacement for something that previously just sourced config files, so i didn't even have to change the config files.
But maybe it would be better to simply make a decision about the syntax required; maybe:
- do not quote variables unless you want the quotes to be part of the value
- no trailing comments
???

preferably in accordance with some existing standard, although there are too many to choose from...

Online

#22 2019-02-16 07:28:52

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

ohnonot wrote:

i tested the new one once (!) and it seems to work.

Thanks!

for me the good thing about this function is that i could use it as a drop-in replacement for something that previously just sourced config files, so i didn't even have to change the config files.

This is what I was hoping.

Not yet quite sure about these though:

maybe:
- do not quote variables unless you want the quotes to be part of the value

I think often people expect to put quotes around things when they contain spaces or unusual characters, like '#AA3355' or 'string with spaces'. It's possible to get surrounding quotes as part of the string if you surround them with the other kind of quotes '"quoted string"'.

- no trailing comments

They are not supported in all ini files, but support here has been achieved, and I guess most people would expect trailing comments to be ignored, even if they didn't use them. But banning trailing comments would be required if it's going to be possible to have hashmarks inside strings without having to quote them.

My biggest question right now is about unquoted strings. I'm leaning now to allowing them only for single words (no spaces) but not sure.

preferably in accordance with some existing standard, although there are too many to choose from...

This...

I'll have a look round some config files of apps we use like network-manager, lightdm etc and see if there's any kind of consensus.

Any input appreciated!


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#23 2019-03-08 03:14:14

Bearded_Blunder
Dodging A Bullet
From: Seat: seat0; vc7
Registered: 2015-09-29
Posts: 730

Re: bash script function to parse config files

johnraff wrote:

If you don't need to put the configs into arrays but just set individual variables then this smaller function seems to do the job:

# Usage: parse_config_vars <file>
# No arrays, just read variables individually.
# Preexisting variables will be overwritten.
parse_config_vars(){
    [[ -f $1 ]] || { echo "$1 is not a file." >&2;return 1;}
    local line key value entry_regex
    entry_regex="^[[:blank:]]*([[:alnum:]_]+)[[:blank:]]*=[[:blank:]]*('[^']+'|\"[^\"]+\"|[^#]+)"
    while read -r line
    do
        [[ -n $line ]] || continue
        [[ $line =~ $entry_regex ]] || continue
        key=${BASH_REMATCH[1]}
        value=${BASH_REMATCH[2]#[\'\"]} # strip quotes
        value=${value%[\'\"]}
        value=${value%${value##*[![:blank:]]}} # strip trailing spaces
        declare -g "${key}"="${value}"
    done < "$1"
}

If you can describe the kind of line you're looking for as a regular expression, and enclose the bits you want to extract in ( ) , then bash has this nice array ${BASH_REMATCH} which holds all the matches and makes the rest of the code quite easy.

So anyway, I said I had a glitch, and I do using the above, my hard coded in script config has:

IMG_PATH="/usr/share/images/bunsen/exit/light"
# Then entries like
BTN_HIBERNATE="$IMG_PATH/hibernate.png"

Which works fine till I parse a config file, afterwards $IMG_PATH no longer gets expanded running the script.
Clues to fix that? I mean other than the obvious "Put the full path in every time" which would work, but would make user edits somewhat more tedious.


Blessed is he who expecteth nothing, for he shall not be disappointed...
If there's an obscure or silly way to break it, but you don't know what.. Just ask me

Offline

#24 2019-03-08 06:16:40

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 5,616
Website

Re: bash script function to parse config files

^Good point, but the reason to use a parsing function instead of just sourcing the file as if it was a shell script fragment, is precisely because no shell interpretation will be done.

The above case is fixable because if IMG_PATH is defined already then all that remains is to define the filename, not the full path.
eg in config file(s):

IMG_PATH="/usr/share/images/bunsen/exit/light"
HIBERNATE_IMG=hibernate.png

And in script itself, after parsing config:

BTN_HIBERNATE="$IMG_PATH/$HIBERNATE_IMG"

John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#25 2019-03-08 07:09:06

Bearded_Blunder
Dodging A Bullet
From: Seat: seat0; vc7
Registered: 2015-09-29
Posts: 730

Re: bash script function to parse config files

Much safer I agree smile  I'm pretty tired, ought to have figured that out for myself.
Next stop bed.


Blessed is he who expecteth nothing, for he shall not be disappointed...
If there's an obscure or silly way to break it, but you don't know what.. Just ask me

Offline

Board footer

Powered by FluxBB