You are not logged in.

#1 2018-01-27 20:47:47

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

apt-sourcemgr: simple APT sources.list entry management

John, I think I'll take on

bl-welcome:

Deal with cases where sources.list is completely messed.
https://github.com/BunsenLabs/bunsen-ne … /issues/12

I propose to implement a simple script called bl-sourcemgr, to become part of bunsen-utilities, with the following properties:

1. Purpose is to manage sources.list entries cleanly and in a stable way. There should be actions like add, remove, enable, disable and also a show/check function for determining whether a given sources.list entry is present on the system. For example, from bl-welcome we should be able to check with e.g.

bl-sourcemgr check --uri https://pkg.bunsenlabs.org --distro helium --component main --component contrib --enabled --type deb

if our repository is live on the system. Same,

bl-sourcemgr add --uri https://deb.debian.org --distro stretch -C main -C contrib -C non-free --enabled --type deb
bl-sourcemgr add --uri https://deb.debian.org --distro stretch -C main -C contrib -C non-free --enabled --type deb-src

should add the above repositories if they are not already enabled. An advanced feature could be

bl-sourcemgr check --allow-uri-matches --uri 'http://.+\.debian.org$' -D stretch -C main --enabled

for checking if any Debian mirror in the classic format is enabled for component main on the system.

2. I'm calling this script simple because the required APIs  for managing the sources.list files and entries already exist in python-apt! All that is needed is a sane harness which unites the available functions in a useful interface for us.

3. This kind of tool would allow us to:

3.1 …manage repository entries (deb-multimedia, backports, etc pp) cleanly, be it from bl-welcome or from the command line.

3.2 …allow us to sanitze sources.list configurations in bl-welcome without having to resort to sed/awk/cut magic, and solve the problem of dealing with messed-up sources.list configurations.

------

What do you think? Good to have or overkill? Should I get too busy again and progress on this stall, I wouldn't want this to be a release blocker –– if not for the initial Helium release, then I'd be fine with this to be deployed and integrated later.


Im grünen Wald, dort wo die Drossel singt…

Offline

#2 2018-01-28 00:33:31

tknomanzr
BL Die Hard
From: Around the Bend
Registered: 2015-09-29
Posts: 1,029

Re: apt-sourcemgr: simple APT sources.list entry management

l personally like this idea. It would be even better if you could integrate the mirror selection script that Debian uses into it as well.

Offline

#3 2018-01-28 01:29:32

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

Re: apt-sourcemgr: simple APT sources.list entry management

tknomanzr wrote:

l personally like this idea. It would be even better if you could integrate the mirror selection script that Debian uses into it as well.

That is also included in python-apt by way of the distro/distinfo abstraction and should therefore be possible. The system uses a special info file e.g. https://github.com/BunsenLabs/bunsen-os … python-apt per distribution (of course, Debian has its own info file with a much longer mirror list).


Im grünen Wald, dort wo die Drossel singt…

Offline

#4 2018-01-28 04:25:21

tknomanzr
BL Die Hard
From: Around the Bend
Registered: 2015-09-29
Posts: 1,029

Re: apt-sourcemgr: simple APT sources.list entry management

It is actually not that uncommon a scenario for me to come into a fresh Bunsenlabs install with functional internet but an empty sources.list file because Bunsenlabs is capable of dealing with situations the Debian installer cannot. It is not a big deal to look up on the internet and copy but a script that dealt with those issues would be really nice.

Offline

#5 2018-01-28 05:35:01

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

twoion wrote:

bl-welcome:
Deal with cases where sources.list is completely messed.
https://github.com/BunsenLabs/bunsen-ne … /issues/12

I propose to implement a simple script called bl-sourcemgr, to become part of bunsen-utilities...

...What do you think? Good to have or overkill? Should I get too busy again and progress on this stall, I wouldn't want this to be a release blocker –– if not for the initial Helium release, then I'd be fine with this to be deployed and integrated later.

@twoion this sounds excellent!!
Pull that whole bundle of functionality out of bl-welcome and make it available where other scripts, and users via cli, can access it. Just like a library: any improvements and bugfixes will need to arrive in one place only. cool

Packaging: part of bunsen-utilities, or new package? Bunsen-welcome will need to set a hard dependency once it starts using the bl-sourcemgr calls, and bunsen-utilities is a 2MB download. That might not be a serious problem though.

Timing: if it makes the Helium release that's great (though to actually use it would mean changing bl-welcome, if in a good way) but, since the only change to the user experience would be the new sources.list fixing functionality, a package upgrade after the Helium release should work too.

Anyway, I think utilities like this are a great addition to BunsenLabs, and help make other code simpler and more transparent.
When the api is finalised (even before all the implementation details) maybe you could let me know the syntax, and I'll try recoding bl-welcome, in a test git branch, to use it. It will be interesting to see how much the code can be simplified. smile


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#6 2018-01-28 10:01:22

ohnonot
...again
Registered: 2015-09-29
Posts: 3,191
Website

Re: apt-sourcemgr: simple APT sources.list entry management

re post #20:
it's the DE/distro trap again.
i'm sure this is how monstrosities like GNOME and Ubuntu started 15 years or so ago:
with really good ideas of light abstraction tools that would make their product more usable.

BL devs, if you continue down this road BL will soon become undistinguishable from 90% of the stuff you see on distrowatch.

Offline

#7 2018-01-28 16:28:34

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

Re: apt-sourcemgr: simple APT sources.list entry management

@John, here's a fully-functional proof-of-concept (already works!): https://github.com/BunsenLabs/bunsen-sourcemgr. The actual program is less than 100 lines of code and almost complete. I've written extensive help text (--help) if you'd like to take look.

It's basically a match-and-do concept: Use the specification arguments as filters and apply the action to all matching entries. `add` is different yet similar: It obivously adds stuff if it doesn't exist, but python-apt is smart enough to just modify existing entries if they are similar enough (e.g. only adding a component like non-free).

Not yet implemented:

* `remove` is only `soft` for now because python-apt is missing deletion functionality. As for now, entries which reside in files that have other entries won't get deleted but only commented out/disabled, only deleted if they reside in a single file. Good enough for a first. TODO: Implement deletion also when all entries in a file are scheduled for deletion (delete useless file).
* Logging (--verbose) option.
* Exit codes are not very useful yet. `find` should return non-zero if no matching entries are found, for example.
* Privilege check (except for `find`, all actions require `root`).
* Mirror selection not yet integrated (needs some research how to use it).
* ?


Im grünen Wald, dort wo die Drossel singt…

Offline

#8 2018-01-28 19:57:08

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

Re: apt-sourcemgr: simple APT sources.list entry management

^ OK, I've now added support for python-apt templates! For example on BL Hydrogen, under the premise that python3-apt is installed (which needs to be in our base page set shipped on the ISO and netinstall), assuming a completely deleted sources.list file, regenerating the system APT sources configuration from the template is as easy as

bl-sourcemgr apply-template -T BunsenLabs -d bunsen-hydrogen
bl-sourcemgr apply-template -T Debian -rd '^jessie' # select all distributions starting with jessie (because Debian ships all known distros

I'm developing and testing on Stretch, and

bl-sourcemgr apply-template -T Debian -rd '^stretch'

yields the following sources.list:

deb http://deb.debian.org/debian/ stretch main contrib non-free
deb http://security.debian.org/ stretch/updates main contrib non-free
deb http://deb.debian.org/debian/ stretch-updates main contrib non-free
deb http://deb.debian.org/debian/ stretch-proposed-updates main contrib non-free

Obviously, the --distribution regex needs to be fine-tuned, but it works! What's not yet implemented is mirror selection from the mirror list. The way this works is that the .mirror file specifies a list of mirrors, while the .info file with the distros in it specifies a regex as match_uri which needs to be matched against the mirrors' hostnames in order to find all mirrors that work for this distro. For now, I'm preferreing the base_uri setting which Debian has pointed at the deb.debion.org redirector which should always work for a bootstrap/recovery.

I'll also add a way to exclude some components from being added, which should be mostly covered by --exclude-non-free -- the templates themselves to not distinguish components further, they just say what's available.

This template stuff starts making more sense the longer I'm working with it.


Im grünen Wald, dort wo die Drossel singt…

Offline

#9 2018-01-28 21:26:14

tknomanzr
BL Die Hard
From: Around the Bend
Registered: 2015-09-29
Posts: 1,029

Re: apt-sourcemgr: simple APT sources.list entry management

I haven't had a chance to test this out yet but looking at the source, it is looking good. Should we move posts regarding this program into its own post for better searchability?

Offline

#10 2018-01-28 21:29:53

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: apt-sourcemgr: simple APT sources.list entry management

tknomanzr wrote:

Should we move posts regarding this program into its own post for better searchability?

Yes, that sounds like a good idea smile

@twoion: great work here, thank you for this.

@ohnonot: I'm not sure I agree because this is an extension of an already extant API that can be used to simplify bl-welcome significantly.

I think it could also be pushed upstream (once finished) as it is potentially useful for other Debian-based distributions.


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#11 2018-01-28 21:52:09

dhalgren
Member
Registered: 2015-10-01
Posts: 75

Re: apt-sourcemgr: simple APT sources.list entry management

ohnonot wrote:

re post #20:
it's the DE/distro trap again.
i'm sure this is how monstrosities like GNOME and Ubuntu started 15 years or so ago:
with really good ideas of light abstraction tools that would make their product more usable.

BL devs, if you continue down this road BL will soon become undistinguishable from 90% of the stuff you see on distrowatch.

I usually keep well out of this type of discussion, but ....

+1

If it hasn't been needed, it probably is not needed now.

Offline

#12 2018-01-28 22:00:33

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

Re: apt-sourcemgr: simple APT sources.list entry management

dhalgren wrote:
ohnonot wrote:

re post #20:
it's the DE/distro trap again.
i'm sure this is how monstrosities like GNOME and Ubuntu started 15 years or so ago:
with really good ideas of light abstraction tools that would make their product more usable.

BL devs, if you continue down this road BL will soon become undistinguishable from 90% of the stuff you see on distrowatch.

I usually keep well out of this type of discussion, but ....

+1

If it hasn't been needed, it probably is not needed now.

There's a difference between bloat and replacing existing code with more readable and therefore maintainable code. There will be less code there and better code yonder. I'm seeing this more like some kind of refactoring?


Im grünen Wald, dort wo die Drossel singt…

Offline

#13 2018-01-29 03:03:33

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

^Agreed.

@ohnonot while understanding your fears, in this case I think they're misplaced. Nobody is proposing to replace the current apt configuration with some Gnomish overlay, this is just a tool to make editing simpler and more transparent. Some current code in bl-welcome will use this utility instead of grep/sed/awk and friends. Users will still be able to edit /etc/apt/sources.list by hand and totally ignore bl-sourcemgr, although it will be available as a terminal utility if they want to use it.

Extra disk space will be tiny, and no system resources will be consumed except when the tool is being run. Bunsen-welcome will depend on bunsen-sourcemgr, but you can remove both of them if you want. Neither are essential for experienced users.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#14 2018-01-29 07:35:30

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

This discussion now in its own thread.

@twoion the thing that first comes to mind is making querys of the current system state. Is "find" enough to determine, for example, whether bunsen-backports have been added or not, or would a new verb "query" be needed? (btw "find" is not explained in the --help output.)

For example, on a Jessie system:

john@bunsen1:/data/john/projects$ python3 ./apt-sourcemgr find -Ed jessie-backports
deb http://ftp.riken.jp/Linux/debian/debian/ jessie-backports main contrib non-free
deb https://pkg.bunsenlabs.org/debian jessie-backports main

Also, is it possible to allow commands to use the --simulate option without root privileges?

EDIT: answered my first question:

john@bunsen1:/data/john/projects$ python3 ./apt-sourcemgr find -rEd jessie-backports -u 'bunsenlabs.org'
deb https://pkg.bunsenlabs.org/debian jessie-backports main
john@bunsen1:/data/john/projects$ echo $?
0
john@bunsen1:/data/john/projects$ python3 ./apt-sourcemgr find -rEd jessie-backports -u 'bunsenlabs.net'
john@bunsen1:/data/john/projects$ echo $?
2

Last edited by johnraff (2018-01-29 07:40:09)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#15 2018-01-29 21:31:08

ohnonot
...again
Registered: 2015-09-29
Posts: 3,191
Website

Re: apt-sourcemgr: simple APT sources.list entry management

johnraff wrote:

@ohnonot while understanding your fears, in this case I think they're misplaced. Nobody is proposing to replace the current apt configuration with some Gnomish overlay, this is just a tool to make editing simpler and more transparent. Some current code in bl-welcome will use this utility instead of grep/sed/awk and friends.

i thought you were talking abotu a gui utility a lá ubuntu's software sources.

i retract my comment for this particular situation, but it isn't the first time i said this, so the genral sentiment prevails on my side.

folks, i don't want to dis your efforts, but sometimes it's easy to loose sight of the forest because of too many trees, so i just call out every now and then.

Offline

#16 2018-01-30 00:09:02

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

Re: apt-sourcemgr: simple APT sources.list entry management

johnraff wrote:

Also, is it possible to allow commands to use the --simulate option without root privileges?

Thank you. I addressed this bug in one of the latest commits, and documented `find`. Added Debian packaging;

dpkg-buildpackage -b -us -uc

will now build a package from the local sources if desired.


Im grünen Wald, dort wo die Drossel singt…

Offline

#17 2018-01-30 01:50:39

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

^Thank you!
I'll build a package and try it out today.

ohnonot wrote:

...the general sentiment prevails on my side.
...so i just call out every now and then.

Please continue to do so - this kind of input is very important. BL is a community project after all. cool


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#18 2018-01-30 08:17:47

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

btw does this package not depend on bunsen-python-apt-template?

EDIT built and installed OK.
Quick run suggests it's working, but no extensive testing yet.

Last edited by johnraff (2018-01-30 08:40:35)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#19 2018-01-30 11:51:01

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

Re: apt-sourcemgr: simple APT sources.list entry management

johnraff wrote:

btw does this package not depend on bunsen-python-apt-template?

EDIT built and installed OK.
Quick run suggests it's working, but no extensive testing yet.

Tests need to be written too. No point in doing it manually all the time.


Im grünen Wald, dort wo die Drossel singt…

Offline

#20 2018-02-02 07:24:52

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

@twoion I see this package is still under heavy development. cool

Considering bl-welcome works as it is (albeit with sed and grep), are you OK with moving this project to the post-release section of our TODO list? Except for the promised ability to fix corrupted apt sources, which will of course be very welcome, there would be no significant change to the user interface.

As for bl-welcome's interaction with this, I'm thinking of three stages:
1) Rewrite functions like haveDebianBackports() to call apt-sourcemgr. The functions will be simplified, but the remainder of the scripts don't have to change.
2) Add an early page to inspect the machine's apt sources and offer to fix them if they seem to be corrupted.
3) Rewrite the whole flow of bl-welcome on a check-list basis: collecting user decisions and executing all the operations needed at the end. This would avoid multiple 'apt-get update' calls, and allow users to leave the machine alone once it started doing what it had been asked. (This might be something to introduce with Lithium/Buster.)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#21 2018-02-02 14:23:30

earlybird
ほやほや
Registered: 2015-12-16
Posts: 604
Website

Re: apt-sourcemgr: simple APT sources.list entry management

johnraff wrote:

@twoion I see this package is still under heavy development. cool

Considering bl-welcome works as it is (albeit with sed and grep), are you OK with moving this project to the post-release section of our TODO list? Except for the promised ability to fix corrupted apt sources, which will of course be very welcome, there would be no significant change to the user interface.

Yes that's fine!

As for bl-welcome's interaction with this, I'm thinking of three stages:
1) Rewrite functions like haveDebianBackports() to call apt-sourcemgr. The functions will be simplified, but the remainder of the scripts don't have to change.

Good thing it's modular.

2) Add an early page to inspect the machine's apt sources and offer to fix them if they seem to be corrupted.

Hm, corrupt vs just not configured vs incomplete. After reading the source, python-apt's notion of 'invalid' seems only to apply to LINES in any sources.list file, and not the entries per se. I think without overcomplicating things, this would cover "just not configured" and "incomplete".

3) Rewrite the whole flow of bl-welcome on a check-list basis: collecting user decisions and executing all the operations needed at the end. This would avoid multiple 'apt-get update' calls, and allow users to leave the machine alone once it started doing what it had been asked. (This might be something to introduce with Lithium/Buster.)

A very good design improvement, because the `do` part can be confined to a single script/processor, reduce the surface to introduce weird bugs and on the way make the prozess more transparent. We could save a something like a `welcome.config` file away too which would the user to seed or reapply or change any settings made in BL-Welcome in an easy way. A bit like kernel make.

# welcome.config
ENABLE_BACKPORTS=y
ENABLE_NONFREE=y
# ...

Online

#22 2018-02-02 23:31:00

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

earlybird wrote:

We could save a something like a `welcome.config` file away too which would the user to seed or reapply or change any settings made in BL-Welcome in an easy way. A bit like kernel make.

# welcome.config
ENABLE_BACKPORTS=y
ENABLE_NONFREE=y
# ...

This sounds like a good idea. Have bl-welcome work in two separate stages:
1) collect user choices and make the config file
2) enact whatever is in the file

So, as you say, a user could edit welcome.config by hand and re-run part 2).


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#23 2018-02-02 23:56:24

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

earlybird wrote:

2) Add an early page to inspect the machine's apt sources and offer to fix them if they seem to be corrupted.

Hm, corrupt vs just not configured vs incomplete. After reading the source, python-apt's notion of 'invalid' seems only to apply to LINES in any sources.list file, and not the entries per se. I think without overcomplicating things, this would cover "just not configured" and "incomplete".

The reason this came up, was this GitHub issue: https://github.com/BunsenLabs/bunsen-ne … /issues/12
If BL is installed without internet then apt sources are not configured properly.
The "netinstall" installation route implies net access, so we must be talking about an iso install.

Other unusual install routes, like installing bunsenlabs-meta-all over an existing system, would also result in mixed sources.

(With the iso install we are dealing with the behaviour of the Debian installer. This might change in the future of course.)

Anyway, what would be the most useful way for bl-welcome to handle the situation?

If the user already has some non-standard repos enabled - for whatever reason - keep them, or disable them?

Would this approach work?
Check if the expected sources - and only those - are enabled, and if some non-standard situation is found, to ask the user for permission, then disable everything, keep backup files or commented-out entries, and replace it all with a standard BL repo setup. Finally let the user know how to restore what they had, should they want that.


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

#24 2018-02-03 12:02:22

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,227

Re: apt-sourcemgr: simple APT sources.list entry management

johnraff wrote:

Would this approach work?
Check if the expected sources - and only those - are enabled, and if some non-standard situation is found, to ask the user for permission, then disable everything, keep backup files or commented-out entries, and replace it all with a standard BL repo setup. Finally let the user know how to restore what they had, should they want that.

Yes, but we could no an even more

Check if there are non-stretch sources OR non-helium distros enabled:

./apt-sourcemgr find -rd '^(?!stretch|helium)'

The return value will be 0 if this check is positive.

Next, disabling everything non would work like this:

./apt-sourcemgr disable

Then, enable all relevant Stretch sources and BL Helium sources using the standard templates.

# Eliminate stretch-proposed-updates. We could use a (?!)-type blacklist regex
# here too but a whitelist is more intuitive I think.
./apt-sourcemgr apply-template -T Debian -rd '^(stretch|stretch-updates|stretch/updates)$'
./apt-sourcemgr apply-template -T BunsenLabs -d helium

I think we had a question if the user'd like to enable non-free software too? We can remove non-free from the list like so:

./apt-sourcemgr apply-template -T Debian -rd '^(stretch|stretch-updates|stretch/updates)$' -c '^(?!non\-free)'

Im grünen Wald, dort wo die Drossel singt…

Offline

#25 2018-02-04 02:59:47

johnraff
nullglob
From: Nagoya, Japan
Registered: 2015-09-09
Posts: 4,671
Website

Re: apt-sourcemgr: simple APT sources.list entry management

twoion wrote:
./apt-sourcemgr apply-template -T BunsenLabs -d helium

Where does the information implied by '-T BunsenLabs' come from?
Are we indeed using the bunsen-python-apt-template package?
(If so, it will need to go in the dependencies.)


John
--------------------
( a boring Japan blog , Japan Links, idle twitterings  and GitStuff )
In case you forget, the rules.

Offline

Board footer

Powered by FluxBB