[FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

Bearded_Blunder · 2022-01-11 06:51:11

On a second run bl-welcome will offer to add contrib & non-free even when it has previously added them, at least for the security section.

If you then choose "A" in response, it adds them again & you'd get an ever growing sources.list (I now have two blocks at the foot).

Either it's missing out detecting if it added the entries before, or else that check is somehow failing.

Should be easy enough to reproduce, remove contrib & non-free from the security repo & run bl-welcome twice (I had a reboot in between, though I doubt that matters).

Last edited by Bearded_Blunder (2022-01-17 05:36:33)

johnraff · 2022-01-11 07:05:52

^many thanks I'll have a look at that.
It would be nice to have that new function working properly.

DeepDayze · 2022-01-11 21:14:09

Why not have a first-run section in bl-welcome where things can be asked on the very first run after install and then on subsequent runs those items then get skipped (unless there was a failure).

Bearded_Blunder · 2022-01-11 21:50:01

Maybe because some people blast through it, see something offered & think "later".

It's my fault, I generally find inventive ways to break things.

DeepDayze · 2022-01-11 22:04:01

Bearded_Blunder wrote:

Maybe because some people blast through it, see something offered & think "later".
It's my fault, I generally find inventive ways to break things.

Lol don't we all, it's human nature 8o

johnraff · 2022-01-12 02:27:32

Bearded_Blunder wrote:

I generally find inventive ways to break things.

People who do that are so valuable!
Especially at this pre-release phase.

Anyway I'm just about to dive in and see if I can pin down what's happening...

Bearded_Blunder · 2022-01-12 02:30:55

Best of luck, I'd have looked myself, but the way bl-welcome is structured confuses me, I can sometimes to bordeline usually puzzle out a long monolithic script.. not sure what chunk I'd look for or where with bl-welcome
Hope it's simple!

johnraff · 2022-01-12 05:50:58

This is what I was afraid would happen - I can't reproduce the bug on my VM.

Here is my /etc/apt/sources.list after I commented out the security line, replaced it with one lacking contrib & non-free, and ran bl-welcome:

# 

# deb cdrom:[Debian GNU/Linux bullseye-DI-alpha3 _Bullseye_ - Official Snapshot amd64 NETINST 20201203-12:50]/ bullseye main

#deb cdrom:[Debian GNU/Linux bullseye-DI-alpha3 _Bullseye_ - Official Snapshot amd64 NETINST 20201203-12:50]/ bullseye main

deb http://deb.debian.org/debian/ bullseye main contrib non-free
#deb-src http://deb.debian.org/debian/ bullseye main contrib non-free

#deb http://deb.debian.org/debian-security bullseye-security main contrib non-free
#deb-src http://security.debian.org/debian-security bullseye-security main contrib non-free
#edited for bugtesting
deb http://deb.debian.org/debian-security bullseye-security main


# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.

# bullseye-updates, previously known as 'volatile'
#deb https://deb.debian.org/debian bullseye-updates main contrib non-free

# added by bl-welcome on Tue 23 Nov 13:33:10 JST 2021
# Debian Updates repository
deb https://deb.debian.org/debian bullseye-updates main contrib non-free


# added by bl-welcome on Wed 12 Jan 14:40:20 JST 2022
# Debian Security repository with missing components
deb https://deb.debian.org/debian-security bullseye-security contrib non-free

A subsequent run of bl-welcome found nothing missing, nor another one after doing a reboot.

The only thing that caught my eye was the original entry's http:// vs bl-welcome's https:// but the script seemed to accept either OK.
('inxi -r' also reported the situations accurately.)

@B_B could you maybe post your sources.list for comparison?

EDIT:
~/.cache/bunsen-welcome/bl-welcome.log might have a hint? Look around the page 2 section, just after doing the apt update. When contrib or non-free are found to be missing from the output of 'apt-cache policy' (often the case with security) then a different check is done of 'apt-get indextargets ...' which should return OK. Here's what I have:

[H  ------------------------------[ page 2 of 16 ]------------------------------

  UPDATE SOFTWARE SOURCES, CHECK SOURCES AND UPGRADE INSTALLED PACKAGES
  ---------------------------------------------------------------------
  
  Before we can continue, we need to make sure your system's software is 
  up-to-date. This script will execute the commands:
  
      sudo apt-get update
      sudo apt-get upgrade --with-new-pkgs
  
  It will also check your configured apt sources to make sure that all 
  necessary repositories are available.
  
  If you do not want to upgrade now, you can run this welcome script any time 
  later in a terminal with the command "bl-welcome"
  
  Would you like to upgrade your software? [Y/n] 
Hit:1 http://deb.debian.org/debian bullseye InRelease
Hit:2 https://deb.debian.org/debian bullseye-updates InRelease
Hit:3 https://deb.debian.org/debian bullseye-backports InRelease
Hit:4 http://deb.debian.org/debian-security bullseye-security InRelease
Hit:5 https://kelaino.bunsenlabs.org/~johnraff/debian beryllium InRelease
Reading package lists...
  Finished update
  
  Checking apt sources... 

Checked repositories status.
Data:
declare -A repo_status=([deb_update_main]=" 500 https://deb.debian.org/debian bullseye-updates/main amd64 Packages" [debian_contrib]=" 500 http://deb.debian.org/debian bullseye/contrib amd64 Packages" [bunsen_main]=" 500 https://pkg.bunsenlabs.org/debian beryllium/main amd64 Packages" [deb_bkpt_main]=" 100 https://deb.debian.org/debian bullseye-backports/main amd64 Packages" [deb_bkpt_contrib]=" 100 https://deb.debian.org/debian bullseye-backports/contrib amd64 Packages" [debian_main]=" 500 http://deb.debian.org/debian bullseye/main amd64 Packages" [debian_non-free]=" 500 http://deb.debian.org/debian bullseye/non-free amd64 Packages" [deb_sec_main]=" 500 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages" [deb_bkpt_non-free]=" 100 https://deb.debian.org/debian bullseye-backports/non-free amd64 Packages" )

contrib was found missing from bullseye-security + http://deb.debian.org/debian-security in apt policy - checking local configs

bullseye-security contrib found in user apt sources

non-free was found missing from bullseye-security + http://deb.debian.org/debian-security in apt policy - checking local configs

bullseye-security non-free found in user apt sources

contrib was found missing from bullseye-updates + https://deb.debian.org/debian in apt policy - checking local configs

bullseye-updates contrib found in user apt sources

non-free was found missing from bullseye-updates + https://deb.debian.org/debian in apt policy - checking local configs

bullseye-updates non-free found in user apt sources

OK

Current version of bunsen-welcome: 11.2.2-1

# etc etc

Last edited by johnraff (2022-01-12 06:58:50)

Bearded_Blunder · 2022-01-12 21:28:06

As requested, now I have the 32 bit VM back up, & the contrib & non-free entries removed again

/etc/apt/sources.list As mangled by bl-welcome

# deb cdrom:[Debian GNU/Linux 11.2.0 _Bullseye_ - Official i386 NETINST 20211218-11:16]/ bullseye main

#deb cdrom:[Debian GNU/Linux 11.2.0 _Bullseye_ - Official i386 NETINST 20211218-11:16]/ bullseye main

deb http://mirror.ox.ac.uk/debian/ bullseye main contrib non-free
deb-src http://mirror.ox.ac.uk/debian/ bullseye main contrib non-free

deb http://security.debian.org/debian-security bullseye-security main
deb-src http://security.debian.org/debian-security bullseye-security main

# bullseye-updates, to get updates before a point release is made;
# see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
deb http://mirror.ox.ac.uk/debian/ bullseye-updates main contrib non-free
deb-src http://mirror.ox.ac.uk/debian/ bullseye-updates main contrib non-free

# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.



# added by bl-welcome on Wed 12 Jan 20:37:13 GMT 2022
# Debian Security repository with missing components
deb https://deb.debian.org/debian-security bullseye-security contrib non-free


# added by bl-welcome on Wed 12 Jan 20:41:26 GMT 2022
# Debian Security repository with missing components
deb https://deb.debian.org/debian-security bullseye-security contrib non-free

Once it adds a second time of course I then get a spew of errors of the form below if I try to run apt update again.

W: Target Packages (contrib/binary-i386/Packages) is configured multiple times in /etc/apt/sources.list:26 and /etc/apt/sources.list:31

Could be disconcerting for a new user who just thought the first addition had failed.

Not seeing anything enlightening in bl-welcome.log

Did obviously forget to mention I picked a geographically pretty local mirror during setup, rather than just going with the default.

Now, obviously, off to edit my sources.list to be correct again.

Edit:
My guess would be, it's checking the url it finds, & not the one it adds later, since they're different. Or at least if it is checking both I'd look hard at the possibility the logic is falling through some crack there.

Last edited by Bearded_Blunder (2022-01-12 22:01:59)

johnraff · 2022-01-13 03:04:19

Bearded_Blunder wrote:

My guess would be, it's checking the url it finds, & not the one it adds later, since they're different. Or at least if it is checking both I'd look hard at the possibility the logic is falling through some crack there.

Looking at your sources list that was my guess too.

The script is supposed to work with any mirror the user might set, because it's looking at what apt reports about the sources, rather than trying to parse sources lists itself. So the problem might well be, as you say, that the added url is the default https://deb.debian.org not the user's set mirror. So if the function that adds contrib non-free is passed the url that provided main then it might not be hard to fix.

Fingers crossed...

---
BTW if you're curious, the checking is done on this page:
https://github.com/BunsenLabs/bunsen-we … ck-upgrade
using functions getRepoStatus() and checkAptTargets() defined here:
https://github.com/BunsenLabs/bunsen-we … bl-welcome

johnraff · 2022-01-13 09:21:49

bunsen-welcome 11.3-1 and bunsen-common 11.1-1 have been uploaded, and should fix this bug with any luck.

If you'd like to do an apt update/upgrade, return sources.list to its original condition (with the missing contrib non-free) and run bl-welcome one more time...

@B_B your suggestion of allowing for the quotes round "I understand" has also finally been implemented.

Bearded_Blunder · 2022-01-14 01:13:07

Both things now appear to work too, which is good news.

You *can* still end up with two entries added, but it requires manual intervention & would happen as follows:

Someone who'd set security.debian.org/debian-security decided to switch to the CDN & changed the first entry to deb.debian.org & missed the added one.

If they then ran bl-welcome again it could add the revised version & you end up with deb.debian and security.debian BOTH in the list.

Unlike what was happening previously though, apt doesn't see the added entries as duplicates & seems to continue to update rather than error out.

Above seems a pretty unlikely (though possible) scenario, I'll leave it to you to decide if you want to mark this fixed or not...
"Good enough" or "Needs more polish" is a judgement call at this point.

johnraff · 2022-01-14 02:42:34

I'm going to go with "Good enough".

The error is gone - in fact it's OK for apt to have multiple repos offering the same packages. If the same version of a package is available, the repo written first in the list takes priority. (The previous error was from multiple entries of the same repo.) The user who manually edited their sources list will likely clean up the unnecessary entry eventually...

And many thanks both for finding this bug and helping to squash it!

bl-welcome isn't something people are expected to run regularly - the aim here is to get systems up and running that
1) have been hit by the Debian Installer issue where installs done offline don't get the full apt sources
or
2) are based on a standard Debian netinstall and whose admin might have forgotten to enable config and non-free

1) especially comes up regularly so if we've fixed that it's a step forward.

Bearded_Blunder · 2022-01-14 03:43:48

johnraff wrote:

2) are based on a standard Debian netinstall and whose admin might have forgotten to enable config and non-free

Speaking of artefacts of the netinstall, there's the admins who get adding contrib & non-free right, but don't even think about the fact that deb-src lines waste some bandwidth & slow apt update down, would it be worth bl-welcome offering to comment them?

I thought it used to.. recently I'd have said no, but that's only because at least with Buster switching init & having stuff actually work well enough to use involved recompiling some stuff.

I only tripped over this little glitch because I'd rushed the install in a VM.

johnraff · 2022-01-14 04:42:52

Bearded_Blunder wrote:

deb-src lines waste some bandwidth & slow apt update down, would it be worth bl-welcome offering to comment them?

Not necessary IMO. Our installer comments out deb-src OOTB. Users installing via metapackage on Debian netinstall can take care of themselves. Having deb-src enabled isn't broken and some people want it.

Bearded_Blunder · 2022-01-14 05:06:41

Fair enough, not much of the world on dialup, isdn, or slow dsl anymore, & at least where I am even metered connections are the exception & not the rule.

Bearded_Blunder · 2022-01-14 21:14:59

One other thing though, if someone installs on Devuan, rather than Debian, it offers to add Debian repos to their system, I predict b0rkage if they allow it to.

Perhaps a check they're actually on Debian? Then skip checking if they are? You'll want a check that covers any Linux system people might try it on.

Edit, and don't offer them DEBIAN backports either.

Last edited by Bearded_Blunder (2022-01-14 21:23:05)

DeepDayze · 2022-01-14 21:23:04

Bearded_Blunder wrote:

One other thing though, if someone installs on Devuan, rather than Debian, it offers to add Debian repos to their system, I predict b0rkage if they allow it to.
Perhaps a check they're actually on Debian? Then skip checking if they are? You'll want a check that covers any Linux system people might try it on.

Why not try this in a VM? If there's b0rkage then you know that the script will not work on a Devuan based system without changes such as detection of debian version for example. Or add verbiage to inform users if running the script on Devuan systems they need to skip the question about Debian repositories.

Last edited by DeepDayze (2022-01-14 21:27:53)

Bearded_Blunder · 2022-01-14 21:26:43

I'm posting this as a direct result of trying on Devuan 4,

I suppose I *could* snapshot & see what mess ensues, but the script is going to add debian repos, then apt update etc. if I allow it to.

Plenty of the script, pretty much all.. ought to work, provided it doesn't go adding debian sources to non debian distros.

DeepDayze · 2022-01-14 21:29:01

Bearded_Blunder wrote:

I'm posting this as a direct result of trying on Devuan 4,
I suppose I *could* snapshot & see what mess ensues, but the script is going to add debian repos, then apt update etc. if I allow it to.
Plenty of the script, pretty much all.. ought to work, provided it doesn't go adding debian sources to non debian distros.

Try skipping the question and see what happens if you are trying to run bl-welcome on Devuan.

#1 2022-01-11 06:51:11

[FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#2 2022-01-11 07:05:52

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#3 2022-01-11 21:14:09

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#4 2022-01-11 21:50:01

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#5 2022-01-11 22:04:01

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#6 2022-01-12 02:27:32

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#7 2022-01-12 02:30:55

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#8 2022-01-12 05:50:58

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#9 2022-01-12 21:28:06

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#10 2022-01-13 03:04:19

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#11 2022-01-13 09:21:49

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#12 2022-01-14 01:13:07

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#13 2022-01-14 02:42:34

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#14 2022-01-14 03:43:48

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#15 2022-01-14 04:42:52

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#16 2022-01-14 05:06:41

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#17 2022-01-14 21:14:59

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#18 2022-01-14 21:23:04

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#19 2022-01-14 21:26:43

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

#20 2022-01-14 21:29:01

Re: [FIXED][Beryllium] bl-welcome offers to re-add contrib & non free.

Board footer