Working on bl-welcome to fix missing repos

johnraff · 2021-03-28 06:35:40

Still a WIP but this is where my "BL time" is going atm. Cases of people installing without a net connection, and ending up with the main Debian repos missing from sources.list, keep coming up, and it looks like a good candidate for bl-welcome to help with.

Instead of grepping the files directly I made a function to parse the output of 'apt-cache policy' and generate an array holding the state of the various repos. It's then easy for following code to check if a certain repo is enabled or not - that's what I'm working on now. The only catch is, changes to the apt sources files aren't reflected in 'policy' till another 'apt update' is done. Don't want to do repeated updates, so some grepping might still be needed to check if eg Debian Backports have just been enabled or not.

Still, this should:
*) Check if Debian Bullseye, Debian Updates and Debian Security are enabled, and add if needed.
*) Check if 'contrib' and 'non-free' are enabled, and offer to add them. Allow free-only users to continue.
*) Likewise, check the BL repo.
*) List up any other, non-standard, repos found and check if the user is OK with having them. I'm hoping this will also catch left-over old repos after an upgrade.

That looks like a potential enhancement to me.

Wanted to keep you all up to speed on this, and also offer the chance to jump in with warnings/reservations/criticism etc

Is there some value in posting this in the public Development forum? Too early?

Here's the policy-parsing function. You could put it inside a script to run getRepoStatus() and then inspect the repo_status array (eg with 'declare -p repo_status' or 'echo ${repo_status[debian_main]}'). It won't do anything to your system.

# https://manpages.debian.org/testing/apt/apt_preferences.5.en.html
# generate global array repo_status based on state of 'apt-cache policy' at this point
# generated repo_status keys are:
# debian_{main,contrib,non-free} (each component is tested and recorded separately)
# deb_sec_{"} (debian security)
# deb_update_{"} (debian updates)
# deb_bkpt_{"} (debian backports)
# bunsen_main (bunsenlabs)
# Bunsen_bkpt_main (bunsenlabs backports)
# (bunsenlabs only provide "main" packages at the moment)
# Any other repositories found are given keys of unknown_<some number>
getRepoStatus() {
    declare -gA repo_status
    local -a policy
    mapfile -t policy < <(apt-cache policy)
    for ((i=0;i<${#policy[@]};i++))
    do
        line=${policy[i]}
        [[ $line =~ (([voanlcb]=[- [:alnum:]]+(,|$))+) ]] || continue
        line="${BASH_REMATCH[0]}"
        line="${line// /\\ }"
        src="${policy[i-1]}"
        local v o a n l c b # version origin archive/suite codename label component architecture
        unset v o a n l c b
        eval "${line//,/ }"
        case "$o" in
        '')
            continue
            ;;
        Debian)
            case $n in
            "${debian_base}")
                case $l in # in Buster, security codename is "buster" so need to check label
                Debian)
                    repo_status[debian_$c]=$src
                    ;;
                Debian-Security)
                    repo_status[deb_sec_$c]=$src
                    ;;
                *)
                    repo_status[unknown_$i]=$src
                    ;;
                esac
                ;;
            "${debian_base}-security") # in Bullseye, security codename is "bullseye-security"
                repo_status[deb_sec_$c]=$src
                ;;
            "${debian_base}-updates")
                repo_status[deb_update_$c]=$src
                ;;
            *)
                repo_status[unknown_$i]=$src
            esac
            ;;
        'Debian Backports')
            [[ $n = "${debian_base}-backports" ]] && {
                repo_status[deb_bkpt_$c]=$src
            }
            ;;
        bunsenlabs)
            case $n in
            "${bunsen_codename}")
                repo_status[bunsen_$c]=$src
                ;;
            "${debian_base}-backports")
                repo_status[bunsen_bkpt_$c]=$src
                ;;
            *)
                repo_status[unknown_$i]=$src
            esac
            ;;
        *)
            repo_status[unknown_$i]=$src
            ;;
        esac
    done
}

Last edited by johnraff (2021-03-28 07:26:54)

nobody · 2021-03-28 10:01:57

I have a dead project that went into the direction, but isn't ready: https://github.com/BunsenLabs/apt-sourcemgr

I think any logic written in a script should try to use "add-apt-repository" as much as possible; it's not pretty code under the hood but it is well used code (by Ubuntu, that is).

The difference between add-apt-repository and apt-soucremgr is like your script: it supports everything that add-apt-repository does not. It does not support detection, but adding and removing sources should work. It MIGHT depend on a specific file naming under sources.list.d, and it MIGHT depend on python-apt-templates (which we ship in bunsen-os-release for "BunsenLabs" distro)

I should work on Beryllium as well.

There were some bugs, but I don't really remember them.

Could revive the project? Although I would write the CLI part very differently today (mostly with click, dropping much of the custom command line handling).

johnraff · 2021-03-30 08:53:23

if you felt like picking up apt-sourcemgr again it would have some advantages:
*) More flexible, allowing users or writers of other scripts to access the same functions, not just locked inside bl-welcome.
*) Available for other derivative distros to pick up and use too.

But I think it might be a lot of work. Right now, as far as I can tell, it's using python3-apt which is getting regular updates OK.

But as for "add-apt-repository", it's provided by software-properties-common, source software-properties which is looking a bit neglected. One of its more recent upgrades was to work round a bug with gpg keys by using the deprecated apt-key. The source repo listed on the Debian Dev. page is outdated, but the salsa repo was last updated 2 years ago. Another outstanding bug is that it doesn't handle the new debian security url. So you can see I'm not so enthusiastic about "add-apt-repository".

Anyway, what bl-welcome needs is a way to test if various Debian repos and/or components are enabled, send the user appropriate messages and offer options to: add the missing elements, exit and fix it manually, or (in the case of non-free components) continue without them. Then apply the requested actions. I guess apt-sourcemgr can act as a backend for querying the apt sources, and then for adding the missing items if needed.

So, some UI will be needed anyway. I've sketched out a checkRepos() function to look at the repo_status array populated by the previous getRepoStatus(), and addRepo() to do the work, but it wouldn't require much change (I guess) to use apt-sourcemgr instead. Our needs are a rather small subset of what apt-sourcemgr is potentially capable of.

Here's what I've got atm (minus a couple of variables set elsewhere) - it could be adapted to use your utility, or if you hit a wall, or don't feel like doing it, I think there's enough code here to continue anyway:

checkRepos(){
    local -a missing_components
    local repo name shortname unknown_list said
    said=false
    repos_changed=false # checked by calling code
    printf '%s' 'Checking apt sources... '
    getRepoStatus || return 1
    log "Checked repositories status.
Data:
$(declare -p repo_status)"
    for repo in debian deb_sec deb_update # for now, only Debian repos are checked
    do
        case "$repo" in
        debian)
            name="main Debian ${debian_base} repository"
            shortname='Debian main'
            ;;
        deb_sec)
            name="Debian Security ${debian_base} repository"
            shortname='Debian Security'
            ;;
        deb_update)
            name="Debian Updates ${debian_base} repository"
            shortname='Debian Updates'
            ;;
        esac
        if [[ -z ${repo_status[${repo}_main]} ]]
        then
            said=true
            say "
The $name seems to be missing from your apt sources.\
 This can happen if an install is done without an internet connection.
This script can add the missing entry now, or you might prefer to exit and edit /etc/apt/sources.list yourself.\
 In that case, you can run this script in a terminal later with the command 'bl-welcome'"
            if prompt "Add the $shortname now?"
            then
                say $'\n'$"Adding $name repo to apt sources..."$'\n' 1
                addRepo "$shortname" || return 1 # all components will be added
                repos_changed=true
            else
                say 'Exiting...' 3
                exit 0
            fi
        fi
        for c in contrib non-free
        do
            [[ -n ${repo_status[${repo}_"$c"]} ]] || missing_components+=("$c")
        done
        if [[ ${#missing_components[@]} -gt 0 ]]
        then
            said=true
            say "
Non-free components are missing from your ${shortname} apt sources (${missing_components[*]}).\
 Without these, some packages will not be installable - for example some possibly necessary firmware.\
 This script can add the missing entries to your apt sources now, but you might prefer to exit and edit /etc/apt/sources.list yourself.\
 (You can run this script in a terminal later with the command 'bl-welcome'.) Alternatively you might want to continue with free software only."
            say "Would you like to add the missing components (A), exit now (Q) or continue anyway (C)?"
            local REPLY
            while true
            do
                read -r -p ' "A" to add non-free, "Q" to quit now, "C" to continue: '
                case $REPLY in
                A)
                    say $'\nAdding missing components ${missing_components[*]} to ${shortname} in apt sources...\n' 1
                    addRepo  "$shortname" "${missing_components[@]}" || return 1 # adds extra line with missing components
                    repos_changed=true
                    return 0
                    ;;
                Q)
                    say 'Exiting...' 3
                    exit 0
                    ;;
                C)
                    say 'Continuing...' 3
                    ;;
                esac
            done
        fi
    done
    for u in "${!repo_status[@]}"
    do
        [[ $u = unknown_* ]] || continue
        unknown_list+="${repo_status[$u]}"$'\n'
    done
    if [[ -n $unknown_list ]]
    then
        said=true
        say "
You have some unknown repositories enabled:
$unknown_list
If you are aware of these and want to go on using them, you can continue.
Otherwise, you might want to exit this script now and investigate
/etc/apt/sources.list and /etc/apt/sources.list.d
before running 'bl-welcome' again.
"
        prompt $"  Hit \"Enter\" to continue, or \"q\" to quit..." Q && exit 0
    fi
    [[ $said = true ]] || echo 'OK'
}

# Usage addRepo <name> [component [component]]
# If no components passed, all (main contrib non-free) are added.
addRepo() {
    local name components repo_msg repo_line missing sources_file
    sources_file='/etc/apt/sources.list'
    [[ -n $1 ]] || { echo "${0}: addRepo() needs a repo name" >&2; return 1;}
    name=$1
    shift
    missing=''
    if [[ -z "$*" ]]
    then
        components=(main contrib non-free)
    else
        missing=' with missing components' # extra comment
        components=("$@")
    fi
    repo_msg="# added by bl-welcome on $(date)
# $name repository${missing}
"
    case $name in
    'Debian main')
        repo_line="deb $default_mirror ${debian_base} ${components[*]}

"
        ;;
    'Debian Security')
        repo_line="deb ${deb_security_source} ${components[*]}

"
        ;;
    'Debian Updates')
        repo_line="deb $default_mirror ${debian_base}-updates ${components[*]}

"
        ;;

    *)
        echo "${name}: repo name not supported" >&2
        return 1
        ;;
    esac
    sudo tee -a <<< "${repo_msg}${repo_line}" "$sources_file" >/dev/null
    grep -q "^${repo_line}\$" /etc/apt/sources.list || { echo "Failed to add $name to apt sources." >&2; return 1;}
}

johnraff · 2021-03-30 09:05:21

Slightly away from the main topic, but changes are coming to Debian apt sources config syntax.

1) For better security, a signed-by option will tie a certain apt source to a specific gpg key, instead of throwing them all into /etc/apt/trusted.gpg.d.

2) The Deb822 file format will become the standard eventually. Even if we don't use it our tools will need to cope with it (one advantage of parsing 'apt-cache policy').

I just tested this on a VM in /etc/apt/sources.list.d/bunsen.sources and apt is quite happy:

Types: deb 
URIs: https://kelaino.bunsenlabs.org/~johnraff/debian
Suites: beryllium
Components: main
Signed-By: /usr/share/keyrings/lithium-dev.asc
Enabled: yes

https://wiki.debian.org/DebianRepository/UseThirdParty

Last edited by johnraff (2021-03-30 09:06:24)

nobody · 2021-03-30 10:21:17

Deb822 is for obvious reasons a much better format for configuring repositories than the old format...really no reason sticking to the old format. I think it would be even better to move to deb822 and "cope" with the old format instead, because that's what technical debt feels like

johnraff · 2021-03-31 04:29:14

^sure, let's do that. It's no particular extra work to write our sources in deb822, but of course any parsing tools we write will have to be able to handle both formats for some time to come. (That's why I favour parsing 'apt-cache policy' and leaving interpreting the source config to apt.)

The Debian installer still writes old format apt lines btw.

Anyway, if we're writing a new entry, easy to use deb822. If it's about editing an old-format entry, then there's the option of removing it and creating a deb822 source from scratch I guess.

nobody · 2021-04-01 15:44:44

I'll try to find some time over the Easter holidays to work on apt-sourcemgr.

johnraff · 2021-04-05 06:45:10

^Thanks! In that case I'll pause work on bl-welcome till we see how it goes.

I was hitting a snag with parsing 'apt-cache policy':
even if eg 'contrib' is enabled in apt sources, if there are no 'contrib' packages in the repo's release file then that component does not appear in 'policy'. Bl-welcome then thinks it needs to be enabled and prompts the user to add it, which triggers an apt Warning about double sources.

Does apt-sourcemgr have a way of going to what is actually configured in /etc/apt/sources.list{,.d/*}?

---
Anyway, while apt-sourcemgr is presumably meant to be a general utility with a potentially wide range of use cases, the specific tasks I would like bl-welcome to be able to use it for are:

*) Test for the presence of Debian, Debian-Security and Debian-Updates sources so if any are missing bl-welcome can show the user a message and prompt for action in each case.

*) Test for the existence of 'contrib' and 'non-free' in each of the above entries and suggest that the user might want to add them, if missing.

*) Test for Debian & BL backports so they can be offered to the user as an option.

*) List up any non-standard repos found (eg former Debian releases/Ubuntu/Virtualbox...) and ask the user if they want to keep them.

*) Add/remove any user-chosen entries in the apt sources.

If repeated queries would be time-consuming it might be desirable if the whole sources data could be dumped in a format that bl-welcome could use to populate an array, and query at leisure.

Optional:
rewrite all the apt sources in deb822 format

NOTE the Debian-Security sources format changed in Bullseye: https://wiki.debian.org/NewInBullseye

Last edited by johnraff (2021-04-06 07:54:38)

johnraff · 2021-04-06 07:54:58

(Added Backports to above list.)

johnraff · 2021-08-04 05:54:47

Just dropping this here for possible future reference: 'apt-get indextargets'

man apt-get wrote:

indextargets
Displays by default a deb822 formatted listing of information about all data files (aka index targets) apt-get update would download. Supports a --format option to modify the output format as well as accepts lines of the default output to filter the records by. The command is mainly used as an interface for external tools working with APT to get information as well as filenames for downloaded files so they can use them as well instead of downloading them again on their own.

Detailed documentation is omitted here and can instead be found in the file /usr/share/doc/apt-doc/acquire-additional-files.md.gz shipped by the apt-doc package.

Especially the --no-release-info option which bypasses remote repo downloads:

Note that by default targets are not listed if they weren't downloaded.
If you want to see all targets, you can use the `--no-release-info`, which
also removes the *Codename*, *Suite*, *Version*, *Origin*, *Label* and *Trusted*
fields from the output as these also display data which needs to be
downloaded first...

Acquire additional files in 'update' operations

#1 2021-03-28 06:35:40

Working on bl-welcome to fix missing repos

#2 2021-03-28 10:01:57

Re: Working on bl-welcome to fix missing repos

#3 2021-03-30 08:53:23

Re: Working on bl-welcome to fix missing repos

#4 2021-03-30 09:05:21

Re: Working on bl-welcome to fix missing repos

#5 2021-03-30 10:21:17

Re: Working on bl-welcome to fix missing repos

#6 2021-03-31 04:29:14

Re: Working on bl-welcome to fix missing repos

#7 2021-04-01 15:44:44

Re: Working on bl-welcome to fix missing repos

#8 2021-04-05 06:45:10

Re: Working on bl-welcome to fix missing repos

#9 2021-04-06 07:54:58

Re: Working on bl-welcome to fix missing repos

#10 2021-08-04 05:54:47

Re: Working on bl-welcome to fix missing repos

Board footer