How the Debian Packaging/Tracking Works : A Primer

Horizon_Brave · 2016-01-27 03:06:30

Also should be noted that this is a W.I.P, and will be updated and appended to as I go!
+Please, if I make a mistake, please please please feel free to edit it and let everyone know. I'd hate to spout wrong information...

Hoping to give the curious minded, a look at where packages are maintained, how they are handled and the process they go through. I'll be using the tracker.debian.org site which provides a plethora of information about tracking the life cycle of a package, who maintains them, and who verifies them.
As well as provide some acronym clarification. This will also serve as a sort of tutorial for the tracker.debian.org interface. I try not to assume any prior knowledge, and try to explain as I go about the Debian branches of Stable/Testing/Unstable and their interoperability. As many here as pointed me towards, a great primer for how a package is started from developer to release can be found here:
Packaging Intro

I'll try to provide as many sources as possible to my explorations and findings.

Let's get started!

The Ever Growing List of Acronyms (EGLoA)

BTS - Bug Tracking System
VCS - Version Control System
E&W - Errors and Warnings
RC - Release Critical / Release Candidate
std-ver- Standards Version
debu: Stable update
I&N- Important and Normal
M&W - Minor and Wishlist
F&P - Fixed and Pending
nmu- non-maintainer upload

Debian Versioning Syntax:

This sort of threw me off in the beginning so I hope to spare you the troubles. There *should* be a sort of standard to version software numbering. According to Debian policy, the format of a source package should look like:

[epoch:] upstream_version [-debian_revision]

To use the source package "bind9" as an example:

1 : 9 . 9 .5 .dfsg -9 +deb8u4

This is just one of it's many many versions. That looks cryptic as hell. Let's break this apart.

The Epoch Number
The number before the colon is the epoch number. This is a single digit number. It may be omitted, in which case zero is assumed. The epoch is provided to allow mistakes in the version numbers of older versions of a package, and also a package's previous version numbering schemes, to be left behind. For example, if the devs want to change the numbering scheme, they can just increment the epoch by 1. If it is omitted though, then the upstream_version field can not contain any colons, or else confusion arises.

*interrupt* Before we continue, remember how this would play out. If I'm a software developer, I would write a program. I want this program on as many distros as possible (most likely). I would save and tar my program, and give it a pretty generic name. What I just did was create the "native version". I'd keep this original program on my own computer. I'd then send a copy to Debian. The maintainers there refer to me as the "upstream".

The Upstream Version Number
After the epoch number (if present) is the upstream version number. To define, "upstream" means the author/developer. The developer may create version 0123 of his program. That is the "native" or original package version. If the package somehow is approved without him altering it for specifically Debian, then the native version number (0123) will remain just like this.
This field can only contain alphabet, numbers, and the characters . + - ~ and it should always begin with a digit. If there is no trailing debian_revision field, then the hyphen can not be used.

The Debian Version Number

The last portion of the Version Number is the debian number. This is the version of Debian that the package is built for. May contain only + . ~ But remember it is always prefixed with a hyphen. It is optional, but when included, it means the software build is different than the original "native" version, that is most likely still on the developer's PC. The developer of the software created his original version. Then created a customized version of it to fit into the debian arch.

Some Examples for it to hit home:

123+nmu3 = Third non-maintainer upload based on package version 0123

121-3+deb7u2 = Second stable update of the package 121-3 in Debian 7

TRACKER.DEBIAN.ORG

To be continued....

The following post or two, will provide a walkthrough of tracker.debian.org. How to use it, how to really dig down and get some really good information on packages, software and how to read it. I invite everyone to contribute as well. I'll try to make updates atleast once a day if possible, even if they're smaller ones.

Last edited by Horizon_Brave (2016-01-27 04:21:43)

Horizon_Brave · 2016-01-30 15:50:32

I'm going to assume that what I posted above is at least generally accurate since johnraff,damo,nobody,hhh, and the other elites haven't beaten me up yet.

Anyway This post will be to introduce:

The Life of a Source Package
Maintainers, Uploaders and Bears oh my!

Source to Binary: A Hobbit's Tale

Just to make sure we're all on the same page here, software has two "stages" First, when written and coded by the upstream developer(s), it is written as source code. This is the actual programming language that the software is written in. Late on we'll see that a 'finished' source package is usually comprised of three main files, a .dsc, a .orig.tar.gz, and a .debian.tar.gz (or .diff.gz).
These files allow creation of binary packages (.deb) from the source code files of the program.
When the package is given to ( or retrieved from a place like github or the author's website), it needs to be compiled into the proper binary files. This is done by the maintainer, usually using a variety of autoconf, automake, build essentials, and error checking programs like lintian. This process is actually pretty extensive and also involves gathering multi-architecture files, and linking libraries that the new program may or may not rely on.

Below shows a very very simple compilation...

(Re)-Naming

Once done, it may be named in anyway that the author chose at the time. So it's up to the Debian Maintainer to change the name, once the package has been compiled. This is to keep the formatting in at least a reasonable uniform pattern with the rest of debian software library. If the upstream source is named Example-Pack-0.9.12.tar.gz, you can take Example-Pack as the (source) package name and 0.9.12 as the upstream version number.

Once properly compiled and built, we're left with a bunch of packages, including a bunch of files, the original source .tarball, and of course the shiny new .deb binary files. Here's a list of some of the things you'd see, once the total build of the package is done:

example_program_0.9.12.orig.tar.gz This is the original upstream source code tarball, merely renamed to the above so that it adheres to the Debian standard. Nothing inside is changed as we want / need to have an original of the author's source.

example_program_0.9.12-1.dsc This is a summary of the contents of the source code. The file is used when unpacking the original source code. (expand upon)

example_program_0.9.12-1.debian.tar.gz This is a compressed tarball. It contains your debian directory contents. The debian directory is like your place to put all changes and addition you make to the original source code. Each and every addition you made to the original source code is stored here in, debian/patches.

example_program_0.9.12-1_i386.deb This is the completed binary package. Bascially where all the goodies are. You can use dpkg or apt to install and remove this just like any other package.

example_program_0.9.12-1_i386.changes This file describes all the changes made in the current package revision; it is used by the Debian FTP archive maintenance programs to install the binary and source packages. It is partly generated from the changelog file and the .dsc file.
As time goes one, new features will be added. People downloading your package can look at this file and quickly see what has changed.

Testing of the Package

The package must be installed to make sure the binaries were successfully built. We don't want any half baked cookies! There are many commands, and techniques to test rather the binary files made the trip intact and can run properly. This must be done before we get ready to upload to the public Debian servers.

Uploading and Updating

Afterward, the package is set to 'release'. It will most likely be forwarded to Unstable (Sid). It will soon need to be updated once bugs are found. (there will be, sorry) Let's say that a bug report was filed against the example package as #654321, and it describes a problem that you can solve.

The maintainer would correct the problem in the newly created package source for the upstream bug
If applicable, correct any problem in the previous patch;
Add a new revision at the top of the Debian change-log file for others to see what changed.
Set the status to "Unstable"
The Package is then reuploaded. THIS TIME though, the original source file archive isn't included, as it was the first time you uploaded it. This is because the debian server already has it! No changes were made to the original source package. The only changes were applied to the debian release package of it.
So the original already exists on the server.

Check out below for a sample changelog:

Note; you have the name and version of the source package, also the distribution to which it is initially uploaded to (unstable release), and the urgency level (used by Debian bot system to determine when the package is ready to enter testing release).

If the Author/Developer Upstream Makes Any Changes
When preparing packages of a new upstream release for the Debian archive, the maintainer must check the new upstream release, first. We don't want any malicous dev. slipping some rogue code in!

For this, usually the upstream changelog, NEWS, and whatever other documentation they may have released with the new version is consulted and read over.
If it checked out, and no funny business occurred, the swap is pretty simple and the new files replace the old. It's rebuilt and retested for installation.
Now that you have tested your new package thoroughly, you want to release it back to Unstable in the changelog. For this new upload you need to re-ship the entire contents including the author's source (since it's the source itself that was changed).

The Aftermath

This cycle will continue until the bug bots determine that the package is stable enough to move from unstable to debian's testing environment, appropriately labeled Testing. At this point the package(s) are ported to the Testing repos and it's off to the races.

From what I can tell those are the biggest,and most impactful steps in the process. Of course even when in Testing, bugs can and will still be reported in which the package will still be patched and re-uploaded.

This post took longer than I thought, so I delay the tour of the tracking system for next time!

damo · 2016-01-30 16:51:55

I'm going to assume that what I posted above is at least generally accurate since johnraff,damo,nobody,hhh, and the other elites haven't beaten me up yet

Hey, I'm definitely not "elite" when it comes to sources and packaging! I'm appreciating your precis of the Debian documentation though, and looking forward to the next installment.

Good idea to have something like this on the forum, for reference

Horizon_Brave · 2016-01-30 18:50:03

It's really the least I can do to provide something tangible. Just as a reminder, if i find any flaws or updates to prior posts ill make note in one of the newer posts and of,course edit the original. Anyway next post to demonstrate tracker.debian.org will be Sunday, ideally.

Horizon_Brave · 2016-01-31 02:59:50

Tracking a Software Package Part 1 of 2

Alright so I think I explained the packaging process as well as I understand it. I'd still like to point you to the resources that have been given to me, that I used as a great foundation. It's a lot of reading and information but it's definitely very thorough.

Debian Policy for Source Code
Debian Maintainers Guide
Developers Guide for Debian

Okay so this post will start at least a visual walkthrough of a very important site (in my opinion) called tracker.debian.org. This allows us a very cool, transparent and surprisingly deep look into the life cycle of packages and individual binaries. This will be part 1 of 2. Or even 1 of 3.

With that, let's mosey on over to tracker.debian.org

THE SEARCH PAGE:

This is the main interface to the site. Two things of note here. The first is the main search box. This allows you to type in a software package's or binary's name. As long as the package is being maintained by Debian, it will be retrieved. A word of note here. Many binaries, that appear as individual software packages are actually handled under one larger package. For example, the familiar "passwd" command. Passwd is a binary. But it belongs to, and is apart of a bigger source package of apps. If you were to type passwd into the search box, you'd see that the result is the retrieval of the "shadow" package. (sound equally familiar?) Let's take a look at what comes up… And don't worry we'll come back to the 'other' thing of note I mentioned on the homepage.

MAIN HOMEPAGE:

Lots and lots of information here, but let's break it down piece by piece. We searched for the password (passwd) tool because we're curious to see where, and how Debian updates this software.

Let's exam each of the sections:

Title of Package:
The resulting page returns the shadow package. So we can infer that the passwd tool is a binary inside of the shadow suite of packages. This is confirmed if you scroll down and look under the "binaries" section. We'll look at this a bit later.

General: Shows basic information about the current package.
The source link gives us links to the source packages that have been released. For example :

The above shows the source packages available of the 'shadow' package in Sid (unstable), Stretch (testing) and the last 3 stable releases.

Version: Shows the most current testing/unstable version of the package. This isn't the Debian Stable version.

Maintainer: This is actually a very useful page that displays what package version is currently 'assigned' what debian version. (oldstable, stable, testing, unstable, experimental). This actually shows ALL of the packages that this maintainer is responsible for. In this example there's only one so there's one row. But if this maintainer had more on his plate, there would be multiple rows here, each with it's own software package. Anyway, you can go through the fields, and by using some logic and version comparing you can see the difference in time between when packages were accepted to Testing, and when they went Stable, what architectures they are compatible with, and best of all, under the Deb Check column, you can see a list of issues that may effect the debian release version of the software. The S, T, U here each link to the bug history of each software package for Stable, Testing, and Unstable. In that case of shadow, there seems to be errors about how the package has dependencies on optional packages.

There's a few key points here I wish to point out. Notice the versioning numbers of the shadow software. Note how the version under testing and unstable is 1:4.2-3.1. This is different than the stable release version of 1:4.2-3+deb8u1. The deb8u1 denotes that this package is a debian stable release. This notation is applied by the maintainer.

Also on the far right there is a column labeled 'Watch'.

If you hover over the version number displayed here, it will tell you if the version that is currently being 'watched' either on the author's site or where ever it was originally taken from, is up to date with the current debian one that's on our server. Or vice versa. So if we uploaded version 1.2.3 yesterday and it's in Test. And suddenly the upstream author makes a change and updates to 1.2.4 this will say that there is a newer version available. Likewise, the debian crew make a change to the debian release, it'll say that our version is newer than what is being tracked.

Uploaders: The fine folk who are uploading the files, and their contact information, and most importantly, all other projects they've uploaded. Both Debian and non-affiliated Debian contributors are listed here.

Arch: Which CPU architectures will be supported by the package.

std-ver: Standards Version. The version of the Debian standards compliance that it adheres to.

VCS: The version control system. This is the software used to keep track of incremental software uploads, change logs, and management. Most popular systems to be Git.

ACTION NEEDED:

This menu shows a bunch of recent actions that require some level of attention. If you click the down arrow, you get a tad more detail about summary. Look below:

So notice two things. First is on the right hand side we have a scale of importance for each alert. The scale looks to be "High", "Normal", "Low", and "Wishlist".

Second, if you click the down are to the left of each action summary, a drop down appears like the one above for "Does not build reproducibly during testing". The more detailed info appears with a few nifty pieces. It explains that 'building reproducibily' is the act of the Debian maintainers, and or 3rd party users of the shadow package, to build the source but received unexpected results or failed to build successfully. These more detailed summaries come in handy to cross reference versions.

Another example is below:

Look above, this is the Action Needed menu, when it has been detected that the upstream author has posted a new release but no action has been taken yet by the maintainer of this package. (hey wake up!)

BUGS:

This section, to the right details the total number of bugs found in the current release of the package. The bugs are broken up into five categories. All, RC (release critical), I&N (Important and Normal), M&W (Minor and Wishlist) and F&P (Fixed and Pending). With release critical, these are show stoppers. These need attention ASAP. I&N, are the everyday bugs we find that cause the program to do or not do something, but overall functionality will remain generally stable. M&W are small cosmetic tweaks, things that should be fixed if you get around to it. Lastly Fixed & Pending are just as the name sounds, bugs that have been squashed or are in the process of getting patches applied.
Clicking the number to the right will launch a page that lists the bugs and their description, plus it links you to the comments about the bug. Some involve some very interesting and insight email chains between the debian maintainers and the upstream developers.

VERSIONS:

This section shows which package version is 'attached' to. For example :

The above shows the three familiar branches of Debian releases, Stable, Testing and Unstable. It then shows that the software package your looking at assigned to each. So package 1.4.0 -2 ( 1.4.0 is the upsteam developer version. and the -2 means it's the 2nd debian packaging. Notice the version numbers. Stable has a lower version number than testing and unstable. And think about it, it makes sense. Once a package is set to stable, it shouldn't change that often. Testing and Unstable on the other hand since they are in more of a stage of flux will have higher (often much higher) version numbers.
Clicking on each of the versions give you a great amount of detail telling you what binaries and libraries are included in the package. This is a great way to compare any additions or subtractions of binaries and files between version updates.

NEWS

Okay, so this is my favorite section on all of tracker. Check out the screenshot below:

In the center of the page is a list of news updates from from package that you're looking up. The list is show in reverse chronological order, so the newest items are on top. Generally what you'll be seeing here, is the constant cycle of acceptance, and migration of packages to and from different debian branches. For example above, you can see, that on June 27th 2015, version 1.14.2-1 of LightDM was accepted into Unstable by the maintainer Perez. Note the -1, which denotes that this is the first debian compiled version of 1.14.2.
Then, July 3rd, a few days later, LightDM package 1.14.2-1 moved to debian testing by the maintainer Britney. It's looking good so far for 1.14.2-1!

Then on August 25th, a new version 1.14.2-2 was uploaded into Unstable (goodbye 1.14.2-1) Later still, 1.14.2-2 would be put into Testing as well. Just notice that the -1, incremented to -2, meaning the change took place at the debian maintainer's level.
By the way, on the top left of this section is the word NEWS. This is actually a link to the full history of the updates. I think whoever is responsible for this site should make it a bit more apparently that you can click the NEWS link and it's not just plain text like the other headers on the page.

Clicking on the news links won't garner you a ton of info, a lot of it is just the PGP hashes used by the maintainer to "certify" that the package successfully installed, and it gives you the hash to compare it with if you download it. It also gives you a list of the binaries that that package has inside.

Next Time:

Okay so I'm stopping here for today! Covered a good amount of real estate on site. I hope this helps and makes the software management and life a bit more real. Even writing this, I'm learning as I'm going, so it's definitely mutual.

Next update will include:

The rest of the tracker main page
The Debian Software Teams at https://tracker.debian.org/teams/

See ya!

#1 2016-01-27 03:06:30