ISO building notes

johnraff · 2017-01-17 06:47:51

After playing with live-build for the first time...

I'm leaving a few notes here just for reference, based on my experimenting so far.

*) It's a very good idea to edit auto/config and set --mirror-bootstrap to a fast local mirror. When it was set to httpredir I had a couple of builds fail partway through because the remote server had closed the connection. It went much more smoothly with a local mirror (ie one located nearby, in Japan in my case). This does not affect the settings in apt's sources.list for the generated system - that comes from --mirror-binary.

*) I was thinking we should perhaps set --iso-publisher to BunsenLabs.

--iso-publisher NAME
sets the PUBLISHED field in the header of a resulting CD/DVD image. By default, this is set to 'Live Systems project; http:/live-systems.org/; debian-live@lists.debian.org'. Remember to change this to the appropriate values at latest when you distributing custom and unofficial images.

*) The system beep is not due to an unrecognized character but a "bell" character. It's not necessary to edit system files to remove it: as long as you've got a config/bootloaders/isolinux/menu.cfg file you can do it there.

*) Most of the configuration is in the auto/config options. Various other files/directories are created under config/ at the 'lb config' stage - most of them can be deleted before starting again, but some are necessary.

*) I tried adding a memtest86+ option to the boot menu - it seems to work OK.

*) I added 0500-fix-dpkg-new-files.hook.chroot to config/hooks to patch up the .dpkg-new config files issue.

*) Added config/includes.chroot/usr/local/bin/bunsen_preseed and called it from config/includes.installer/preseed.cfg like this:

d-i preseed/late_command string \
    in-target /usr/local/bin/bunsen_preseed ;\
    in-target rm -f /usr/local/bin/bunsen_preseed

This works, and will make tweaking the late_command stuff easier in the future.

*) You don't necessarily want to run --purge every time you do an 'lb clean'. It's safer that way, but you'll have to rebuild the cache every time, and sometimes you can save a lot of time if you don't --purge. Just as long as you're sure there's nothing in the cache that you want to change.

*) There's quite a lot of live-build config code out there in the form of git repositories. I've found it's much easier to look around these if you clone them locally first. First, the Debian live-build configs, along with live-build itself and various other stuff, here: https://alioth.debian.org/anonscm/git/debian-live/
Any of those .git urls can be cloned with, eg,

git clone https://alioth.debian.org/anonscm/git/debian-live/live-images.git

The live-images repo has all that's needed to build Debian live isos.

Then, there's Kali Linux. Raphael Herzog, the current live-build maintainer, is closely involved with this:

git clone git://git.kali.org/live-build-config.git

And Handy Linux, by Arpinux (remember him from #!?):

git clone http://gitorious.org/handylinux/handylinux.git

That's it for now, but more suggestions might follow...

Last edited by johnraff-admin (2022-07-03 03:46:37)

johnraff · 2017-01-19 06:30:39

Forgot to mention in the OP - I feel nervous running a huge bunch of shell scripts with sudo, without reading through each and every one, so at the moment I'm doing live-build iso building in a VM.

H_o_a_S, you clearly know more than I do about using linux containers and chroots, so if you have any advice about such alternatives to a VM then please post.

Head_on_a_Stick · 2017-01-19 07:38:41

johnraff wrote:

alternatives to a VM

The simplest to use is systemd-nspawn, I do all my packaging with that.

Guide here:

https://wiki.archlinux.org/index.php/Systemd-nspawn

That is all applicable to Debian although the systemd version in jessie is a little old.

I think you will need the systemd-container package from jessie-backports.

EDIT: actually, that means you would need the backported systemd as well, maybe not the best idea

Also, would pbuilder or sbuild work for ISO images?

Last edited by Head_on_a_Stick (2017-01-19 07:40:08)

johnraff · 2017-01-19 08:13:20

^Yes, I was wondering about pbuilder.

But at the moment it's so easy to fire up a virtual machine and do it there, completely isolated. The generated isos seem to work fine. I guess the main disadvantage is extra system resource use? As long as you've got enough RAM?

Head_on_a_Stick · 2017-01-19 20:00:03

johnraff wrote:

I guess the main disadvantage is extra system resource use? As long as you've got enough RAM?

Yes, I suppose.

There are also restrictions on performance in the virtualised environment, even if KVM is
employed.

Also, to clarify:

I wrote:

actually, that means you would need the backported systemd as well, maybe not the best idea

This statement is not correct, from my BL system:

TheLab: ~ $ apt-cache policy systemd{,-container}
systemd:
  Installed: 215-17+deb8u6
  Candidate: 215-17+deb8u6
  Version table:
     230-7~bpo8+2 0
        100 http://httpredir.debian.org/debian/ jessie-backports/main amd64 Packages
 *** 215-17+deb8u6 0
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
systemd-container:
  Installed: (none)
  Candidate: 230-7~bpo8+2
  Version table:
     230-7~bpo8+2 0
        100 http://httpredir.debian.org/debian/ jessie-backports/main amd64 Packages

`systemd-nspawn` works just fine

johnraff · 2017-01-20 05:29:22

Head_on_a_Stick wrote:

There are also restrictions on performance in the virtualised environment

Ah good point. I've got my VMs restricted in access to both host system RAM and CPU usage. Even so, building a BL iso goes in just over an hour. (In fact the main drag in these cases seems to be hard disk access.) I'll have to try systemd-nspawn and see how it compares.

Head_on_a_Stick · 2017-02-04 10:42:18

Just a quick note that the current installer may have some issues with eMMC drives (as seen on the latest generation of netbooks), see this thread for details:

https://forums.bunsenlabs.org/viewtopic.php?id=3295

As I have suggested to the OP in my last post, I do think that the more recent upstream installers may be able to cope with those configurations.

EDIT: haven't tested your image yet John, I will do that later today.

Last edited by Head_on_a_Stick (2017-02-04 10:43:08)

Head_on_a_Stick · 2017-02-04 16:41:23

The ISO image works fine for me with QEMU/KVM:

Everything seems to work, nice to have a fresh install in which `apt update && apt upgrade` does nothing at all...

And the memtest works!

Head_on_a_Stick · 2017-02-04 20:35:25

Just tried a bare metal install of the new ISO image on my Haswell UEFI laptop and it failed 15% of the way through the installation process.

I tried it a few times and it failed consistently at the same point.

The syslog suggests that the squashfs may be corrupted.

The QEMU installation went fine with the same image though so this may be my (very old) USB stick.

I will try again when I can find a spare, fresh USB stick.

damo · 2017-02-04 23:05:51

Working fine after installing in VBox (although I don't think I will try on baremetal Skylake/nvidia laptop 8o )

Last edited by damo (2017-02-04 23:06:07)

Head_on_a_Stick · 2017-02-04 23:27:18

damo wrote:

I don't think I will try on baremetal Skylake/nvidia laptop

Please do, I would love to troubleshoot that bad boy 8)

Apparently, Steve Pusser has his Skylake Optimus laptop working really well with jessie+backports...

On topic: I am posting this from a freshly installed system, my old USB stick was indeed the problem.

The installer freaked me out though — my last Debian installation on this machine was called "Helium" and the installer consistenly brought up the hostname prompt with that preselected

damo · 2017-02-04 23:43:21

Please do, I would love to troubleshoot that bad boy

I tried it a while ago and never got it working properly for gpu rendering with Blender. I eventually gave up trying after finding Mint worked OOTB (Ubuntu didn't btw).

If I try again then expect a lot of PM's from me!

Head_on_a_Stick · 2017-02-04 23:55:08

damo wrote:

If I try again then expect a lot of PM's from me!

Any time

Also, I figured out what the installer was doing:

DEBUG: Getting default hostname from rDNS lookup of autoconfigured address 192.168.1.64

The router gave the hostname for that IP address, very cunning 8)

Anyway, sorry for the drift!

johnraff · 2017-02-05 03:22:17

Head_on_a_Stick wrote:

Just a quick note that the current installer may have some issues with eMMC drives (as seen on the latest generation of netbooks), see this thread for details:
https://forums.bunsenlabs.org/viewtopic.php?id=3295
As I have suggested to the OP in my last post, I do think that the more recent upstream installers may be able to cope with those configurations.

Do you have a link to such information? I had a quick look at the changelog, but couldn't spot it.

Adding features from newer releases to the Deuterium builds is theoretically possible - backporting, patching... but might not be trivial to do.

Helium will use a much newer live-build and associated tools, so that of course will be a different story.

Head_on_a_Stick · 2017-02-05 11:07:08

johnraff wrote:

Do you have a link to such information? I had a quick look at the changelog, but couldn't spot it.

No, not at all, I just thought that it is likely that the newer version of the Debian jessie installer will have some bug fixes.

johnraff · 2017-02-06 05:43:32

^Not the Jessie installer. There is no "newer version" as far as I know. The Stretch packages are getting upgraded though.

Head_on_a_Stick · 2017-02-06 07:19:46

But the ISO image itself has been updated, yes?

Or am I mistaken there?

I thought that @BB had raised this before:

https://forums.bunsenlabs.org/viewtopic.php?id=2824

In the example above, the actual problem was due to a package *not* present in https://packages.debian.org/search?suit … -installer and so the changelog to which you have linked is not relevant.

EDIT: in respect of eMMC support, I would expect that the updated kernel package would supply the missing functionality.

Last edited by Head_on_a_Stick (2017-02-06 07:48:19)

hhh · 2017-02-06 23:09:57

We don't have new ISOs yet, all upgrades are through Jessie or our repo, afaik.

johnraff · 2017-02-07 04:16:13

Head_on_a_Stick wrote:

But the ISO image itself has been updated, yes?
Or am I mistaken there?

'fraid so. Making a new iso image incorporating patches or backports from Sid or Stretch would be a major undertaking, involving much trial-and-error, repeated testing and rebuilds - each taking over an hour - which neither hhh nor I have taken on. Personally, I think we're better to postpone all that till we start building Stretch/Helium isos and have all the new tools available without any hacking.

In the example above, the actual problem was due to a package *not* present in https://packages.debian.org/search?suit … -installer and so the changelog to which you have linked is not relevant.

What do you mean, a package not present in debian-installer? d-e is a package in itself. EDIT: not true, it's a conglomeration of many small udeb packages. The d-e changelog I linked shows nothing about the eMMC issue, but as you say, it might well be a kernel issue, nothing to do with d-e anyway.

EDIT:
Re-reading that BL bug report, the package at fault there was partman-ext3. We could probably fix that particular bug in our iso just with a line in the d-i preseed file, as BB suggested, or, somewhat more work, by rebuilding partman-ext3 as stevep suggested. I'll try BB's suggestion in today's build, but it will need some testing to see if the issue has really gone. (It still exists in Jessie isos btw.)

That would fix the installer partitioning hang issue, but not the recently posted eMMC problem. I found a Debian mailing list thread that looks like the same issue: https://lists.debian.org/debian-boot/20 … 00041.html It's very long but seems to reach no solution. Some other googling failed to find anything too encouraging. I think it might be somewhat over-optimistic to suggest that a fix for this in the BL isos was imminent...

Last edited by johnraff-admin (2022-07-03 03:56:00)

johnraff · 2017-02-07 04:18:00

Re Deuterium isos, I think we can have some - incorporating the same updated BL and Debian packages - pretty soon, but getting EFI or eMMC support in there would be a totally different story.

#1 2017-01-17 06:47:51

ISO building notes

#2 2017-01-19 06:30:39

Re: ISO building notes

#3 2017-01-19 07:38:41

Re: ISO building notes

#4 2017-01-19 08:13:20

Re: ISO building notes

#5 2017-01-19 20:00:03

Re: ISO building notes

#6 2017-01-20 05:29:22

Re: ISO building notes

#7 2017-02-04 10:42:18

Re: ISO building notes

#8 2017-02-04 16:41:23

Re: ISO building notes

#9 2017-02-04 20:35:25

Re: ISO building notes

#10 2017-02-04 23:05:51

Re: ISO building notes

#11 2017-02-04 23:27:18

Re: ISO building notes

#12 2017-02-04 23:43:21

Re: ISO building notes

#13 2017-02-04 23:55:08

Re: ISO building notes

#14 2017-02-05 03:22:17

Re: ISO building notes

#15 2017-02-05 11:07:08

Re: ISO building notes

#16 2017-02-06 05:43:32

Re: ISO building notes

#17 2017-02-06 07:19:46

Re: ISO building notes

#18 2017-02-06 23:09:57

Re: ISO building notes

#19 2017-02-07 04:16:13

Re: ISO building notes

#20 2017-02-07 04:18:00

Re: ISO building notes

Board footer