Grub EFI secure-boot boot issues with live-build on Buster

cog · 2019-11-30 22:58:06

johnraff wrote:

This desktop has an EFI boot, so I could test the iso here, though with some fear and trepidation for my working box...

Lol, I know what you mean. This is my main system as well but wanted to give the ISO the old college try after doing an rsync backup. So I got my install working and might have found the source of the problem. It doesn't look like it's "update-grub" anymore but instead "grub-install"

I used the live-session to chroot into the install and fix grub. I ran the following steps several times inside the chroot:

This resulted a bootable system

rm -r /boot/grub
rm -r /boot/efi/EFI
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
    --bootloader-id=debian --recheck --no-floppy
update-grub

This is what the live-build installer results as and grub can't find it's config or boot

rm -r /boot/grub
rm -r /boot/efi/EFI
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
    --bootloader-id=bunsenlabs --recheck --no-floppy
update-grub

I have no experience with live-build but if there's a way to make it use "debian" as the bootloader-id on UEFI installs it might just work. The "debian" bootloader-id isn't visable to the enduser unless they go into their BIOS as grub's menu is showing "BunsenLabs" on all the entries and the BunsenLabs grub theme.

As a side note, the live-build installer appears to be doing GPT correctly and all that good stuff on a UEFI based system.

Last edited by cog (2019-11-30 22:59:55)

cog · 2019-12-01 02:20:34

Another progress report... also got an encrypted root install working using the stuff I mentioned above.

Just a thought guys, if this stuff is not fixable in the live-build ISO, I could always write a post-install tutorial on how to chroot and fix the grub-efi bootloader stuff.

Everything else is working really good on this end. (writing this from the fixed install)

Last edited by cog (2019-12-01 02:30:05)

johnraff · 2019-12-01 06:06:16

@cog thanks- you've certainly put your finger on something that needs fixing there.
(I don't know if Damo's errors or Bronto's delay are related.)

Live-build uses the regular Debian Installer, but BL uses a custom package in that: live-installer_57.
The changes made there are pretty minimal, though, and should be unrelated to grub, or bootup.

If --bootloader-id=bunsenlabs is the problem I guess it's less surprising no bug has been reported, since it would be for derivatives like BL only, not straight Debian.

Anyway it worked OK before, so it remains to check what changed - Debian Installer or Live-Build - then look for the best place to put our tweak to get it working again. Since live-build is all text files (huge maze of shell scripts) it should be possible to grep around and track it down. Then either patch it to use id=debian or else get it to accept id=bunsenlabs. I think it should be doable.

EDIT
Possibly related: https://bugs.launchpad.net/ubuntu/+sour … ug/1450783

Last edited by johnraff (2019-12-01 08:23:43)

cog · 2019-12-02 04:56:01

John, that bug report looks real close. Might be it. It's kind of baffling that it would show up from stretch to buster.

here's another link..

This OP looks identical to what I experienced post install before repairing the EFI partition and regenerating grub.cfg.

https://askubuntu.com/questions/1129269 … g-myubuntu

johnraff · 2019-12-02 09:54:45

cog wrote:

This is what the live-build installer results as and grub can't find it's config or boot
rm -r /boot/grub
rm -r /boot/efi/EFI
grub-install --target=x86_64-efi --efi-directory=/boot/efi \
    --bootloader-id=bunsenlabs --recheck --no-floppy
update-grub

cog, where are you reading this output? On the target machine during installation, switching to the Alt+F4 console? I'm not getting very far yet - the string "bootloader-id" isn't anywhere in the debian installer as far as I can see...

cog · 2019-12-02 10:16:31

John,

Everything I've got is pretty much a binary shot in the dark with what the installer leaves you with after the fact since the BunsenLabs ISO live-build system source code isn't on the organization github repos as far as I can see.

There seems to be some custom configuration on the legacy end though like the gtk theme in the graphical installer and such. I was hoping a grep for an all lowercase "bunsenlabs" might find the culprit.

I don't have much else to go on ATM. hope this helps

cog · 2019-12-02 10:25:54

The installer ends with a /boot/efi/EFI/busnenlabs setup. The exact same grub cfg after installation works with a /boot/efi/EFI/debian directory and boots properly. It's strange. Maybe it's hardcoded into the EFI stub like the Ubuntu launchpad bugtracker mentioned.

EDIT: by Johnraff, pasted in content from cog's post of 2019-12-03 12:22:12:

I backported the latest VirtualBox 6.0.14 (it seems to UEFI VM's really well) to my lithium-dev rig and have been able to successfully recreate the bug and fix with the 3rd ISO we've got. I've made a series of screenshots and uploaded them to my server if yall want to check it out. I put them on there instead of a free image host to keep prying eyes out.

https://willscreel.us/lithium-uefi/

Edit: You'll notice this time I left the grub.cfg intact after the fresh install only running grub-install with a --boot-loader-id=debian and it worked, which hints that our grub.cfg is good.

Last edited by johnraff (2019-12-13 02:09:48)

cog · 2019-12-04 04:26:43

Short Version

Ok, so I figured out the bug or guess you could say what changed from Stretch to Buster..

Basically it's just the "grub-efi-amd64-signed" package. It didn't exist in helium/stretch but does now in lithium/buster. It's pulled in automatically by "grub-efi-amd64-bin" as a recommends which is a dependency of the "grub-efi-amd64" package. By default, GRUB is always installed for EFI secure boot if the package grub-efi-amd64-signed is installed. This always requires the ESP to be at /boot/efi/EFI/debian because it's hardcoded and signed by debian.

Long Version with Screenshots

1.)

Here you can see the conflict between $prefix & $cmdpath after a fresh lithium install in the grub console it drops you to.

All you really have to do to make grub find it's cfg is "configfile $cmdpath/grub.cfg" Then you're in like a dirty shirt.

2.)

Just showing grub now ready to boot into lithium.

3.)

We're in.

4.)

Here I just removed the installed ESP as well as grub-efi-amd64-signed package. Then re-ran grub-install simply without any parameters except the harddrive location as debian-installer would if grub-efi-amd64-signed wasn't being pulled into the installer. Just for refference I listed the contents of the ESP before and after each step. You can see now it looks just like Helium or even what Kali is doing now on a Buster base.

5.)

On reboot lithium finds and goes straight to the grub.cfg. After some examination from the grub console there are no conflicts anymore between $prefix & $cmdpath.

6.)

This is a screenshot of Helium from the grub console in UEFI mode. Lithium looks identical now.

7.)

A successful boot!

Possible Fixes

a.)
The most sustainable thing seems to be rolling our own d-i grub-installer package that does a no-reccommends install of grub-efi-amd64, grub-efi-amd64-bin, and efibootmgr that would then install just like Helium.

Edit:
After looking over this with fresh eyes this morning.. If mostly what our debian-installer does is copy the contents of the resulted live-build image, then probably the easiest thing would be to make sure "grub-efi-amd64-signed" doesn't get built into the image.

b.)
Another approach would be like what Kali is doing and just roll our own grub2 package that in turn wouldn't have "grub-efi-amd64-signed" as a recommended package for "grub-efi-amd64-bin". This seems not as sustainable and we wouldn't get updates from debian upstream. The grub2 source package builds a slew of packages if you look at the control file.

c.)
This would be sustainable as well but not sure if it's even possible. Roll a d-i grub-installer package that runs "--no-uefi-secure-boot" when it runs "grub-install". I don't think it's possible because grub-installer doesn't seem to run these commands, the packages it pulls in does. Then we'd be back to rolling our own grub2 package.

d.)
We could just write a tutorial on how to fix UEFI post install. These new steps in the screenshots don't require a live environment or any chroot magic, just a few commands while removing the culprit package.

Edit:

Yall can disregard my previous set of screenshots a couple posts back. They'll still be on my server if yall need to review them. It's now evident the --bootloader-id=debian thing was just making the install play nice with the signed secure boot stuff, as if we weren't using a debian derivative.

Just for reference if we ever need it, the grub-install command gets it's /boot/efi/EFI/target-dir or --bootloader-id from the following snippet it runs.

dpkg-vendor --query vendor | tr '[:upper:]' '[:lower:]'

That's why it's not grep'able in live-build or debian installer. It changes the case on the fly. The vendor info is something deeply rooted in BunsenLabs release info and repos.

Last edited by cog (2019-12-04 16:34:24)

cog · 2019-12-05 05:34:38

Ok boys, I got lucky, we got us a winner winner chicken dinner on the second test build. After running a UEFI install with the test ISO it generated everything booted up just fine.

The fix is creating "live-build-root/config/apt/preferences" with:

Package: grub-efi-amd64-signed
Pin: version *
Pin-Priority: -1

I ran this too which excludes grub-efi-amd64 from the live-session:

lb config --uefi-secure-boot disable

We'd probably wanna put the following in "live-build-root/auto/config" instead of running the above command every time before build time:

--uefi-secure-boot disable \

Edit:

Just to make sure, I probed grubs variables, & /boot/efi/EFI/bunsenlabs as well. All their values came out identical as Helium.

Just got back from a bare-metal UEFI install as well with the LVM encrypted root auto-partitioning option, booted up just fine. Posting from that install now.

I don't have the ISO anymore as I wiped it out during the bare metal install (It's dd'ed to my USB drive and not sharable anymore). Anyways, with the above snippets @John will be able to roll our official test ISOs that should install and boot on UEFI.

Last edited by cog (2019-12-05 07:51:01)

johnraff · 2019-12-06 03:31:02

cog this is excellent! Many thanks for getting to the bottom of it while I was goofing off in the mountains.

I suspected that the arrival of secure boot might make things more complicated for us, as a derivative.

I guess it's not possible to use debian's signed grub to boot our modified-debian system? Even though the Linux kernel itself is the same?

I'll try a bit of googling on "debian derivative secure boot"...

and:

cog wrote:

Just for reference if we ever need it, the grub-install command gets it's /boot/efi/EFI/target-dir or --bootloader-id from the following snippet it runs.
dpkg-vendor --query vendor | tr '[:upper:]' '[:lower:]'

Where did you find that code? In live-build all I could grep out was this reference in live-build/scripts/build/binary_grub-efi:

	# Secure Boot support:
	# - create the EFI directory in the ESP with uppercase letters to make
	#   certain firmwares (eg: TianoCore) happy
	# - use shim as the boot<arch>.efi that gets loaded first by the firmware
	# - drop a grub.cfg (same reason as below) in the cfg directory as configured
	#   by the signed grub efi binary creation. This is set dynamically when grub2 is
	#   built with the ouput of dpkg-vendor, and can be overridden by the builder, so
	#   we do the same here in live-build.
	# - the source paths are taken from shim-signed:
	#    https://packages.debian.org/sid/amd64/shim-signed/filelist
	#   and grub-efi-amd64-signed, currently in Ubuntu:
	#    https://packages.ubuntu.com/xenial/amd64/grub-efi-amd64-signed/filelist
	#    https://packages.ubuntu.com/bionic/arm64/grub-efi-arm64-signed/filelist
	#    E.g., gcdx64.efi.signed is the boot loader for removable device, like CD or
        #    USB flash drive, while grubx64.efi.signed is for hard drive.
        #    Therefore here gcdx64.efi.signed is used for amd64 and gcdaa64.efi.signed is 
	#    for arm64.

In the debian-installer tree the only greps for dpkg-vendor were in busybox, flash-kernel and debootstrap, mostly checking for Ubuntu etc.

Anyway, as long as we don't want secure boot then you've found us a dead easy solid-looking config fix for live-build. Massive thanks!

cog · 2019-12-06 07:31:10

johnraff wrote:

I suspected that the arrival of secure boot might make things more complicated for us, as a derivative.
I guess it's not possible to use debian's signed grub to boot our modified-debian system? Even though the Linux kernel itself is the same?

I don't think the live-build or debian-installer code base supports secure-boot on derivatives as of buster. Kali still doesn't support secure-boot out of the box and they're already on a bullseye base with live-build guru Raphael Hertzog overseeing their ISO build pipeline.

https://kali.training/topic/booting-kali-in-live-mode/

Our first tests also confirm this since we had to hoodwink the install into acting like it was vanilla debian post-install.

johnraff wrote:

Where did you find that code? In live-build all I could grep out was this reference in live-build/scripts/build/binary_grub-efi:

It's in the grub2 source package so that's why we couldn't grep it in live-build or debian-installer.

https://salsa.debian.org/grub-team/grub … efi-images

https://salsa.debian.org/grub-team/grub … bian/rules

johnraff wrote:

Anyway, as long as we don't want secure boot then you've found us a dead easy solid-looking config fix for live-build. Massive thanks!

Np bud. It's been really exciting working on this stuff and being able to contribute something. My vote is not to support secure boot simply because our previous release didn't and it's already the expectation of the enduser.

Last edited by cog (2019-12-06 07:46:46)

johnraff · 2019-12-06 14:19:38

cog wrote:

I don't think the live-build or debian-installer code base supports secure-boot on derivatives as of buster. Kali still doesn't support secure-boot out of the box and they're already on a bullseye base with live-build guru Raphael Hertzog overseeing their ISO build pipeline.

If RH, the live-build maintainer, hasn't yet enabled it for Kali then it's obviously not a trivial task...

johnraff wrote:
Where did you find that code? In live-build all I could grep out was this reference in live-build/scripts/build/binary_grub-efi:
It's in the grub2 source package so that's why we couldn't grep it in live-build or debian-installer.
https://salsa.debian.org/grub-team/grub … efi-images
https://salsa.debian.org/grub-team/grub … bian/rules

Thanks!

My vote is not to support secure boot simply because our previous release didn't and it's already the expectation of the enduser.

Yes, agreed, at least for our test iso and first official release. If we find how to sign our own bootloader we can always add SB support later - it would be cool to have, now that regular Debian supports it, and would save users some extra hassle.

---
The search can continue though, eg on a VM: https://wiki.debian.org/SecureBoot/VirtualMachine
Also: https://bugs.debian.org/cgi-bin/bugrepo … bug=821084

Last edited by johnraff (2019-12-06 14:30:35)

johnraff · 2019-12-06 14:26:15

cog wrote:

The fix is creating "live-build-root/config/apt/preferences" with:
Package: grub-efi-amd64-signed
Pin: version *
Pin-Priority: -1
We'd probably wanna put the following in "live-build-root/auto/config":
--uefi-secure-boot disable \

I'm wondering if the second config entry alone might be enough, without the apt pin? I'll have a poke around the live-build code and see what --uefi-secure-boot does...

---
Anyway, since the UEFI boot issue can be fixed in the l-b config, maybe we could share the patched test iso with the forum to get some community feedback before tackling any other details?

cog · 2019-12-07 01:37:31

johnraff wrote:

I'm wondering if the second config entry alone might be enough, without the apt pin? I'll have a poke around the live-build code and see what --uefi-secure-boot does...

John, just for reference, I had ran 3 test builds with --debug back before the successful vm and baremetal install.

#1 was with your current config just to see the build.log

#2 was only with --uefi-secure-boot disable, so the configs weren't touched. The resulting debian-installer installation still failed to boot from UEFI. I did notice from the build.log that grub-efi-amd64-signed wasn't pulled in at one point so I figured I'd still run it on #3.

#3 was with --uefi-secure-boot disable and the apt pin. This was after reading from the live-manual on how not to do install-recommends for one particular package. (It's the last paragraph on the page) The goal was to get d-i grub-installer to install the recommended efibootmgr package from grub-efi-amd64-bin package, but not the recommended grub-efi-amd64-signed package.

https://live-team.pages.debian.net/live … on.en.html

We might be able to get by without "--uefi-secure-boot disable" and let it default to "auto", but the pin seems to be what's making it work without having to maintain our own grub packages.

--

Just a side note from my past experiences with secure-boot, DKMS gets even squirly-er than it already is.

https://wiki.ubuntu.com/UEFI/SecureBoot/DKMS

Last edited by cog (2019-12-07 02:32:14)

johnraff · 2019-12-08 05:04:13

^Thanks cog for all you've done so far!

---
One thing I have to keep reminding myself is that there are two separate grub/secure-boot situations here:
1) Booting the iso.
2) Booting the installed system.

And it's sometimed tricky to remember which grub a certain L-B config or bit of L-B code is being applied to.
(In fact it's usually the iso, not the installer, that the manual or help are referring to, but of course we need to have both of them working.)

johnraff · 2019-12-08 06:09:12

cog wrote:

#3 was with --uefi-secure-boot disable and the apt pin. This was after reading from the live-manual on how not to do install-recommends for one particular package. (It's the last paragraph on the page) The goal was to get d-i grub-installer to install the recommended efibootmgr package from grub-efi-amd64-bin package, but not the recommended grub-efi-amd64-signed package.

...just to mention, we set --apt-recommends false so packages installed in the l-b chroot should not pull in any recommends anyway. It just happens that we already have efibootmgr on our install list, but I wonder if there might be another package or two that we ought to add to make the grub install work better?

---
OTOH this code in /live-build/scripts/build/binary_grub-efi seems to me to be saying that if --uefi-secure-boot hence ${LB_UEFI_SECURE_BOOT} is disable then shim-signed and grub-efi-amd64-signed will not be installed:

# Check UEFI Secure Boot setting and depends
# By default (auto) do a best-effort build: if the signed binaries are available use
# them, but don't fail if they are not, just print a warning.
case "${LB_ARCHITECTURES}" in
	amd64|i386)
		_SB_EFI_PLATFORM="x86_64"
		_SB_EFI_NAME="x64"
		_SB_EFI_DEB="amd64"
	;;
	arm64)
		_SB_EFI_PLATFORM="arm64"
		_SB_EFI_NAME="aa64"
		_SB_EFI_DEB="arm64"
	;;
esac

_PRE_SB_PACKAGES="${_LB_PACKAGES}"
_LB_PACKAGES="shim-signed grub-efi-${_SB_EFI_DEB}-signed"
case "${LB_UEFI_SECURE_BOOT}" in
	auto)
		# Use Check_installed, as Check_package will error out immediately
		set +e
		Install_package
		set -e
		Check_installed chroot /usr/lib/grub/${_SB_EFI_PLATFORM}-efi-signed/gcd${_SB_EFI_NAME}.efi.signed \
			grub-efi-${_SB_EFI_DEB}-signed
		_GRUB_INSTALL_STATUS="${INSTALL_STATUS}"
		Check_installed chroot /usr/lib/shim/shim${_SB_EFI_NAME}.efi.signed \
			shim-signed

		if [ "${INSTALL_STATUS}" -ne 0 -o "${_GRUB_INSTALL_STATUS}" -ne 0 ]
		then
			Echo_warning "UEFI Secure Boot disabled due to missing signed Grub/Shim."
		else
			Echo_message "UEFI Secure Boot support enabled."
		fi
	;;
	enable)
		Check_package chroot /usr/lib/grub/${_SB_EFI_PLATFORM}-efi-signed/gcd${_SB_EFI_NAME}.efi.signed \
			grub-efi-${_SB_EFI_DEB}-signed
		Check_package chroot /usr/lib/shim/shim${_SB_EFI_NAME}.efi.signed \
			shim-signed
		Install_package
		Echo_message "UEFI Secure Boot support enabled."
	;;
	disable)
		Echo_message "UEFI Secure Boot support disabled."
	;;
esac
_LB_PACKAGES="${_PRE_SB_PACKAGES}"

So I don't yet understand why your case #2 build failed to boot.

Was this perhaps a typo:

#2 was only with --uefi-secure-boot disable, so the configs weren't touched. The resulting debian-installer installation still failed to boot from UEFI. I did notice from the build.log that grub-efi-amd64-signed wasn't pulled in at one point so I figured I'd still run it on #3.

Meaning that grub-efi-amd64-signed was pulled in without the apt pin? Or if it wasn't pulled in, there was another reason why it didn't boot?

---
Anyway, what I'll do now is actually try some builds with different config combinations, as you did, and put them on Google Drive (or somewhere else, if preferred) for people to try out. And also look at the build logs, having built with --debug and --verbose set.

Last edited by johnraff-admin (2022-07-03 04:39:28)

johnraff · 2019-12-08 06:19:19

Interesting for future reference, discussion on the live-build git repo:
https://salsa.debian.org/live-team/live … requests/3

also
https://tails.boum.org/blueprint/UEFI_Secure_boot/
https://mjg59.dreamwidth.org/17542.html
https://www.youtube.com/watch?v=V2aq5M3Q76U

Last edited by johnraff (2019-12-08 07:50:52)

cog · 2019-12-08 17:49:23

For clarification, all my tests were for mainly was the Debian-installer.

The live-build/live-session worked/booted fine on all 3 tests.

So I think it goes back to what you were saying there are two situations here.

1) —-uefi-secure-boot disable & —-apt-recommends false affects the live-build/live-session

2) the pin creates and apt policy that Debian installer respects when its installing grub packages

Last edited by cog (2019-12-08 17:53:00)

johnraff · 2019-12-13 02:19:06

Pasting in some links so they'll be easy to find later.

GNU GRUB Manual 2.04
https://www.gnu.org/software/grub/manual/grub/grub.html

Ubuntu on Grub troubleshooting
https://help.ubuntu.com/community/Grub2/Troubleshooting

Two Debian bug reports with some very useful info mixed in:

live-build: UEFI shows grub> on HP250 G6 2SX60EA
https://bugs.debian.org/cgi-bin/bugrepo … bug=924053

live-build: support syslinux-efi as (additional) bootloader
https://bugs.debian.org/cgi-bin/bugrepo … bug=922251

johnraff · 2019-12-14 09:07:58

Still researching, but just had a thought for a possible Easy Fix to enable S-B.
Our Vendor branding as BunsenLabs comes from bunsen-os-release which installs files in /etc/dpkg/origins that are looked at by dpkg-vendor.

We could just leave bunsen-os-release off the package list, and then use a d-i preseed late_command to install it at the end of the debian-installer process. Late commands are run after grub install, so grub will see the system as plain Debian (which it is more-or-less) and S-B might be enabled. Late_command might also have to rerun update-grub to get the right name in the grub menu though. Haven't tried this yet...

EDIT: Just confirmed that installing bunsen-os-release and running update-grub is enough to change the grub menu entry.

Last edited by johnraff (2019-12-14 09:16:39)

#1 2019-11-30 22:58:06

Grub EFI secure-boot boot issues with live-build on Buster

#2 2019-12-01 02:20:34

Re: Grub EFI secure-boot boot issues with live-build on Buster

#3 2019-12-01 06:06:16

Re: Grub EFI secure-boot boot issues with live-build on Buster

#4 2019-12-02 04:56:01

Re: Grub EFI secure-boot boot issues with live-build on Buster

#5 2019-12-02 09:54:45

Re: Grub EFI secure-boot boot issues with live-build on Buster

#6 2019-12-02 10:16:31

Re: Grub EFI secure-boot boot issues with live-build on Buster

#7 2019-12-02 10:25:54

Re: Grub EFI secure-boot boot issues with live-build on Buster

#8 2019-12-04 04:26:43

Re: Grub EFI secure-boot boot issues with live-build on Buster

#9 2019-12-05 05:34:38

Re: Grub EFI secure-boot boot issues with live-build on Buster

#10 2019-12-06 03:31:02

Re: Grub EFI secure-boot boot issues with live-build on Buster

#11 2019-12-06 07:31:10

Re: Grub EFI secure-boot boot issues with live-build on Buster

#12 2019-12-06 14:19:38

Re: Grub EFI secure-boot boot issues with live-build on Buster

#13 2019-12-06 14:26:15

Re: Grub EFI secure-boot boot issues with live-build on Buster

#14 2019-12-07 01:37:31

Re: Grub EFI secure-boot boot issues with live-build on Buster

#15 2019-12-08 05:04:13

Re: Grub EFI secure-boot boot issues with live-build on Buster

#16 2019-12-08 06:09:12

Re: Grub EFI secure-boot boot issues with live-build on Buster

#17 2019-12-08 06:19:19

Re: Grub EFI secure-boot boot issues with live-build on Buster

#18 2019-12-08 17:49:23

Re: Grub EFI secure-boot boot issues with live-build on Buster

#19 2019-12-13 02:19:06

Re: Grub EFI secure-boot boot issues with live-build on Buster

#20 2019-12-14 09:07:58

Re: Grub EFI secure-boot boot issues with live-build on Buster

Board footer