How To - Kernel Compile With Custom Options

verndog · 2018-05-27 21:41:26

Head_on_a_Stick wrote:

...
sleekmason wrote:
Newer the kernel, better the security.
^ I would strongly disagree with this assertion: the kernel developers seem hell-bent on adding as many features as possible as quickly as possible and all of these added features have the possibility of adding new vulnerabilities.
The Debian Security Team keeps track of any kernel vulnerabilities and backports the fixes very quickly indeed — with your suggested method the user would have to keep track of vulnerabilities themselves and manually recompile the new version (with all the extra, added, undiscovered vulnerabilities).

...

I was wondering about this issue regarding security. Thanks @Head_on_a_Stick, you have answered a question I was going to ask elsewhere.

sleekmason · 2018-05-28 00:28:30

The security patches that Debian uses come from the upstream kernel source. Not from Debian.
Debian may submit bug reports, etc., but kernel security comes with each new Linux Kernel version posted. The Debian security team patches are the same security issues that have already been addressed in the upstream kernel. Not the other way around.

For instance, here is linus Torvalds discussing how the latest patch addresses the new spectre issues.
https://lwn.net/Articles/755739/
You can be sure the fix will get to Debian within a day or two, and incorporated into whatever version they are currently working on.

Debian uses the Linux Kernel just like every other distribution, and keeps the same version with just security updates from upstream as they work out kinks and bugs (system calls, and such).

This link has people discussing the merits of Debian vs other distros as far as security. You will see that they generally have no opinion because security is not distro specific, but kernel specific.
The hype about Debian's security is just that. They are quick to implement the patches that are submitted upstream. That's all.
https://www.infoworld.com/article/31188 … urity.html

The latest release IS the most secure.
If you want to know more about security issues affecting the Linux kernel:
https://lwn.net/Alerts/

To find out what's going on with the kernel:
https://lwn.net/Kernel/

As a note:

the kernel developers seem hell-bent on adding as many features as possible as quickly as possible and all of these added features have the possibility of adding new vulnerabilities.

This is certainly true of arm, where kernel developers try to implement as many items as possible. Arm is much more complex.
Here is a link to a kernel how-to on xda developers I wrote for the gpad. I also maintained two separate versions for different use. The steps involved are nuts.
https://forum.xda-developers.com/showth … ?t=2628951

In the kernels linked you will find ton's of governors, IO schedulers, and other features that I brought to the gpad, and some I was unable to implement:)

In Linux, This just simply is not the case. Everything comes down from above. Very few people are trying to release kernel versions for linux in general, and the one's that do, have their own issues to keep on top of. It's just too hard to incorporate everything at once.

I hope this answers your question a little more thoroughly, and may give a better sense of the security measures in the Linux kernel. Regards,

Head_on_a_Stick · 2018-05-28 00:39:09

sleekmason wrote:

The hype about Debian's security is just that. They are quick to implement the patches that are submitted upstream. That's all.

I think you are unfairly diminishing the hard work of the Security Team.

And my point remains: if the user is not relying on Debian to provide timely security updates then what is your proposed method for said user to track new vulnerabilities and ensure that their kernel is kept sufficiently up to date?

DeepDayze · 2018-05-28 00:48:04

I like your post Sleek but keep in mind that the BL team has based their work on Stable and adding Testing/Sid repos indeed do pose a danger and note that backporting kernels from those other repos can end up requiring dependencies that are not available in Stretch, thus the risk that Testing/Sid packages can then get pulled in thus risking breakage of stable.

If you wanted to, you CAN upgrade Helium to Sid and it has been smooth in my case and you can then use the latest kernels.

DeepDayze · 2018-05-28 00:49:22

A case in point: Kernel 4.17 depends on a newer version of libc6 than what is available in Stretch. To backport it may require patching to make it work with the libc6 in Stretch, unless I am wrong.

Last edited by DeepDayze (2018-05-28 00:50:22)

sleekmason · 2018-05-28 01:14:46

I think you are unfairly diminishing the hard work of the Security Team.

Absolutely did not mean that the way it looked. They are indeed on the ball with everything they do. No disrespect intended to the Debian security team. I was writing specifically of their rapid response to incorporating patches, Not the brilliantly hard work they do to make it all happen.

If you wanted to, you CAN upgrade Helium to Sid and it has been smooth in my case and you can then use the latest kernels.

This isn't a bad idea at all. I have no problem throwing the option in as a preferred method, with what I did as a "at your own risk scenario". I haven't had any issues after an upgrade, but fully concede of the possibility.

sleekmason · 2018-05-28 03:04:37

And my point remains: if the user is not relying on Debian to provide timely security updates then what is your proposed method for said user to track new vulnerabilities and ensure that their kernel is kept sufficiently up to date?

People compiling their own kernel are be doing so to keep up with the latest implementations. Recompiling with the latest sources insures the strongest security.

The latest kernel version 4.17rc7 should already has implemented the most up to date security patches, with everybody else implementing the patches as fast as able to the older kernel versions the distro maintains, while working on drivers and implementing new features to the user base.
*Edit - I should note that the Linux Kernel Archives also support older versions fully patched. wanna run 4.14.44? no problem.

While some items are removed as they become deprecated, others are added that may even be more beneficial even to older hardware. intel turbo boost comes to mind.

Reading some of the entries here should :give you the idea.
https://www.linux.com/news/2017/7/linux … r-forecast

This site explains the different active kernel releases, and the definition of distro kernels:
https://www.kernel.org/category/releases.html

Many Linux distributions provide their own "longterm maintenance" kernels that may or may not be based on those maintained by kernel developers. These kernel releases are not hosted at kernel.org and kernel developers can provide no support for them.
It is easy to tell if you are running a distribution kernel. Unless you downloaded, compiled and installed your own version of kernel from kernel.org, you are running a distribution kernel. To find out the version of your kernel, run uname -r:
# uname -r
3.7.5-201.fc18.x86_64
If you see anything at all after the dash, you are running a distribution kernel. Please use the support channels offered by your distribution vendor to obtain kernel support.

Last edited by sleekmason (2018-05-28 03:10:26)

Head_on_a_Stick · 2018-05-28 09:07:06

First of all, thanks for changing the OP and adding the warning but if I may draw your attention to these bits:

sleekmason wrote:

When Buster becomes stable, you may safely add the second three dependencies and proceed with kernel image 4.16 and above.
[...]
Add the following:
deb http://ftp.us.debian.org/debian/ stretch main contrib non-free
#deb http://ftp.us.debian.org/debian/ testing main contrib non-free

When buster becomes stable then the testing entry will point to bullseye so it's probably best to change that.

Head_on_a_Stick · 2018-05-28 09:12:38

sleekmason wrote:

People compiling their own kernel are be doing so to keep up with the latest implementations. Recompiling with the latest sources insures the strongest security.
The latest kernel version 4.17rc7 should already has implemented the most up to date security patches

That's all very well but what about next week when a CVE for the kernel is announced and patches submitted upstream?

How does the user learn of these vulnerabilities?

Are you suggesting that perhaps they recompile every time a new version is released?

That looks like a lot of work for very little gain, especially given that Liquorix can supply a pre-compiled BFQ-enabled kernel with all the latest vulnerabilities features.

And I still think RC kernels are a bad idea but we will have to agree to disagree about that.

sleekmason wrote:

Many Linux distributions provide their own "longterm maintenance" kernels that may or may not be based on those maintained by kernel developers. These kernel releases are not hosted at kernel.org and kernel developers can provide no support for them.

With Debian it is the other way around: the upstream kernel.org 3.2, 3.16 & 4.9 LTS branches are all maintained by Debian.

sleekmason · 2018-05-28 12:25:15

sleekmason wrote:
People compiling their own kernel are be doing so to keep up with the latest implementations. Recompiling with the latest sources insures the strongest security.
The latest kernel version 4.17rc7 should already has implemented the most up to date security patches
That's all very well but what about next week when a CVE for the kernel is announced and patches submitted upstream?

What ifs.
Beat's me. At some point I stop having answers.
Because this is not a mandatory item, only the interested will do this, and will compile probably with each new release. Ultimately I have no idea what others will do with their systems. Up to them.

sleekmason · 2018-05-28 12:35:59

And I still think RC kernels are a bad idea but we will have to agree to disagree about that.

We may not disagree as much as you think:) When I wrote the guide, I took into account that folks always want the latest and greatest. I recommend the latest stable, with updates at least once per month, which is far more than most distros do. The only reason this rc is important, is because of the major removal of over 200,000 lines of code. They have only done this three times since it's inception. Because of the size difference and other optimizations, there is a palpable difference towards speed and efficiency.

I have amended the post towards focusing on a stable build first. Hopefully people will see the prudence in that. And this is indeed what I recommend.

sleekmason · 2018-05-29 12:08:01

First of all, thanks for changing the OP and adding the warning but if I may draw your attention to these bits:

Noted, Thank you.
Guide updated with added information and links.

sleekmason · 2018-05-31 17:03:22

And of course because karma has a sense of humor, upon a reinstall I followed my own guide successfully without enabling the testing repo. Only stable needed. Sorry for the confusion. Note taking error. No excuse.

Guide updated.

sleekmason · 2018-06-02 12:00:49

The 4.17 Linux Kernel Release goes stable in the next day or two. (generally on a Sunday). The 4.17 release is a major release among major releases, with tons of code removed and other new features.

Per Phoronix,

- A huge DRM subsystem update with AMDGPU DC by default for all capable GPUs, Intel Cannonlake graphics are stable, AMD WattMan, Intel HDCP, and more.

- Initial NVIDIA Tegra "Xavier" SoC support for this new high-performance ARM chip with Volta graphics.

- Eight obsolete CPU architectures were removed resulting in a code savings of about a half million lines.

- POWER4 CPU support is being dropped too as part of a separate pull. The POWER4 support was already broken since 2016 with no one apparently noticing until now.

- IBM s390 is still working on its Spectre defense.

- Continued maturing of the RISC-V architecture code.

- A new CPU architecture port for Linux 4.17 is the Andes NDS32 architecture.

- The Linux Kernel Memory Consistency Model has been formalized for Linux 4.17.

- Fixes for the Macintosh PowerBook 100 series.. Yes, with the Motorola processors from the early 90's.

- The new ACPI TAD driver for some interesting wake-up/alarm functionality as well as other CPUFreq and power management updates.

- PhoenixRC flight controller support.

- Multi-touch support for the Razer Blade Stealth.

- Thunderbolt USB/SL4 security level support.

- USB Type-C support improvements.

- Lost and Found support for F2FS along with performance enhancements and other work.

- EXT4 gets protection for maliciously crafted container images.

- Lazy time support for XFS.

- Btrfs gets a no SSD spread option and other improvements.

- New sound drivers plus USB Audio Class 3.0 support.

- Linux 4.17 staging has shed some weight (lots of lines of code).

- Various other PCI, crypto, and more updates as well as SPARC and BMC updates.

And for the 4.18RC Pre-Patch Kernel Release there should be Schedutil CPU Frequency Scaling Governor Improvements for battery life. Waiting for benchmarks.

Be aware - Pre-patch or "RC" kernels are mainline kernel pre-releases that are mostly aimed at other kernel developers and Linux enthusiasts. They must be compiled from source and usually contain new features that must be tested before they can be put into a stable release. Prepatch kernels are maintained and released by Linus Torvalds.

Also, While these are the main releases, there are intermediary releases containing updates and patches applied on a regular basis. These are generally released once a week.

I will update here at each major release with info on what's to be expected. (every 2-3 months, or whenever it happens:) I will also update certain news items kernel related that seem/are relevant.

*NOTE For Those that are new and wander in here - Compiling your own kernel is not a requirement. The Debian kernel image that comes with your system is regulary updated with patches and other bug fixes by the Debian kernel maintainers as warranted. Nothing else is needed.
If however, you would like to find out what's under the hood, check out the guide.

sleekmason · 2018-06-07 22:51:29

Interesting tidbit I just found on running kernel size.

To find the size of your currently running kernel, use uname -r to get the version number, then use the version number to find out running kernel size. I'm using kernel version 4.17.01:

uname -r

Output: ---> 4.17.01

ls -lha /boot | grep 4.17.01

Output:

-rw-r--r--  1 root root 103K Jun  7 13:02 config-4.17.01
-rw-r--r--  1 root root 6.4M Jun  7 16:51 initrd.img-4.17.01
-rw-r--r--  1 root root 1.8M Jun  7 13:02 System.map-4.17.01
-rw-r--r--  1 root root 3.6M Jun  7 13:02 vmlinuz-4.17.01

Last edited by sleekmason (2018-06-07 22:55:20)

stevep · 2018-06-08 20:56:08

Interesting...I'm backporting the 4.17.0 kernel right now, using a modified ~rc7 debian folder from the version that was in Experimental briefly--got it from snapshot.debian.org. But the only changes I made were:

* Adapt for gcc-6.
* Disable features/all/drivers-media-dvb-usb-af9005-request_firmware.patch
* Use 1000 Hz kernel timer frequency instead of 250.
* Use debian/lib folder from 4.15 kernels for python-gencontrol.
* debian/rules.real: stop the gigantic -dbg packages from building.

So what would you consider the most important other config changes I could do? I could also have the OBS build and host the packages like I did the 4.16.12 kernel: https://build.opensuse.org/package/show … linux-4.16

Doesn't look like any build platform will ever become available for arm64 with the specs requested in the contraints file, though.

Note that the Debian kernels take a lot of disk space when building--over 30 GB for the i386 build, I believe, and close to 20 for the amd64.

sleekmason · 2018-06-08 22:30:31

I'm backporting the 4.17.0 kernel right now, using a modified ~rc7 debian folder from the version that was in Experimental briefly.

Getting the kernel source directly from the Linux Kernel Archives, and using existing packages from the stretch repo works well here. gcc-6 is in stretch by default. That being said, sounds like you have your work cut out for you:)

So what would you consider the most important other config changes I could do?

Bunches:) Look in the guide at the "ideas" section for a list of some of the changes I have made that were beneficial.
These are relative to personal use, and I would subtract a few items if I was compiling for a distro.

Some, like changing to slub from slab, and removing the huge number of supported cpu's in the general section are all noted there. Others are not, but may be in the future after going through them again.

Another is removing file systems you won't be using. The more connections to unneeded drivers and options, the slower the reads.

If you plan on compiling frequently, or do other load intensive work, switching to the BFQ scheduler may make sense as well, as continued and improved support keeps coming down the line.
Note, if you have a solid state drive, fiops seems to work best for me. Changing to BFQ is in the guide if desired.

In my opinion, the single most important factor for the home user, is size. Performing a "make localmodconfig" as shown in the guide can drastically reduce the size of your kernel image, but I do not recommend it as the goto target, as devices may not register without being plugged in.

That being said, just going through the device drivers and removing what you don't use will do volumes to speed up kernel actions. in particular, removing the staging drivers (experimental) and the huge number of networking options that are rarely used and seldom needed. getting rid of raid, character devices, parallel port drivers, virtio and virtualization drivers, android drivers, chrome drivers, etc . . . you get the point:)

Doesn't look like any build platform will ever become available for arm64 with the specs requested in the contraints file, though.

I'm not up to date on arm anymore. Developed a couple of kernels for the lg gpad on xda Developers, (stock with bunches of options, and Cyanogenmod). haven't messed with android kernel stuff in a few years.

Note that the Debian kernels take a lot of disk space when building--over 30 GB for the i386 build, I believe, and close to 20 for the amd64.

After taking the time to remove everything not related to your computer, this can be done in less than 5 GB. I started compiling for my BL setup in a 10 GB partition, operating system included:)
Also, It now takes about 15 minutes for compile on 2009 Intel dual core. I'll be interested if I ever get a newer computer as to how fast this will go!

stevep · 2018-06-09 00:16:13

Well, I finally just got the 4.17.0 kernel to build on my hexacore i7 8750H laptop, which takes about an hour--but the Debian kernels build several different kernels, including the realtime variant. I had to figure out how to disable that, because the rt patches for rc7 failed to apply to the final release, and caused the build to error out.

I leave all the extra drivers in there, because I share my builds in the MX repo. I do hear that the 4.17 kernel is supposed to save some power for recent Intel CPUs, too.

Edit: just uploaded the source files to the OBS repo, let,s see if it builds there.

https://build.opensuse.org/package/show … linux-4.17

Last edited by stevep (2018-06-09 00:32:57)

sleekmason · 2018-06-09 01:37:01

How cool! So you have been doing this for a while then:)

MX just went to the top of new distros to multi boot with. Love learning new things. Anti helped me with my first kernel compile back ten years or so. Thought it was the neatest thing ever. Still do!

I imagine you would have to leave most drivers in place for a distribution build. For android, it's enter the arch and done. No worries about peoples computers going south on them during an upgrade. (well, not much anyway).

Trying to find a balance between realtime use and desktop use has to be hard. They nicely give a 300Hz option that supposedly is a good medium, but I've seen a thread or two where folks had problems with it and had to revert back to 250.

For a distribution, about the only changes I might make, would be possibly switching to BFQ as a few other distributions have, but it really seems more of a hassle than the value of switching, which is in itself arguable. I'm kind of a proponent because of the hype, and I think it's neat, but have yet to see the huge difference others had stated.

I do think the schedutil governor will be the way to go in a few months, but that will probably be made obvious to all by then anyway, if it is the ideal governor the devs suggest:)

stevep · 2018-06-09 18:27:44

Yeah, I started doing backports for MEPIS years ago, and it's evolved into this now. We also build ports of the Liquorix kernel for the MX repo, and that's also good for low-latency audio work, so the realtime kernel kernel is not really missed.

I found that the ZFS, Nvidia 390.48, broadcom-sta, ndiswrapper, Virtualbox-5.2.12, and bbswitch DKMS packages from the MX, Debian, or MX test repos build successfully and work with this kernel. The virtualbox-guest-dkms package fails in a VBox guest, so it's not going to be so good in a guest. Booting the new Coffee Lake MSI laptop with this kernel eliminates the annoying ACPI errors I was seeing on bootup with earlier kernels, so it's good for something. It's hard to measure the idle power savings it's supposed to provide for Intel CPUs, though...guess I just have to trust that they are there.

#21 2018-05-27 21:41:26

Re: How To - Kernel Compile With Custom Options

#22 2018-05-28 00:28:30

Re: How To - Kernel Compile With Custom Options

#23 2018-05-28 00:39:09

Re: How To - Kernel Compile With Custom Options

#24 2018-05-28 00:48:04

Re: How To - Kernel Compile With Custom Options

#25 2018-05-28 00:49:22

Re: How To - Kernel Compile With Custom Options

#26 2018-05-28 01:14:46

Re: How To - Kernel Compile With Custom Options

#27 2018-05-28 03:04:37

Re: How To - Kernel Compile With Custom Options

#28 2018-05-28 09:07:06

Re: How To - Kernel Compile With Custom Options

#29 2018-05-28 09:12:38

Re: How To - Kernel Compile With Custom Options

#30 2018-05-28 12:25:15

Re: How To - Kernel Compile With Custom Options

#31 2018-05-28 12:35:59

Re: How To - Kernel Compile With Custom Options

#32 2018-05-29 12:08:01

Re: How To - Kernel Compile With Custom Options

#33 2018-05-31 17:03:22

Re: How To - Kernel Compile With Custom Options

#34 2018-06-02 12:00:49

Re: How To - Kernel Compile With Custom Options

#35 2018-06-07 22:51:29

Re: How To - Kernel Compile With Custom Options

#36 2018-06-08 20:56:08

Re: How To - Kernel Compile With Custom Options

#37 2018-06-08 22:30:31

Re: How To - Kernel Compile With Custom Options

#38 2018-06-09 00:16:13

Re: How To - Kernel Compile With Custom Options

#39 2018-06-09 01:37:01

Re: How To - Kernel Compile With Custom Options

#40 2018-06-09 18:27:44

Re: How To - Kernel Compile With Custom Options

Board footer