Skip to content

Linux: Break out the full builds into a smaller base + then per-feature initrds

Martin Roukala requested to merge kernel_rework into main

The amd64 linux kernel size has ballooned beyond control, mostly thanks to nvidia's GSP but also due to Intel's microcode. This is annoying because not all users need this, so it is time for a big rework of our artifacts!

By merging the firmware and modules initrd then splitting it by feature, we can enable users to opt-in the features they want by simply loading more initrds besides b2c:

  • Intel microcode
  • GPU drivers
  • WiFi drivers

The depmod initrd is needed if you want to be able to modprobe your modules, and rather than duplicating its content in every feature initrd I've kept it separate which also allowed me to compress the whole archive.

An additional change this MR introduces is the compression of modules and firmware files as it brings improved boot time, lower boot time RAM usage, and enables efficient runtime filtering of the list of modules/firmware files sent to a test machine without first having to decompress the whole initrd then having to send an uncompressed initrd (a feature found in CI-Tron). This makes it possible to have a single initrd to support every GPU vendor without having to send the entirety of the initrd to every test machine. The choice of xz for firmware and modules is contrary to what modern distros are doing, but XZ still reigns supreme when it comes to compression ratio (for netbooting speed and lower RAM usage) while only a handful of modules/firmware files will end up being loaded/decompressed which makes decompression time irrelevant.

Before:

-rw-r--r-- 1 mupuf mupuf 35992576 Dec 19 07:10 linux-x86_64
-rw-r--r-- 1 mupuf mupuf 58253824 Dec 19 07:10 linux-x86_64.firmware.cpio.xz
-rw-r--r-- 1 mupuf mupuf  5088768 Dec 19 07:10 linux-x86_64.modules.cpio.xz
Total: 99335168 bytes

$ time make manual_test B2C_RUN=""
real    0m6.307s
user    0m5.552s
sys     0m0.713s

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            821          61         431         320         328         428

After individual compression of firmware files:

-rw-r--r-- 1 mupuf mupuf 49443840 Dec 19 07:02 linux-x86_64
-rw-r--r-- 1 mupuf mupuf 60612608 Dec 18 16:08 linux-x86_64.firmware.cpio
-rw-r--r-- 1 mupuf mupuf  5084160 Dec 18 16:08 linux-x86_64.modules.cpio.xz
Total: 115140608 bytes (+16%)

$ time make manual_test B2C_RUN=""
real    0m5.585s (-12%)
user    0m4.898s
sys     0m0.691s

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            898          94         475         320         329         471  (+43M)

After individual compression of modules:

-rw-r--r-- 1 mupuf mupuf 49443840 Dec 19 15:59 linux-x86_64
-rw-r--r-- 1 mupuf mupuf 60612608 Dec 18 16:08 linux-x86_64.firmware.cpio
-rw-r--r-- 1 mupuf mupuf  6145536 Dec 18 16:09 linux-x86_64.modules.cpio
Total: 116201984 bytes (+0.9%)

$ time make manual_test B2C_RUN=""
real    0m5.383s (-3.6%)
user    0m4.673s
sys     0m0.665s

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            898          59         537         293         302         533 (+62M)

After splitting into per-feature initrds, and adding more gpu support bringing in ~10.5MiB worth of firmware from Intel which explains 100% of the increased size:

-rw-r--r-- 1 mupuf mupuf 13042688 Dec 19 06:57 linux-x86_64
-rw-r--r-- 1 mupuf mupuf   139264 Dec 19 06:57 linux-x86_64.depmod.cpio.xz
-rw-r--r-- 1 mupuf mupuf 74607616 Dec 19 06:57 linux-x86_64.gpu.cpio
-rw-r--r-- 1 mupuf mupuf 13017088 Dec 19 06:57 linux-x86_64.ucode.cpio
-rw-r--r-- 1 mupuf mupuf 25482752 Dec 19 06:57 linux-x86_64.wifi.cpio
Total: 126289408 bytes (+8.6%)

$ time make manual_test B2C_RUN=""
real    0m3.315s (-38.4%)
user    0m2.642s
sys     0m0.645s

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            958         121         550         278         287         546 (+13M)

Overall:

Total size: 99335168 -> 126289408 bytes (+27%)
Boot time: 0m6.307s -> 0m3.315s (-47%)
Available RAM: 428 -> 546 (+118M)
Edited by Martin Roukala

Merge request reports

Loading