Bootloaders

Warewulf uses iPXE as its default network bootloader. As a tech preview, support for GRUB is also available, which adds support for secure boot.

Also as a tech preview, Warewulf may also use iPXE or GRUB to boot a dracut initramfs as an initial stage before loading the image. This is called a two-stage boot.

Booting with iPXE

The /etc/warewulf/ipxe/ directory contains text/templates that are used by the Warewulf configuration process to configure the ipxe service.

$digraph G{ node [shape=box]; compound=true; edge [label2node=true] bios [shape=record label="{BIOS | boots from DHCP/next-server via TFTP}"] subgraph cluster0 { label="iPXE boot" iPXE; ipxe_cfg [shape=record label="{ipxe.cfg|generated for each node}"]; iPXE -> ipxe_cfg [label="http"]; } bios->iPXE [lhead=cluster0,label="iPXE.efi"]; kernel [shape=record label="{kernel|ramdisk (root fs)|wwinit overlay}|extracted from OS image"]; ipxe_cfg->kernel[ltail=cluster0,label="http"]; }$

Starting in v4.5.0, Warewulf no longer includes an iPXE binary. Instead, by default Warewulf uses the iPXE that comes with the host OS.

Unfortunately, we’ve encountered a few instances where bugs in the OS-provided iPXE that sometimes make booting a full OS image as an “initrd” unreliable.

Building iPXE locally, using a more recent “version” of the iPXE source code, can alleviate some of these issues.

Another alternative is Two-stage boot: dracut, which uses the Linux kernel to load the full OS image, avoiding the issue entirely.

Building iPXE locally

By default (as of v4.5.0) Warewulf packages use iPXE from the host operating system rather than bundling iPXE binaries with Warewulf. However, sometimes the specific build included in the host OS has bugs or missing features, and a local build of iPXE is necessary.

The Warewulf project provides a build-ipxe.sh script to simplify the process of building iPXE locally.

# curl -LO https://raw.githubusercontent.com/warewulf/warewulf/main/scripts/build-ipxe.sh
# bash build-ipxe.sh -h
Usage: build-ipxe.sh
         [-h] (help)
TARGETS: bin-x86_64-pcbios/undionly.kpxe bin-x86_64-efi/snponly.efi bin-arm64-efi/snponly.efi
IPXE_BRANCH: master
DESTDIR: /usr/local/share/ipxe

Running build-ipxe.sh

The script, by default, builds iPXE for x86_64 BIOS, x86_64 EFI, and arm64 EFI from the master branch on the iPXE project GitHub and stores the resultant builds in /usr/local/share/ipxe/. (These parameters can be adjusted by setting TARGETS, IPXE_BRANCH, and DESTDIR environment variables, with the current values shown in the -h output for reference.)

# mkdir -p /usr/local/share/ipxe
# bash build-ipxe.sh
[...]
# ls -1 /usr/local/share/ipxe/
bin-arm64-efi-snponly.efi
bin-x86_64-efi-snponly.efi
bin-x86_64-pcbios-undionly.kpxe

Note

Building for aarch64 requires the package gcc-aarch64-linux-gnu.

Build options

By default, build-ipxe.sh enables support for ZLIB and GZIP images, as well as commands for managing VLANs and the framebuffer console. The x86_64 build also enables support for the serial console.

Additional build options can be configured by editing the build-ipxe.sh script. For example, the x86_64 build is configured in the configure_x86_64 function.

function configure_x86_64 {
  sed -i.bak \
      -e 's,//\(#define.*CONSOLE_SERIAL.*\),\1,' \
      -e 's,//\(#define.*CONSOLE_FRAMEBUFFER.*\),\1,' \
      config/console.h
  sed -i.bak \
      -e 's,//\(#define.*IMAGE_ZLIB.*\),\1,' \
      -e 's,//\(#define.*IMAGE_GZIP.*\),\1,' \
      -e 's,//\(#define.*VLAN_CMD.*\),\1,' \
      config/general.h
}

For example, the imgextract command can be explicitly enabled.

function configure_x86_64 {
  sed -i.bak \
      -e 's,//\(#define.*CONSOLE_SERIAL.*\),\1,' \
      -e 's,//\(#define.*CONSOLE_FRAMEBUFFER.*\),\1,' \
      config/console.h
  sed -i.bak \
      -e 's,//\(#define.*IMAGE_ZLIB.*\),\1,' \
      -e 's,//\(#define.*IMAGE_GZIP.*\),\1,' \
      -e 's,//\(#define.*VLAN_CMD.*\),\1,' \
      -e 's,//\(#define.*IMAGE_ARCHIVE_CMD.*\),\1,' \
      config/general.h
}

Note

IMG_ARCHIVE_CMD is already enabled by default in the iPXE master branch, but only takes effect when at least one archive image format is configured. This is the case in the default state of build-ipxe.sh, which enables support for ZLIB and GZIP archive image formats.

Configuring Warewulf (>= v4.5.0)

In Warewulf v4.5.0, Warewulf can be configured to use these files using the tftp.ipxe and paths.ipxesource configuration parameters in warewulf.conf.

# warewulf.conf
tftp:
  ipxe:
    "00:00": bin-x86_64-pcbios-undionly.kpxe
    "00:07": bin-x86_64-efi-snponly.efi
    "00:09": bin-x86_64-efi-snponly.efi
    "00:0B": bin-arm64-efi-snponly.efi
paths:
  ipxesource: /usr/local/share/ipxe

Restart warewulfd following the change to warewulf.conf. Then remove any previously-provisioned files from /var/lib/tftpboot/warewulf/ and use wwctl configure tftp and wwctl configure dhcp to re-provision the TFTP files and update the DHCP configuration.

# sudo systemctl restart warewulfd
# rm /var/lib/tftpboot/warewulf/*
# wwctl configure tftp
Writing PXE files to: /var/lib/tftpboot/warewulf
Enabling and restarting the TFTP services
# wwctl configure dhcp
Building overlay for wwctl1: host
Enabling and restarting the DHCP services

Configuring Warewulf (< v4.5.0)

Prior to v4.5.0, Warewulf packages included bundled builds of iPXE and did not provide a mechanism for configuring which iPXE to use. To use a custom iPXE before v4.5.0, replace the bundled builds included with Warewulf. After that, remove any previously-provisioned files from /var/lib/tftpboot/warewulf/ and use wwctl configure tftp to re-provision the TFTP files.

# cp /usr/local/share/ipxe/bin-arm64-efi-snponly.efi /usr/share/warewulf/ipxe/arm64.efi
# cp /usr/local/share/ipxe/bin-x86_64-efi-snponly.efi /usr/share/warewulf/ipxe/x86_64.efi
# cp /usr/local/share/ipxe/bin-x86_64-pcbios-undionly.kpxe /usr/share/warewulf/ipxe/x86_64.kpxe
# rm /var/lib/tftpboot/warewulf/*
# wwctl configure tftp
Writing PXE files to: /var/lib/tftpboot/warewulf
Enabling and restarting the TFTP services

Booting with GRUB

Support for GRUB as a network bootloader (replacing iPXE) is available in Warewulf as a technology preview.

digraph G{
node [shape=box];
compound=true;
edge [label2node=true]
bios [shape=record label="{BIOS | boots from DHCP/next-server via TFTP}"]

bios->shim [lhead=cluster1,label="shim.efi"];
subgraph cluster1{
label="Grub boot"
shim[shape=record label="{shim.efi|from ww4 host}"];
grub[shape=record label="{grubx64.efi | name hardcoded in shim.efi|from ww4 host}"]
shim->grub[label="TFTP"];
grubcfg[shape=record label="{grub.cfg|static under TFTP root}"];
grub->grubcfg[label="TFTP"];
}
kernel [shape=record label="{kernel|ramdisk (root fs)|wwinit overlay}|extracted from OS image"];
grubcfg->kernel[ltail=cluster1,label="http"];
}

Instead of the iPXE starter a combination of shim and GRUB can be used with the advantage that secure boot can be used. That means that only the signed kernel of a distribution can be booted. This can be a huge security benefit for some scenarios.

In order to enable the grub boot method it has to be enabled in warewulf.conf.

warewulf:
  grubboot: true

Nodes which are not known to Warewulf are booted with the shim/grub from the Warewulf server host.

Secure boot

digraph foo {
node [shape=box];
subgraph boot {
"EFI" [label="EFI",row=boot];
"Shim" [label="Shim",row=boot];
"Grub" [label="Grub",row=boot];
"Kernel" [label="kernel",row=boot];
EFI -> Shim[label="Check for Microsoft signature"];
Shim -> Grub[label="Check for Distribution signature"];
Grub->Kernel[label="Check for Distribution or MOK signature"];
}
}

If secure boot is enabled at every step a signature is checked and the boot process fails if this check fails. The shim typically only includes the key for a single operating system, which means that each distribution needs separate shim and grub executables. Warewulf extracts these binaries from the images. If the node is unknown to Warewulf or can’t be identified during the TFTP boot phase, the shim/grub binaries of the host in which Warewulf is running are used.

Install shim and efi

shim.efi and grub.efi must be installed in the image for it to be booted by GRUB.

# wwctl image shell leap15.5
[leap15.5] Warewulf> zypper install grub2 shim

# wwctl image shell rocky9
[rocky9] Warewulf> dnf install shim-x64.x86_64 grub2-efi-x64.x86_64

These packages must also be installed on the Warewulf server host to enable node discovery using GRUB.

HTTP boot

Modern EFI systems have the possibility to directly boot per http. HTTP boot is part of the GRUB boot chain described above, and is not a separate bootloader. The node firmware fetches shim.efi over HTTP instead of TFTP, and the rest of the chain proceeds as it does for a TFTP-booted GRUB node. Warewulf does not serve an iPXE binary over HTTP, so HTTP boot cannot be used to start the iPXE boot method.

The flow diagram is the following:

digraph G{
node [shape=box];
efi [shape=record label="{EFI|boots from URI defined in filename}"];
shim [shape=record label="{shim.efi|replaces shim.efi with grubx64.efi in URI|extracted from OS image}"];
grub [shape=record label="{grub.efi|checks for grub.cfg|extracted from OS image}"]
kernel [shape=record label="{kernel|ramdisk (root fs)|wwinit overlay}|extracted from OS image"];
efi->shim [label="http"];
shim->grub [label="http"];
grub->kernel [label="http"];
}

Warewulf delivers the initial shim.efi and grub.efi via http as taken directly from the node’s assigned image.

Note

HTTP boot has the following limitations in the current implementation.

The DHCP rule that hands out the HTTP boot URI is generated differently by each DHCP backend. With ISC dhcpd (the default) the rule is written only when grubboot is enabled, and it matches any HTTPClient vendor class, so the rule matches clients of any architecture. With dnsmasq the rule is always written, but it matches only the vendor class HTTPClient:Arch:00016, which is the IANA processor architecture type for “x64 uefi boot from http”. An aarch64 client booting over HTTP presents architecture type 19, “arm uefi 64 boot from http”, and is therefore not matched.
The HTTP boot URI is built from the server’s IPv4 address, so HTTP boot requires an IPv4 address on the provisioning interface and is not usable on an IPv6-only Warewulf server.
The /efiboot/ route identifies the requesting node from the wwid query parameter or, when that is absent, from a lookup of the client’s address in the IPv4 ARP cache. See Server Routes for details.

Two-stage boot: dracut

Some systems, typically due to limitations in their BIOS or EFI firmware, are unable to load an image of a certain size directly with a traditional bootloader, either iPXE or GRUB. As a workaround for such systems, Warewulf can be configured to load a dracut initramfs from the image and to use that initramfs to load the full image.

The two-stage boot is independent of how the first-stage bootloader reaches the node. It runs over the normal network boot path, whether iPXE or GRUB was delivered over TFTP or over HTTP, and it does not require HTTP boot to be configured.

Warewulf provides a dracut module to configure the dracut initramfs to load the image. This module is available in the warewulf-dracut subpackage, which must be installed in the image.

With the warewulf-dracut package installed in the image, you can then build an initramfs inside the image.

# Enterprise Linux
wwctl image exec rockylinux-9 --build=false -- /usr/bin/dnf -y install https://github.com/warewulf/warewulf/releases/download/v4.7.0/warewulf-dracut-4.7.0-1.el9.noarch.rpm
wwctl image exec rockylinux-9 -- /usr/bin/dracut --force --no-hostonly --add wwinit --regenerate-all

# SUSE
wwctl image exec leap-15 --build=false -- /usr/bin/zypper -y install https://github.com/warewulf/warewulf/releases/download/v4.7.0/warewulf-dracut-4.7.0-1.suse.lp155.noarch.rpm
wwctl image exec leap-15 -- /usr/bin/dracut --force --no-hostonly --add wwinit --regenerate-all

Note

In some systems, such as rockylinux:8, it may be necessary to remove /etc/machine-id for dracut to properly generate the initramfs in the location that Warewulf is expecting.

To direct iPXE to fetch the node’s initramfs image and boot with dracut semantics, set an IPXEMenuEntry tag for the node.

Note

Warewulf configures iPXE with a template located at /etc/warewulf/ipxe/default.ipxe. Inspect the template to learn more about the dracut booting process.

wwctl node set wwnode1 --tagadd IPXEMenuEntry=dracut

Note

The IPXEMenuEntry variable may be set at the node or profile level.

Alternatively, to direct GRUB to fetch the node’s initramfs image and boot with dracut semantics, set a GrubMenuEntry tag for the node.

Note

Warewulf configures GRUB with a template located at /etc/warewulf/grub/grub.cfg.ww. Inspect the template to learn more about the dracut booting process.

wwctl node set wwnode1 --tagadd GrubMenuEntry=dracut

Note

The GrubMenuEntry variable may be set at the node or profile level.

During boot, warewulfd will detect and dynamically serve an initramfs from a node’s image in much the same way that it can serve a kernel from an image. This image is loaded by iPXE (or GRUB) which directs dracut to fetch the node’s image during boot.

The wwinit module provisions to tmpfs. By default, tmpfs is permitted to use up to 50% of physical memory. This size limit may be adjusted using the kernel argument wwinit.tmpfs.size. (This parameter is passed to the size option during tmpfs mount. See tmpfs(5) for more details.)

Network configuration in the dracut initramfs

The :dracut entry passes rd.neednet=1 and an ip= argument for each of the node’s network devices, so that the initramfs can fetch the image from the Warewulf server. The autoconfiguration method is chosen per device from the node’s configuration:

a device with an ipaddr6 and no ipaddr gets ip=<device>:auto6, so that the initramfs configures IPv6 (SLAAC, and DHCPv6 if the router advertisement asks for it) and does not wait for a DHCPv4 lease;
every other device, including dual-stack devices, gets ip=<device>:dhcp.

This matters because on an IPv6-only fabric a request for DHCPv4 is not merely redundant. NetworkManager’s initrd generator translates ip=<device>:dhcp into ipv4.method=auto with ipv4.may-fail=no, which makes a DHCPv4 lease a requirement for the connection to activate. Where no DHCPv4 server exists the connection never activates, nothing that waits on the network proceeds, and the boot ends in the dracut emergency shell with FATAL: Unable to load stage: system.

The value a node will be served can be read without booting it.

curl -s "http://<server>:<port>/ipxe/<hwaddr>" | grep 'set dracut_net'

That prints one line for the :dracut entry and one for :dracut_static. The latter builds its ip= argument from ipaddr, gateway and netmask, so it does not support a node that has only an ipaddr6.

Note

When adjusting the method by hand in a copy of the template, prefer auto6 over dracut’s either6. NetworkManager’s initrd generator does not recognise either6: it falls back to interpreting the argument positionally, fails to read the device name as an address, and discards the whole ip= argument.