Diagnosing and Fixing Boot Failures in AlmaLinux 8 and 9

Identifying the Point of Failure

When an AlmaLinux system fails to boot, the specific screen where the process stops determines whether the issue lies with the bootloader, the initial RAM disk, or the operating system configuration. Identifying this boundary prevents applying the wrong fix, such as rebuilding a kernel when the issue is actually a corrupted GRUB configuration.

Symptom	Failure Layer	Likely Cause
Black screen or "No bootable device found"	Firmware/GRUB2	Corrupt MBR/EFI partition or missing bootloader
Drops to dracut# shell	Initramfs	Root device not found or missing storage drivers
Drops to Give root password for maintenance	Systemd/OS	/etc/fstab errors or failed system units
Kernel Panic (text wall)	Kernel	Incompatible kernel update or hardware failure

Phase 1: GRUB2 and Bootloader Recovery

If the system cannot find the bootloader or drops to a grub rescue> prompt, you must use AlmaLinux rescue media (ISO) to restore the environment.

1. Boot from the AlmaLinux installation media and select Troubleshooting > Rescue a AlmaLinux system.
2. Choose 1 to mount the system to /mnt/sysroot.
3. Enter the chroot environment: chroot /mnt/sysroot.
4. Reinstall the GRUB binaries. Note that the command differs by boot mode:
   • BIOS: grub2-install /dev/sda (Replace /dev/sda with your boot disk).
   • UEFI: The bootloader is typically managed via efibootmgr; ensure the EFI partition is mounted at /boot/efi.
5. Regenerate the configuration file:
   • BIOS: grub2-mkconfig -o /boot/grub2/grub.cfg
   • UEFI: grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg

Risk: Writing the grub.cfg to the wrong path will result in the system continuing to use the old, broken configuration.

Phase 2: Resolving Dracut (Initramfs) Shells

The dracut shell appears when the kernel has loaded but cannot mount the root filesystem. This often happens after changing disk controllers or updating storage drivers.

Diagnostic Checks

Run these commands within the dracut shell to identify the missing device:

• blkid: List all available block devices and their UUIDs.
• lvm vgchange -ay: Attempt to activate Logical Volume Management (LVM) volumes if the root is on LVM.

Fixes

If blkid shows the disk but the boot fails, the initramfs may be missing the required drivers. Boot into a rescue environment, chroot, and run:

dracut --force

If the UUIDs in /etc/fstab do not match the output of blkid, update /etc/fstab with the correct UUIDs to ensure the kernel can locate the root partition.

Phase 3: Systemd Emergency Mode and Fstab

If the system boots but stops at an emergency prompt, the kernel is functional, but a critical system service or mount point has failed.

Diagnostic Step

Run the following command to find the specific error causing the halt:

journalctl -xb -p err

Look for entries mentioning Timed out waiting for device or Failed to mount.

Common Fixes

• Faulty Mounts: If a non-critical network drive or secondary disk is missing, edit /etc/fstab and add the nofail option to that entry. This allows the system to boot even if the device is unavailable.
• SELinux Relabeling: If the system hangs during login after a major migration, boot with enforcing=0 added to the kernel line in GRUB. If this fixes the boot, trigger a full relabel: touch /.autorelabel and reboot.

Kernel Panic Rollback

If a kernel panic occurs immediately after a dnf update, the fastest resolution is to roll back to the previous known-working kernel.

1. Reboot the system.
2. At the GRUB menu, use the arrow keys to select the second or third kernel entry (the previous version).
3. If the system boots successfully, the new kernel or its associated initramfs is corrupted. You can remove the broken kernel using dnf remove kernel-core-VERSION and reinstall it.

Escalation Criteria

Stop manual troubleshooting and escalate to hardware vendors or deep-system recovery if:

• smartctl reports "Critical Warning" or failing sectors on the boot drive.
• xfs_repair or e2fsck reports recurring metadata corruption after multiple repair attempts.
• Kernel panics persist across multiple different kernel versions, suggesting a RAM or CPU failure.

Verification

To verify the fix, perform a "Clean Boot Test":

1. Reboot the system twice. The first boot ensures the fix is active; the second ensures no transient state (like an .autorelabel) is masking a recurring issue.
2. Run systemctl get-default to confirm the system is returning to the expected target (usually multi-user.target or graphical.target).