Recovery From GRUB Configuration Issues in Proxmox

Even with careful planning, GRUB parameter changes can sometimes render your system unbootable. Let's understand how to recover your system when this happens, just as we did when our framebuffer parameters caused issues.

Understanding What Happens When GRUB Configuration Breaks

When GRUB parameters prevent proper system boot, you'll typically encounter one of these scenarios:

• A black screen with no console output
• A system that appears to start booting but freezes
• Boot errors that you can't read because the display isn't functioning
• A system that cycles through boot attempts

Think of this situation like being locked out of your house - you need an alternative way to get in. In our case, that alternative access comes through a rescue system.

Recovery Process Using a Rescue USB

Step 1: Prepare Your Rescue Media

First, you'll need a Debian-based installer USB. The Debian installer works well because:

• It provides a robust rescue system
• It can handle various filesystem types
• It includes essential recovery tools
• It's compatible with Proxmox VE's Debian-based system

Step 2: Boot Into Rescue Mode

1. Insert your USB drive and boot from it
2. When the Debian installer menu appears, choose "Advanced Options"
3. Select "Rescue Mode" from the advanced options menu
4. Follow the initial prompts about language and keyboard layout
5. When you reach disk selection, choose your Proxmox installation disk
   • In most cases, this will be /dev/pve/root
   • If you're unsure, look for the disk that matches your Proxmox installation size

Step 3: Mount Your System

1. When asked about mounting the system:
   • Choose to mount the system
   • This makes your Proxmox installation accessible for repairs
2. When prompted about additional mount points:
   • Generally, you can skip these for GRUB repairs
   • The root filesystem mount is sufficient

Step 4: Access and Repair GRUB Configuration

Choose to execute a shell in the mounted system

Navigate to the GRUB configuration directory:

cd /etc/default/

If you followed our earlier advice about backups, check your backup files:

ls -l grub*
# You should see files like grub.backup-20240202

Check your current problematic configuration:

cat grub

Restore the working configuration:

# If you have our backup
cp grub.backup-YYYYMMDD grub

# Or manually edit the file to use our known working parameters
nano grub

# Add these working parameters:
GRUB_CMDLINE_LINUX_DEFAULT="nosplash nomodeset verbose debug intel_iommu=on iommu=pt rd.driver.pre=vfio-pci pcie_acs_override=downstream,multifunction"

Step 5: Apply Changes and Reboot

Update GRUB from within the rescue environment:

update-grub

Update the initramfs:

update-initramfs -u -k all

If you get this message:
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Dont worry, it is not an error

Verify your changes:

cat /etc/default/grub

Exit the rescue shell:

exit

Choose to reboot the system, remove your rescue USB and wait for your system to reload.

Preventing Future Issues

To avoid needing rescue operations in the future:

• Always maintain dated backups of working GRUB configurations
• Make one change at a time when modifying GRUB parameters
• Test new parameters in a temporary GRUB entry first
• Keep a record of what each parameter does and why you added it

Remember that GRUB configuration issues, while stressful, are always fixable. The key is to make changes methodically and always maintain a path back to a working configuration. This rescue process has saved many systems, including our own when we experimented with framebuffer parameters.