I am desperate. Linux is not booting

oiram15@lemmy.sdf.org to Linux@lemmy.ml – 98 points –

SOLUTION BELOW

The actual bug

I have never been in a more confusing situation regarding Linux.

I have a Dell XPS 15 9560, which had a dual boot Windows 10 / EndeavourOS setup. It was running fine for months. 10 days ago I updated Linux and after restart it couldn't boot anymore. It got stuck at "A start job is running for /dev/disk/by-uuid/..." (which is the root partition).

First, with the help of a friend of mine who is quite knowledgeable about Linux (he runs vanilla Arch, etc), we spent 5 hours trying to fix it but had no luck.

Then I decided to back up everything and do a fresh install. Aaaand the same error happened again on the first boot. Then I though "ok, probably some problem with Arch, lets try Fedora". Nope. Some similar error about not finding the root partition. (Here I must say that the kernel which was shipped with the ISO was working fine, but after updating to the latest one, it failed.) Here I thought "ok, then it might be a problem with the latest kernel, let's install EndeavourOS with the LTS kernel." Nope, LTS kernel also didn't boot. Then I tried Ubuntu and it worked, but that's not solving the problem. Then I decided to put another nvme drive in the laptop and try there. The same error again.

Now the greatest part: If I put the nvme drive into an external usb case, EndeavourOS installs, updates, boots without any problem, no sign of the error.

So now I don't know how to proceed... Maybe there is something wrong with the pcie port in my laptop, but except for the booting problem, windows is working, I can also mount and access every partition in the ssd through a live usb. So no other signs of problem with the port whatsoever.

I would be grateful for any advice as I've lost several days trying to solve this and I am out of ideas...

Solution: The last working kernels are from 11. August 2023 (both linux and linux-lts) linux-6.4.10.arch1-1 and linux-lts-6.1.45-1. You can download them from here: linux / linux-lts and install them with

sudo pacman -U the_path_to_the_package

Thank you all for the help!

59

What is the output of blkid and the content of /etc/fstab of the installed-but-not-booting system? You will need a live usb to get the those, probably.

[root@EndeavourOS /]# blkid
/dev/nvme0n1p1: UUID="6903-7FA3" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="86a0a714-815f-45db-ad0e-9eba16861903"
/dev/nvme0n1p2: LABEL="endeavouros" UUID="9ae3c50f-be08-4594-ac30-2d094375868d" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="endeavouros" PARTUUID="9b52162c-a4bf-4e5e-8096-bb7f144c481a"
/dev/loop0: BLOCK_SIZE="1048576" TYPE="squashfs"
/dev/sda2: SEC_TYPE="msdos" LABEL_FATBOOT="ARCHISO_EFI" LABEL="ARCHISO_EFI" UUID="6121-B369" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="1bdb3cc0-02"
/dev/sda1: BLOCK_SIZE="2048" UUID="2023-08-05-13-57-43-00" LABEL="EOS_202308" TYPE="iso9660" PARTUUID="1bdb3cc0-01"
[root@EndeavourOS /]# sudo cat /etc/fstab 
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
#                      
UUID=6903-7FA3                            /efi           vfat    defaults,noatime 0 2
UUID=9ae3c50f-be08-4594-ac30-2d094375868d /              ext4    defaults,noatime 0 1
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0

Please use tripe backticks to create codeblocks - much easier to read. Like this:

```

some text

```

results in

some text

What are the kernel parameters? cat /proc/cmdline

EDIT actually that will show the live system config. Assuming you're using grub, what is the content of /etc/default/grub?

Or /boot/loader/loaders/something.conf if you're on systemd-boot

I'm on systemd-boot. Where isn't a directory loaders under loader, but I found the parameters under /etc/kernel/cmdline:

nvme_load=YES nowatchdog rw root=UUID=9ae3c50f-be08-4594-ac30-2d094375868d

My bad, I think in your case it's in /efi/loader/entries/something.conf

Since / is not mounted, yet, bootloader will not be able to read anything under /etc/. Unless it's used to automatically populate the loader.conf.

Also check /efi/loader/loader.conf.

I found it!

[liveuser@eos-2023.08.05 ~]$ cat /mnt/efi/loader/entries/02ef85f9edc146d598502c1b296ff64a-6.4.12-arch1-1.conf 
# Boot Loader Specification type#1 entry
# File created by /etc/kernel/install.d/90-loaderentry.install (systemd 254.1-1-arch)
title      EndeavourOS
version    6.4.12-arch1-1
machine-id 02ef85f9edc146d598502c1b296ff64a
sort-key   endeavouros-6.4.12-arch1-1
options    nvme_load=YES nowatchdog rw root=UUID=9ae3c50f-be08-4594-ac30-2d094375868d systemd.machine_id=02ef85f9edc146d598502c1b296ff64a
linux      /02ef85f9edc146d598502c1b296ff64a/6.4.12-arch1-1/linux
initrd     /02ef85f9edc146d598502c1b296ff64a/6.4.12-arch1-1/initrd

I've never used machine-id with systemd-boot, but everything appears to be corrent. Presumably, /boot contains a directory named 6.4.12-arch1-1, which contains files linux and initrd, correct?

You could try rebuilding the initramfs with mkinitcpio --allpresets while chrooted.

they are under /02ef85f9edc146d598502c1b296ff64a/6.4.12-arch1-1/, but yes.

EndeavourOS is using dracut by default.

Edit: we tried rebuilding initramfs before, but it didn't help

OK, I see nothing wrong. Let's try building a new config that's as minimal as possible. Copy linux and initrd files to /boot/.

/efi/loader/entries/test.conf

title      Test
options    root=/dev/nvme0n1p2
linux      /linux
initrd     /initrd

I got another errors now

https://postimg.cc/Z9dyg1Vn

I think failure to change power states is a big issue, but this is out of my depth now. Sorry :(

It matches the observation with the external USB enclosure though. I think the ASPM / ACPI path would be the most promising.

If you know a last working and a first broken kernel version, maybe do a bisection.

[root@EndeavourOS /]# blkid
/dev/nvme0n1p1: UUID=“6903-7FA3” BLOCK_SIZE=“512” TYPE=“vfat” PARTUUID=“86a0a714-815f-45db-ad0e-9eba16861903” /dev/nvme0n1p2: LABEL=“endeavouros” UUID=“9ae3c50f-be08-4594-ac30-2d094375868d” BLOCK_SIZE=“4096” TYPE=“ext4” PARTLABEL=“endeavouros” PARTUUID=“9b52162c-a4bf-4e5e-8096-bb7f144c481a”
/dev/loop0: BLOCK_SIZE=“1048576” TYPE=“squashfs”
/dev/sda2: SEC_TYPE=“msdos” LABEL_FATBOOT=“ARCHISO_EFI” LABEL=“ARCHISO_EFI” UUID=“6121-B369” BLOCK_SIZE=“512” TYPE=“vfat” PARTUUID=“1bdb3cc0-02”
/dev/sda1: BLOCK_SIZE=“2048” UUID=“2023-08-05-13-57-43-00” LABEL=“EOS_202308” TYPE=“iso9660” PARTUUID=“1bdb3cc0-01”

[root@EndeavourOS /]# sudo cat /etc/fstab
/etc/fstab: static file system information.
Use ‘blkid’ to print the universally unique identifier for a device; this may
be used with UUID= as a more robust way to name devices that works even if
disks are added and removed. See fstab(5).

UUID=6903-7FA3 /efi vfat defaults,noatime 0 2
UUID=9ae3c50f-be08-4594-ac30-2d094375868d / ext4 defaults,noatime 0 1
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0

I assume it should look like this

So this occurs after an update. Is it not possible to boot into the prior kernel?

If possible to boot into the prior kernel, can you inspect logs or the journal to see where your error is cropping up?

This issue sounds like a regression of sorts with a driver, but log/debug would help confirm. This would be one worth reporting to upstream if you can rescue some logs (I gather you can if you can boot the disk from another enclosure).

If you can boot into the machine, investigate note from the journal:

  • journalctl --list-boots
  • journalctl -b -1,
    • where -1 was the prior boot, -2, the one before that, etc

-- If you are booting into a live environment or are otherwise mounting the disk:

  • journalctl -D /var/log/journal/ID_GOES_HERE
  • example path: /var/log/journal/2dff8304d5114c44bfb1311357a3cd87

-- Keep us posted.

If truly a driver regression, but you can boot from the prior kernel (if you don't have it, install it via livecd or so), definitely report this one and remain on the prior kernel until resolved. Bleeding edge things.

I have already wiped everything, so no logs... The only way to have it booting is to install EndeavourOS using the offline installer, which is using kernel 6.4.8. There is an option to install the LTS kernel alongside. So the system is booting with 6.4.8, but after updating, neither the new 6.4.12, or the LTS, which is 6.2, doesn't boot. I haven't tried booting with the LTS kernel before updating, to see if the same kernel is working before and after or not. I will try to reinstall it using the offline installer and then try to gather some logs after updating.

If you do go this route - try to update the kernel only, not the rest of the system. Yes, it's bad under normal circumstances, but I think we're way past those.

sudo pacman -Syy linux

This would rule out some other part of system getting in the way - like systemd.

I updated only the kernel and I got the following:

:: kernel-install installing kernel 6.4.12-arch1-1
dracut: Executing: /usr/bin/dracut --no-hostonly --force /efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd-fallback 6.4.12-arch1-1
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'multipath' will not be installed, because command 'multipath' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoemon' could not be found!
dracut: dracut module 'fcoe' will not be installed, because command 'fcoeadm' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'dcbtool' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'fipvlan' could not be found!
dracut: dracut module 'fcoe-uefi' will not be installed, because command 'lldpad' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'nbd' depends on 'network', which can't be installed
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: modsign ***
dracut: *** Including module: i18n ***
dracut: *** Including module: btrfs ***
dracut: *** Including module: crypt ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 56-lvm.rules
dracut: Skipping udev rule: 60-persistent-storage-lvm.rules
dracut: *** Including module: mdraid ***
dracut: Skipping udev rule: 64-md-raid.rules
dracut: *** Including module: nvdimm ***
dracut: *** Including module: qemu ***
dracut: *** Including module: qemu-net ***
dracut: *** Including module: lunmask ***
dracut: *** Including module: resume ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: virtiofs ***
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode:                     real
dracut: Method:                   memcmp
dracut: Files:                    1987
dracut: Linked:                   8 files
dracut: Compared:                 0 xattrs
dracut: Compared:                 449 files
dracut: Saved:                    1.42 MiB
dracut: Duration:                 0.022128 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing AuthenticAMD.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd-fallback' ***
dracut: *** Creating initramfs image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd-fallback' done ***
dracut: Executing: /usr/bin/dracut --hostonly --no-hostonly-cmdline -f /efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd 6.4.12-arch1-1

(Continues in the second comment)

dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'kpartx' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'squash' will not be installed, because command 'mksquashfs' could not be found!
dracut: dracut module 'squash' will not be installed, because command 'unsquashfs' could not be found!
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: i18n ***
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: Skipping udev rule: 70-persistent-net.rules
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode:                     real
dracut: Method:                   memcmp
dracut: Files:                    780
dracut: Linked:                   2 files
dracut: Compared:                 0 xattrs
dracut: Compared:                 38 files
dracut: Saved:                    356.65 KiB
dracut: Duration:                 0.010006 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd' ***
dracut: *** Creating initramfs image file '/efi/69f1b920c192454c97831ba6e72bf777/6.4.12-arch1-1/initrd' done ***
(4/5) Check if user should be informed about rebooting after certain system package upgrades.
(5/5) Checking which packages need to be rebuilt

Did it boot or did it get stuck at the same spot again?

It got stuck at the same spot.

Implies it is the kernel. Mark it on hold and stick to the working kernel for the time being.

Does EndeavourOS use pacman?

You might consider modifying /etc/pacman.conf to include the option IgnorePkg=linux linux-lts until this is resolved. 6.4.12.arch1-1 was added ~4 days ago. If you check the releases arch linux kernel releases here, they have nearly a weekly cadence. This may be a case you can ride out until a newer kernel is released that might solve your issue.

If you need access to older releases, see the archive.

For root cause analysis -- is it possible for you to extract/view the journal logs now that you have upgraded the kernel causing the issue? -- /var/log/journal is a good start. My time is limited, but I'm curious to see what's going on in there. In any case per your and Illecors' testing, it seems you have isolated the issue to the linux package.

I just realized you're also on systemd-boot (I am too). I'll check into a way to revert back to prior kernels (for I also may run into a similar issue).

edit: updated IgnorePkg line to include both your mainline and LTS kernel (Pacman -Syu updates both) if you opt to hold them for updates.

This is the log https://ufile.io/p5wj1hu0

I'm not sure if this is relevant, but I added pci=nommconf to the kernel parameters as I had errors similar to the one in this topic, otherwise the log was over 40 000 lines long.

Edit: I'm not sure if this is exactly a kernel problem, as I get the same error when booting with the LTS kernel, which is older than the one installed with the offline installer.

No worries. When checking that output, it is for the working 6.4.8-arch1-1 kernel. The broken kernel boot attempt would be most useful, but I don't want to make you suffer to get it, if you are back to a working system. I think at this point it is safe to say your laptop isn't a fan of the newer kernels.

I would :

  1. (fresh install/andor working machine) update your /etc/pacman.conf to ignore updates to packages linux and linux-lts
  2. Devise a way to add multiple systemd-boot boot entries. I was working on this just a bit ago but I don't have it fool proof and it drops you to an emergency shell. So I am hesitant to share this at the moment.

--

Ideally: You could (from a working system) install a known working LTS image (pkg linux-lts), and exclude that from updates until you land on a working kernel release (keep an eye on testing and core repos once a week or so). in this way, you'll have a working LTS, and can upgrade/downgrade mainline kernels as you please, booting back into LTS to correct issues should they arise.

edit: minor

I found the problem! It was actually the kernel, just both the latest and the lts versions released after 11th of August don't work. So for now I need to stick to the versions from 11th. Thank you very much for the help!

Edit: this is the bug

According to your logs when installing the kernel, your nvme modules are nowhere to be found, and basic system tools are unavailable. Your core system seems severely borked... can you do a memtest (broken ram can corrupt your storage and your attemps at finding the problem) and check your smartctl --all of that ssd?

when it gets stuck on something like that, it's because of a hardware related change; did you update or modify your bios recently or change hardware drivers?

Use a live USB to back up your home directory and a list of your packages, then reinstall. I don't think it's worth the trouble.

AHAHAHA that is a proper insane bug. One PCIe device shouldn't be able to slap others off of the "bus", "we're not on a bus all you did was mess up your own personal lanes mate".

I lost 3 days of my life + now I need reinstall everything 🥲

That's a weird issue. Do you have encryption on by any chance? I had a similar error pop-up when I didn't have the correct systems hooks for the kernel so that after install update of kernel would make the system not boot.

Share the errors you are experiencing, please

Could you please take a look at the comments from abrer and Illecors, I have shares a lot of info there.

Could you link to them or quote them, please?

The problem is already solved (At least I found the bug and a temporary solution) At the beginning of the post there is a link to the bug report.

I had se similar issue but with bluetooth, worked on windows not on linux. Fixed it by resetting the BIOSwith the CMOS battery thing

Shot in the dark: Did SecureBoot get re-enabled somehow? Could you check the UEFI?

Hard to not sound condescending in text, so forgive me.

If systemd is trying to mount the drives, we're at a point where the (most likely) ramdisk has already booted the kernel - secure boot would've prevented that.

I won't take it as condescending then! Thank you for the input :)

the help of a friend of mine who is quite knowledgeable about Linux (he runs vanilla Arch, etc)

Contrary to popular belief using and maintaining arch is a novice exercise. The btw crowd likes to believe their actions are '1337' 'minimal' 'no bloat', etc. and they've sure got the neofetch receipts to prove it!

The truth of it is that arch is easy, that's the whole allure. It is the most convenient distro for experienced users with specific (yet broad) set of needs.

The btw crowd are simply misled kooks deserving of pity and mercy, but not trust.

https://en.m.wikipedia.org/wiki/Four_stages_of_competence

Note to future self: Don't dual boot any system EVER.

Dual (or triple or quad) booting isn't the problem, I did it for a long time, until I had a machine that could handle more than one virtual machine while leaving the base OS also usable.

I always dual boot and have never had an issue in many years. I wonder what I am doing differently.

The only way I could dual boot without Windows breaking things was to unplug the Windows drive, install Linux on another drive, plug the Windows drive back in, and boot to Windows by selecting it in the BIOS.
Trying to use grub would always have issues eventually after booting into Windows.

That's definitely not the norm. Used to be that installing Windows would wreck Grub, but you just needed to but a rescue disk and reinstall Grub one time to fix it. Most people dual booted for decades without any issue there.

To be fair this would have been decades ago. Ubuntu 8.xx and either XP or 7. It was my first dive into Linux so there's a good chance I did something wrong.
I was always able to repair grub when windows would wreck it thankfully.

My current setup is using the former, but I did the latter for so many years and the only issue I had was the clock would get fucked up after booting Windows. But I agree, it's much safer to have your Windows SSD physically removed when installing Linux so grub doesn't get installed on the same SSD as Windows. I wouldn't do it with all the horror stories I have read. My good experience was probably dumb luck.

Couldn't you use grub on one drive to point to the bootloader on a separate drive? Windows should leave that configuration alone, at least in theory.

Probably, but I just didn't want to give it the chance. It ended up being a non-issue in the end because I never booted back into Windows.

1 more...