Recommendations on Changes to Homelab

monty@lemmy.one to Selfhosted@lemmy.world – 19 points –

I currently run a Lenovo minipc and an rpi4 to host all of my homelab services, and I just purchased a new minipc with an 11th Gen i5 in order to better handle Plex and transcoding. I would also like to add Immich to my homelab. Here is a breakdown of my current state:

Lenovo Mini PC running Proxmox: Home Assistant OS VM Ubuntu VM running:

  • Frigate
  • Pi-hole
  • Plex
  • Arr Stack
  • Transmission
  • Has two ext HDs in a mergerfs configuration to host all media and onsite homelab backups.

RPi4:

  • Wireguard VPN

  • Pi-hole

I am going to install Proxmox on the new minipc, and setup both in a cluster. I would like to move Plex, Arrs, Transmission, and Immich to the new mini PC. Because of that I am thinking the external drives should be connected directly to the new minipc to maximize performance. I would also like to have the ability to share the external drives to any nodes in the cluster.

Do you have any recommendations on how to best setup these drives? Should I setup a TrueNAS VM on the new minipc and share to all others via NFS? Any other tips for my setup?

12

You may have trouble keeping a quorum with a proxmox cluster of 2. You should really have 3, or set up a q-device. (Source)

Each node in a cluster has a "vote" in a healthy cluster. If one of your devices goes (or is taken) down, you'll lose quorum, and the GUI and some other stuff. You need 3 votes for things to work reliably. I have an old RPi set up as a q-device and it works fine.

This only applies to high availability clusters, correct?

Unfortunately not, it applies to all clusters. A quorum requires at least 2 votes, which means 2 nodes. But if one node goes down, you only have 1 vote, and the cluster will go into read-only mode, which means you'll lose the GUI and the ability to manage your nodes:

When your cluster is non-quorate (so at least half of all nodes are dead), the remaining nodes will change the PVE management into read-only mode. Because of that you will no longer be able to change and manage your VMs and containers and will also not be able to log into the GUI. This is done to avoid cluster split-brain problems in which they run into inconsistent states. (Source)

But if you have a device that will supply a vote in the event that one of the two votes is unavailable, the cluster will continue to function with a single node, which will allow you to use the normal Proxmox tools and interfaces to diagnose the problem, while also keeping any VMs on the single remaining node up and running (available).

Here is a Proxmox employee explaining it a bit more clearly than the official documentation:

... a cluster needs to be always quorate to work properly, not just for HA. High availability just means that the cluster will try to keep your HA-enabled VMs and containers always available, i.e. if a cluster node fails the HA-manager will launch HA-managed guests on another cluster node.

While a 2 node cluster should work with 2 active nodes, if one of your nodes goes down your cluster will automatically be non-quorate and will no longer work as expected. To have quorum in your cluster, you need a setup of at least 3 nodes, though you do not need 3 full Proxmox installations ... you can setup something like a Raspberry Pi as a QDevice for external vote support.

(Source)

Wow thanks for that detailed explanation! Very helpful and I'll be setting up an rpi as a device.

No need for the TrueNAS VM, Proxmox supports RAID, ZFS, and all that natively.

To share the storage with other nodes you can set up an NFS share and add that in Proxmox and into VMs to use as storage.

I did some searching trying to figure out how I can share the NFS directly from Proxmox and came up short. Any tutorials or other links you know of?

Well it’s a Debian based Distro, so you could install NFS. What I do is I have OMV running in a vm and use that to serve up my smb and NFS shares

It's just a Debian system so any general linux guide for NFS shares will work.

Got it so I'm just overthinking this. I setup an NFS share through the host via the command line and then I'll be able to share that to anything on the network.

Yep! That's how I do my media share that's using ZFS on a Proxmox host

So I'm having some permission issues with this. If I create a file within one container, it is read only when accessed from another. Any suggestions on how to fix this?

Easiest is just chmod -R 777 the files which is what I do. I'm sure there's a proper way to do it, but it's just media files.