Which OS do you use for your homeserver?

melandroph@lemmy.dbzer0.com to Selfhosted@lemmy.world – 122 points –

Edit: wow, this is a never ending comment section!

204

NixOS

I just heard of NixOS for the first time because of this thread. Looked up some videos on it, and my jaw hit the fucking floor.

Same here. I came for the integrated ZFS support and stayed for the declarative config.

how is nix better than debian for servers?

Declarative configuration of services and the rest of the entire system, and everything that brings with it.

  • Want to test some new service, or make changes to an existing one, but don't know if you want to keep it? Sure, just temporarily switch to the new configuration, you can always switch back to the old one and everything will be back as it was.
  • Have multiple servers and want to share configuration between them? Absolutely, just import the same file from both. I have a git repo storing configurations for 10 machines and a huge part of it is shared configuration.
  • Want to use one service's endpoint (such as a socket path) in another? Sure, just use the socket path configuration option for the first service in the configuration for the second, such as here. This works since everything is a single tree of options which all the service configuration files are then generated from, so interpolate stuff as you wish.
  • Checks for configuration correctness during build of the system (NixOS options are type checked during evaluation, and then during the actual system build there's more checks, like nginx config has to succeed nginx -t, otherwise the system build fails and you can't switch to it)
  • Want to spin up a VM to test changes before putting it on the actual target? There's a builtin command (nixos-rebuild build-vm) that makes a script that starts a QEMU VM with your configuration running in it. It's as fast as building the real system, so a couple seconds if you're making small changes.
  • Setting up services is also often as easy as putting services.foo.enable = true; in your configuration. And, if you remove that line, the service is gone, so you're never left with "the random package or file you installed once to test something and has been forgotten about". That's the biggest thing it has over any kind of imperative solution IMO.

I feel like even if I want to distro hop again and end up putting something else on my desktop, NixOS is going to stay on my servers indefinitely. It's pretty much a perfect fit for servers.

It isn't, it's just different. I use NixOS because of stupid easy rollbacks, which is great for experimenting in production, and its declarative nature, which is great in a server setting.

Everything is declared, from packages to configuration, and then I can put it in a git repo locked to versions. If something breaks on updates, you have free rollbacks. Which means you can't screw up too much. Also it has almost all the software.

Proxmox (debian) on the hosts, and Debian for all the VMs and Containers.

Just nice and easy to use, supported by basically everything, and a minimal install uses like 30MB of RAM.

I also have an OSX VM because that's literally the only way you can test a website in Safari (fu Apple).

Love proxmox. Been using it for nearly a decade and while it has its pain points it has been rock solid for me for the past 4 years.

You don't need Safari unless it's for Apple Pay integration or something. WebKit is open source. Use Epiphany or some other browser that uses it.

when vendors pull this kind of crap ill simply not test on their software.

It's more that like 60% of my web traffic is Safari so I want to make sure it works for those people.

thats unexpected, what are you doing thats being used by a majority of apple users?

Perhaps a website for an iOS or Mac app? 😊

Three HP ProLiant servers running ProxMox cluster. Each box has a VM for Portaiber, as well as mismatch of VMs running Home Assistant OS, OpenWRT, Ubuntu, Windows and Debian, along with a Windows file server that connectes to four cheap NAS running Ubuntu LTS with a combined 20 mismatched hard drives by iSCSI and borgs them together with Storage Spaces.

It's a fucking mess, if I'm honest.

Debian.

Stable, well documented, easy to install. I do not need anything else right now.

Debian. It is rock solid. If software doesn't support Debian, chances are it supports something Debian based. You never have to worry about an update breaking your computer. It is the perfect "it just works" distro for a server.

Ubuntu LTS, with all my services in Docker containers.

I know Ubuntu gets a lot of (deserved) hate for some of the shit Canonical pulls, but for now, I like Ubuntu and it works for me.

When I rebuilt my server at the beginning of the month, I was gonna jump to Debian, but my god the Debian website is obtuse. After looking at the site and trying to determine what to download to get Debian with non-free (I’m unfortunately working with an NVIDIA card), I decided to go with Ubuntu. I needed a smooth rebuild process and with Ubuntu I know exactly what I’ll get when I download the LTS server.

Edit: grammar

It's always best to use whatever distro you're most comfortable with. Especially if you're going to install stuff in containers/VMs so the repos of the base distro don't even matter that much.

Exactly. That’s ultimately why I skipped Debian and went with Ubuntu

I went with Ubuntu server and was pleasantly surprised when it offered to pull my pubkey off my github profile for ssh. A nice touch that I haven't seen in other servers flavors of various distros.

After looking at the site and trying to determine what to download to get Debian with non-free (I’m unfortunately working with an NVIDIA card)

FWIW, Debian 12 now includes non-free firmware in the installation media by default and will install whatever is necessary.

I agree that the Debian website has its weaknesses, but beyond finding the right installer (usually netinst ISO a.k.a small installation image on https://www.debian.org/distrib/) there isn't much of a learning curve. I started out with Ubuntu too, but finally decided that enough was enough when snap started breaking my stuff on desktop.

The inclusion of non-free by default was what was unclear to me from the website. Knowing that now, I’ll likely give Debian a spin next time I need an install.

Arch Linux. I am so used to it I just can't live with any other OS

I am super impressed with Arch on my home servers. People seem to think "rolling" means "unstable" but the only issues I've had were due to some weird hardware incompatibility with my motherboard. Once I replaced the mobo my system has been rock solid AND reasonably up-to-date (I do use LTS kernel).

I felt the exact same way. So many comments online told me that running Arch as a home NAS was insane, but after the Jupiter Broadcasting guys did it without much issue, I decided to give it a go and was pleasantly surprised. I think if most of your stuff is running in Docker and you have BTRFS snapshots for your root filesystem, the system's pretty much bullet proof. The rolling updates also mean you'll never have huge upgrade cycles that are a pain in the ass to migrate to. You're always just dealing with small manageable fires instead of large complicated ones and that's a plus.

Proxmox for the the hosts, Debian cloud imagen for the VMs and docker inside

Ubuntu Server with docker/docker-compose on top.

So many guides for Ubuntu specifically makes reading up on something a lot easier and it works just fine.

NixOS, I find the config very easy and quick

I've just dipped my toes into it, but I imagine migrating to another machine to be just gorgeous..

Ubuntu 22.04 server. It works well enough for my purposes and until it doesn't I don't see a reason to switch distros.

OpenBSD for all of them.

How is the OpenBSD experience? I have 2x4TB hard drives in my Libreboot server (Dell T1650 motherboard), can I easily setup RAID 1 through the OS?

Hey @Zeon,
Any chance you still have that libreboot t1650 rom still? I’ve bought your exact setup instead of the rtx 2080 super I have rtx 2070 super founders edition. If you don’t have the rom it’s fine I can build one but just wanted to see the specifics on your rom.

The rom for the T1650 is weird, I tried updating mine and got no display, so I'm sticking with my old rom right now. Are you having the same issue? You can build the rom in lbmk or download the roms from one of the Libreboot mirrors.

I am currently adding support for the Dell Optiplex 9020 MT, it supports a i7 4790K and 32GB of DDR3 1.5v RAM (Non-ECC). The i7-4790K is a little bit faster, like 6% overall, but it's main feature is having AVX2 support. This increases the peformance drastically for machine learning (e.g. LLMs), compiling peformance, and even virtual machines. It's basically modern-like gaming computer. It's actually the first Libreboot desktop computer to support AVX2.

Happy to hear I inspired you to build a Libreboot gaming machine! Stay tuned for the next port, you can expect a release by next week!

OpenBSD is the most pleasing expérience I've had with an OS. It's fully contained and has all the tools you need without needing to install anything (eg a DNS, HTTP, SMTP servers, a proxy, a good firewall). All config files look alike and use the same keywords for the same things, making it straightforward to configure everything.

And regarding RAID 1, I've never done it myself, but it totally works out of the box (as well as full disk encryption).

Proxmox with Debian LXC containers. The most natural transition from Raspberry Pi OS which is a Debian flavor

Same. Haven't had the need for full blown VMs at all. Passing through the iGPU for transcoding took a bit of time to figure out, but works great. I do have an Arch LXC container for some apps without a deb repository, though, to keep them updated through AUR.

Unraid, mostly due to the flexible arrays.

Same here, when I made mine I had a whole mix of different sized drives so it made sense. I like not having to worry about drive size, as long as they are smaller than the parity drives.

I'm running FreeBSD I actually like it a lot.

I picked it for zfs. A lot of the ways things work seem cleaner and simpler than on Linux and zfs is awesome with the copy on write snapshots and filesystem compression and all that. I like rc.conf and pf is way nicer than iptables and even when you upgrade it automatically makes a snapshot so you can rollback.

Sometimes I do need to patch and compile things because people seem to not know freebsd exists but that's really the only downside.

Same here for the same reasons (although I started with FreeBSD 4.x) and have adapted to ZFS and Jails over the years.

The POLA (Principle Of Least Astonishment) when it comes to changes is awesome too.

Just to be controversial, macos. It's nothing fancy, just the arrs and Jellyfin running on an old MacBook air.

Wouldn't Linux be easier to manage and better in terms of performance?

Maybe, but I'm not a huge Linux user and every time I dip my toe in I run out of tinkering time. Plus I had the Air laying around and it all installed so easily.

I was just curious. Use whatever works best for you.

All good! The laptop I was using Linux on I installed batocera on instead 😊

Currently I am using Arch Linux. I am in the process of switching to NixOS.

1 more...

Depends on what you want to do with it. But for most things Debian or Fedora (Server edition) work fine.

i hear bad things about it, how does fedora server compare to debian?

It's much more up to date and in my experience works fine.

I too proxy my moxies, but run various OSes within them (via VMs or containers).

My DIY NAS runs Arch

  • LTS kernel
  • BTRFS snapshots on root fs
  • 4 drive NVMe array using ZFS raidz1
  • podman for my docker containers

It's been working fantastically so far.

I'm currently running something similar, what services are you running and do you use anything in addition to podman to manage your container (cockpit, systemd-units or similar)?

I run Nextcloud and two Jellyfin instances behind Nginx Proxy Manager. I also run a Palworld server. All of them are running under podman. I do use cockpit for checking container status, logs, and viewing the console for each container. I also use docker-compose to create all of my containers (using podman-docker of course). Unfortunately, all of them are running rootful instead of rootless, mostly because most proxies require root and setting things up for rootless like enabling low ports for regular users and allowing processes to run after logout are a pain in the ass.

Hyper-V / ESXi for host. Mostly windows with some Ubuntu server.

Ubuntu LTS, but in the process of replacing it with Debian

What benefit do you expect to get from this switch? Just wondering why there are so many Debian over Ubuntu in this thread

  • There has been some technical decisions over the last few years that I don't think fit my needs terribly well; chief of these is the push for Snaps - they are a proprietary distribution format, that adds significant overhead without any real benefit, and Canonical has been pushing more and more functionality into Snap
  • I previously chose Ubuntu over Debian because I needed more up to date versions of things like Python and PHP, with Docker this isn't really a concern any more, so slower, more conservative approach Debian takes isn't as big of an issue

Proxmox VE with Alpine Linux guests

lxcs are just great. love alpine on proxmox.

Actually, most of the guests are VMs (instead of LXCs) because many services I host are most easily deployed via Docker Compose and Docker in LXCs requires workarounds I don't fully understand thr implications of.

Heck really? I hadn't considered maybe that was some of my issues.

On proxmox I was able to get a docker compose Ark survival evolved server setup on an Ubuntu lxc without much fuss, a few other docker compose things worked okay also, once I figured out that compose isn't going to create the volumes for me.

Otherwise they've all been too deep for me or require other containers to run which I lose motivation on.

i use lcx alpine 3.16 and all i do is add docker docker-cli ...super fast. no hassle.

I went for a much simpler approach lately as I downscaled my hardware for efficiency.

I run NixOS on the bare metal. It gives the system management a declarative approach, just like kubernetes would. On top of that, I run libvirt as a hypervisor. In other scenarios I'd use tinyvmm and cloud-hypervisor, but I found qemu way better for the variety of homelab workloads and libvirt is pretty straightforward.

Some vms have pci passthrough, e.g. my routeros vm gets a bunch of NICs directly, some have various funny network topology. Libvirt used to be a pain in that regard, but it's actually fine with NixOS because you manage both sides of the networking stack in declarative configuration.

I run NixOS on the vms too (now for the sake of easy upgrades), and I have a bit of a split between running services natively (systemd is very good about “containerizing” things nowadays) and using docker (mostly because of laziness, e.g. Elastiflow was easier to deploy this way). Finally, I have a single dokerized Ubuntu that's more like a VM (as in, I never had a dockerfile for it, it's fully stateful) running the matter home automaton bits because I gave up on properly containing the matter python stack and went for an easy way out.

Now, a word about alternatives.

I used to run Ubuntu. No more. Upgrading the OS is always a huge pain even if everything is in docker. I want my OS to be managed in a config file and be able to easily roll back to the previous state. I used to run k3s, but even though it is much thinner than k8s, it is still very much ram hungry and I just don’t want to pay for that. Besides, complex networking is often non-trivial due to how its networking works, and multus is a world of pain. I used to run different hypervisors for the VMs (kubevirt, tinyvmm, a bunch others). I went way back to libvirt mostly because it’s straightforward in tuning very specific qemu bits I cared for in the homelab. I have some cpu overprovisioning, so I want to make my quotas set up extremely precisely, sacrificing the right workloads.

I've got a homemade NAS running unRAID and my arr suite/Jellyfin/qbittorrent, and an orangepi running the orangepiOS (flavor of Ubuntu I think?) Which handles home assistant and associated containers .

Freebsd, but it would be openbsd if it had a better filesystem.

Ubuntu server, I want to switch to debian but I don't know if it'll be worth it

I'm running to servers as hosts for docker. One with Ubuntu and one with Debian. So far I haven't noticed a meaningful difference

That's the boat I'm in, I swapped my laptop from kubuntu to Debian which is solid for me. Server has a lot setup on it that I could move but for now Ubuntu server works, not really feeling the push to change.

ArchlinuxARM, however it does not matter, given that everything I actually run, runs within Docker

I use Proxmox, running a mix of regular and NixOS based LXCs. One of those also runs Docker for simpler services.

TrueNAS Core as main OS and a few jails for the services I run on the machine.

Currently Ubuntu server 22.04

Previous was ESXi

Next one will be Proxmox

OpenMediaVault (Debian), it's the perfect amount of tinkering and simplicity for my first nas/homeserver

That was my first NAS OS, too! It's a gateway drug 😆

FreeBSD & 9Front. BSD for my media server and 9front for my web server.

On what hardware do you run 9front?

I have a VM on my FreeBSD desktop with Bhyve. I have 2 ThinkPads, an x61t & x201t that run it. I also have a Raspberry PI3B+ that runs it, and is my web server.

My 3 hosts all run Proxmox. Publicly available services run in VMs, usually running Ubuntu. Private services are usually Docker containers connected directly to my TailScale network running directly on the host.

Ubuntu normal release running Docker containers.

Had various issues with Debian Bookworm, not being able to install the “server” meta package on one server which left me without all the basics but “apt” and issues with lost IPv6 connections that made me switch to Ubuntu

Right now Debian, but I'll migrate to NixOS pretty soon since it's already running on all of my machines except this server.

I am running Ubuntu server and I am... satisfied with it. It does what it should, no problems, nothing to worry about, stable AF (as any mature distro?). But lately I am thinking about switching to fedora server (I need to reset my system one way or another, because my space on the hard drive for the system ran out of space (it was a small drive)). I am using fedora on my work machine and I really like it, so I thought I could give fedora on my server a try.

FreeBSD as main OS with jails and one bhyve vm for docker stuff. And ubuntu server on a dell optiplex micro for jellyfin with hardware transcoding

Gentoo!

I came back to Gentoo after years of Kubuntu. Once they forced snap down our throats and started pulling other weird crap I knew it was time to make a change. I came back to Gentoo and it's been pretty great. Still a few things to iron out on my laptop installs, but it's great for my home server.

Proxmox w/Debian, TrueNAS Scale, and Home Assistant VMs w/(usually Alpine) Docker containers in some VMs

Ubuntu, but I'm very strongly considering switching out to Debian or Rocky. Ubuntu has a lot of really unnecessary cruft that I think I'd be better off without.

I use Fedora on my laptop now, so going the RHEL/Rocky/Alma route for my servers is really tempting. Especially as I'm also considering switching to Podman.

Proxmox with latest Debian i guess, but all virtual machines are on Ubuntu 23.10. It's just the easiest to install things on, and have a semi-recent kernel, even though I would never use it on the desktop.

I actually wanted to run arch in the vms but I couldn't find any image with the latest arch. The latest I found was with kernel 6.3.

Windows Hyper-V Server on the host with most of the VM's split between Ubuntu and Debian. I also have two Windows VM's that I keep out of necessity.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
DNS Domain Name Service/System
ESXi VMWare virtual machine hypervisor
Git Popular version control system, primarily for code
HTTP Hypertext Transfer Protocol, the Web
LTS Long Term Support software version
LXC Linux Containers
NAS Network-Attached Storage
NVMe Non-Volatile Memory Express interface for mass storage
RAID Redundant Array of Independent Disks for mass storage
RPi Raspberry Pi brand of SBC
SBC Single-Board Computer
SMTP Simple Mail Transfer Protocol
VPS Virtual Private Server (opposed to shared hosting)
ZFS Solaris/Linux filesystem focusing on data integrity
k8s Kubernetes container management package
nginx Popular HTTP server

15 acronyms in this thread; the most compressed thread commented on today has 10 acronyms.

[Thread #478 for this sub, first seen 2nd Feb 2024, 19:45] [FAQ] [Full list] [Contact] [Source code]

I have 4 home servers. 1 running pfsense, 1 running truenas, 1 running proxmox, and 1 is a cloud key gen2 for unifi that I got for free

One with Arch that runs gameservers for my friends, and another that runs Proxmox filled with either Arch or Debian in the containers depending on what it is in them.

Proxmox on physical servers hosting a variety of vanilla Debian installations. I have a physical router running pfsense as well as two HP miniservers running OpenMediaVault.

Ubuntu LTS because that's what I was most comfortable with at the time, now I'd really like to switch over to Debian but I'm not sure I can be bothered until I really have to, everything is working well at the moment. It's running in Proxmox.

A few actually.

Rocky Linux 9, XCP-NG, Windows server 2022 datacenter

Different usecasses.

My setup consists of the following:

Unraid, most services I self host run in docker here. Things like plex/jellyfin, nextcloud, unifi could controller.

Proxmox, used to virtualize my pfsense after I moved away from my unifi USG router. A few Linux and Debian headless virtual machines run here as well. Had pihole virtualized here as well but switched over to pfBlockerNG to consolidate.

TrueNAS, all my media shares. I also sync my desktop environments here to have a consistent windows desktop across my desktops and laptops.

Home assistant running on home assistant yellow. Runs a few add-on services.

I have several servers I’ve acquired over the years setup in a proxmox cluster

Proxmox, Openmediavault, Ubuntu Server. Mostly because I'm lazy and I grew up using Ubuntu.

Ubuntu and WinServer2019 vms split over a proxmox-machine and server2019-hyperv-machine

2 each for replication.

Arch which is great, but I want to move to FreeBSD or Proxmox for security reasons

Gentoo always and for everything!

How stable is it for you? Did you compile everything from source for your server? What other benefits do you see with Gentoo?

I'm considering Alpine

Stable? I never had instability of any kind with gentoo like... Ever... Except for faulty hardware (dead USB Ethernet card, bad memory stick...).

Gentoo let's you build the most skinny and bloatless server you can, just what you need how you want it. No bullshit, no choices made by others (Ubuntu I look at you). And optimized for your hardware too.

Today's compile time is ridiculously small, so that's not a down point for Gentoo.

Also, its so damn adherent to the true Linux philosophy that its surprisingly logical and coherent in it's internal organization that doesn't get in your way.

Ah, and docker and podman support is piece of cake if you like that stuff.

Its even less bloated than a plain text-only fresh Debian install.

You don't even have a logger or a cron daemon by default unless YOU install it.

And there is so much great quality documentation that even navigated Linux people will learn new tricks installing Gentoo.

Check out the Gentoo Handbook online.

And I have more than once installed gentoo on another running Linux, then rebooted remotely to a fresh Gentoo. Do it with any other distro, I dare you!

  • XCP-ng on my virtualization host
  • TrueNAS Core on my NAS
  • Ubuntu Server on my Jellyfin media server
  • Debian on my other server that I use for testing purposes
  • DietPi on my Raspberry Pi

Ubuntu server w/ zfs on the main.

Have a little atom hp home server at my parents house that uses almost no power, that runs freebsd x86, zerotier one, and kopia. Free cloud backup 😁

  • Alpine on Pi4.

  • LMDE on recycled AMD systems (phenoms, opterons, FM2 APUs, oh and a recently dead bulldozer fx-8150).

  • TrueNAS, OPNsense on dedicated hardware.

  • VMware ESXi on my older workstations (currently transitioning toward LXD/Incus and XPG-ng).

You mean XCP-ng?

Yes, 😅. Thank you for letting me know.

I typed correctly I'm pretty sure, but typing it again now it autocorrects to "C - C - P" now 🫤. Even more confused.

I'll edit my original post.

I used unraid for a long while. I recently switched to opensuse microos for a better desktop experience, and it's been fantastic

Arch for stuff I have physical access to. Nothing's ever gone wrong, so it's worth it for the immediate updates and consistency with my other systems. For VPS I use Debian though, occasionally the unstable/Sid branch if I really need the latest updates. There are almost always Debian images available on a VPS.

TrueNAS SCALE as host with an Ubuntu LTS VM running Docker containers.

Original I went with only containers running on top of SCALE but both iX and TrueCharts made it harder to run plain Docker Compose on TrueNAS.

Rocky, but I'll probably give Debian a try next time I decide to nuke my install

I'm currently contemplating switching to Rocky. What makes you want to get away from it?

The OS experience is good, but I want to get away from RHEL/CentOS and clones due to Red Hat's decisions in recent years

3-Node ESXi cluster with 10 Debian VMs, 3 Windows VMs, and one FreeBSD VM

I use Linux Mint, mostly because I'm familiar with it. If I was to redo it, I would likely just run Debian.

XCP-ng hypervisor main box for my VMs, mostly Ubuntu Server but some Alma Linux VMs too. TrueNAS Core for my NAS box.

Might start switching my VMs from Ubuntu Server to Debian soon, we'll see.

Switched from Debian to Ubuntu LTS few years ago.

Any reason why?

I use Raspian on my pi and Ubuntu on my workstation and I maintain a debian server at work.

I love 'em all. Ubuntu Snaps arent my fav. but other than that they've been great

I had some problems caused by different package versions between main workstation (Ubuntu) and Debian. Deluge refused to connect in thin client mode because the version in Debian was older and incompatible with newer version in Ubuntu. I had a server upgrade coming up, reinstall was in order anyway, and I decided to just go with Ubuntu. Livepatch is nice as well, and although snaps aren't great, there is usecase for them as well sometimes.

TrueNAS scale with truecharts for my main server. I'm considering rolling out a compute only server with openSUSE microOS to leave trueNAS only for storage. I like openSUSE philosophy and microOS with cockpit seems solid on my tests. I might start with some old laptops and deploy the ldap server there to test it and remove that responsability from TrueNAS.

Fedora core os (FCOS) vms on XCP-NG with trueNas for persistent storage. With FCOS, vms configurations can stay version controlled and deployed using open Tofu (terraform) and butane/ignition.

Boxes that physically live in my home are mostly Manjaro. They’re also not externally accessible from the internet.

Anything in the cloud I standardize on Debian. Two distros and consistency makes maintenance much easier.

Anything in a container runs whatever it was built on because porting a docker compose file from, say, Alpine to anything else is just not worth the time and energy.

I tried to use fedora server or was it cloud? Idk but I tried fedora as a server and wanted to set up a VM but got confused. Storage pools scared me away. Will try to learn it when I have the time