Looking for good resources for a selfhosted home-server, especially the more advanced stuff

Wow. Thanks. Guess the "your previous linux knowledge doesn't really apply to NixOS" is correct. I already found the lengthy lists of stuff to read up on... I'll add this to my "read later" list :-D

Stalwart sounds nice, too. Since I'm just setting everything up, maybe I can try both mailservers. I've now had 2 people recommend the simple nixos-mailserver.

i guess those who say you should not run your own mail server are maybe those who run one, earning money with it?

I am running my own mail server for >15 years now. What you should do:

• check your ip and the subnet your server is in is not found in blacklists (mxtoolbox has blacklist check)
• make sure you do not run an open relay server, thus only allow authenticated users to send email, use good passwords and youre likely done
• stay up to date and read security notes for your server, especially whatever you run as exposed service, register on security news etc.
• do not use software that is known to make trouble (no M$, maybe better also avoid microsofts systemd)
• setup your config to match exactly what you need, disable unneeded features (like if you use cram-md5 for auth, disable plain and all others)
• send mails via deliver port and only receive them via port 25.
• setup dkim
• setup dmarc
• check DNS to be precisely how it should be, MX record to match PTR, correct dmarc and dkim settings, setup spf records for your domain and *.yourdomain.tld too (using txt records this could collide with letsencrypt certupdates via dns)
• use mailserver check services i.e. mxtoolbox but you will need others to also check dmarc and dkim (services where you can send an email to and they tell you problems)
• use publicly validatrable ssl certificates (letsencrypt) so other servers are not "scared" to send emails using tls instead of plaintext. (disable plaintext transfer anyway) update the certificates regulary before they expire. you can use ssl checkers (ssl labs) to validate your certificate
• verify sending and receiving using some external email accounts from other providers (google microsoft, a small hoster is good too, but for big ones checking with their system sometimes shines light on their bad services ;-)) and keep in mind that the big ones are not doing everything right or sane.
• do not send spam or mass mailings (even if this is your business, please stop, get a good job instead, if you know how to sneak emails through filters, maybe someone pays you to do the opposite)
• regulary check your server logs for weird things
• maybe use vpn to access imap and deliver port do reduce the exposed services to minimum
• disallow ipv6 to all providers that do not allow sending to them via ipv6 (like google)
• use fail2ban to block abusers in the firewall (less for security but for keeping logs cleaner) sometimes you need to block others misconfigured servers forever (like if one server tries to send an email to your server for a domain that you dont host, but this email is one of an autogenerated error sort that just sits in the logs every few minutes for as long as you let it. be aware that fail2ban blocks ip addresses while with ipv6 spammers like all others have billions of them in their range.
• be aware that some providers do weird checks "before" trying to send email to you like deutsche telekom, checking some html page on your domain to show a postal address before they try sending an email to you. these basically betray their customers. depending on how important this one provider is for you, you can do what they want (the postal address telekom checks could be set to the name of their CEO to actually work lol) but you do not have to fulfill every wish other platforms would like to force you into, regulary those who want so send emails to you, will deliver them for their customers (not deutsche telekom though)
• be aware thar your emails that you send to other servers could end up in spamfilters, no matter what you do, spam filters are error prone and CEOs tend to hire less than a quarter of how many admins they need to not betray their customers with ads vs reality, but (!) mostly the receiver (companies) "wants" to get your email thus checks spam folders anyway or you could be added to contacts (friends). if your email is lost completely (microsoft cloud services tended to do so for years and during that time even turned off sending DSNs which i had used to proof the regular email loss to the M$ enthusiasts muahahaa) and if that email is important, you have to "ensure" checking its delivery anyway maybe also send via postal services. thus normally spam filters - as of my feeling for this - are not really a problem, google is a problem (reported emails as received, then always deleted them directly without notice to neither sender nor receiver, betraying their customers while breaking some laws too), microsoft is a problem and some other providers too, but that has exactly nothing to do with your domain or your email server, you are not responsible for their (intentional) errors or crimes.
• regulary repeat your checks especially for blacklists - some blacklists show your ip as blacklisted because another ip that "looks similar" - same provider/network - is found in another blacklist (they dont like you cause you live in the same city as that other guy wich they were told was bad - weird though but some blacklist should be blacklisted.. ) but has nothing to do with you, just a blacklist that is f***ed up. depending on what is the problem, your hoster can help, or you can choose another (cheaper or better) hoster anyway.
• setup monitoring for mail roundtrip using a mailaccount at another provider and there a forwarding back to your server so you get an alarm if something breaks.
• filter incoming emails by SPF (reject before they are received, but do not filter by bad dmarc setup of other providers (the CEO problem mentioned above, even most newsletters i receive show to have a broken email setup even if "professional" paid services are used) . setup all your filters with relaxed settings (warning only) first before you enforce them
• consiser firewall blocking on port probes: every ip who probes ports where you have no services running on, could imho get immediately blocked with packet drop immediately forel 24h just to keep logs a bit cleaner ;-)
• setup your email server with at least two nodes on different hosters in different regions so that any problem local to a hoster or region does not affect both and you stay online with sending and receiving. (hosting your own DNS is even more reliable)

never under no circumstances? sure, of course yes! but it can be some work to do. but if you do, its in my experience more stable than any provider, paid or unpaid, cloud or not, and you get the most possible privacy (all of your non-internal-only emails are available to at least one other server anyway) and flexibility too, and you have the possibility to proof that the other server lost your email, not yours as they like to just blindly claim by default =D

Thank you. That one already appeared in my search results. Seems like a traditional postfix/dovecot/rspamd setup.

I know. People always say you shouldn't run your own mailserver. I have. For like 10 years or so and I'm fine. Well... I'm more than fine. I really like the idea that my mailbox is stored on an encrypted volume at home and not somewhere in the cloud. Also it comes without any noticeable capacity limit, I got a large harddisk in my NAS/server. It's a bit annoying, though. I forward some email. And some of that goes to a gmail.com account of a friend. It's important first contact and admin mail, so the spam filter isn't super strict. And I got greylisted by gmail for doing that. Once a few spam mails slip through, google will stop talking to you. So I currently can't send mail to gmail users. I have a few free email accounts I use as a relay so I myself can still send mails. But it's annoying. It's part of the reason why I now want to redo my mailserver and have an updated rspamd and stuff. I get why people say you should let other people provide email service to you, but I don't see a proper reason except for it's annoying and frustrating and odds are against you. (Additionally it is a bit complicated to set up reverse pointers and MX records.)