My Overkill Home Network - Complete Details 2023

GiantPossum@lemmy.world to Selfhosted@lemmy.world – 518 points –
blog.networkprofile.org

Hopefully this is not too long! There has been a lot of changes since the last time I posted a full overview like this

112

This guy selfhosts.

Me: hey, I own that exact Anker USB power supply. I'm basically as pro as this guy!

Narrator: his old laptop and external hard drive set-up was not as impressive, even with the Anker USB power supply.

Funny you think I'm pro, I just like blinky lights!

Some people harness that desire into Christmas displays.

You made a mini-internet, complete with mini-mini-intenets!

Wow, you must be rich

I think the reason we aren't rich is because we do shit like this.

Anyway, I'm off to buy enough HDDs to get me through the end of the month.

Honestly its all cheaper than you think, 100% of it I bought used bar a few things, and over a long amount of time too. Plus messing with stuff like this has 100% helped me advance my career

You’ll be surprised how cheap some equipment goes for when a company runs out of business. Just sayin

I’ve been trying to hunt down cheap used network equipment lately. It’s a weird thing to be disappointed that there aren’t any failing businesses around me :(

I’m about to make an 8 hour round trip drive for a cheap server rack this coming weekend. Please send help.

You must throw sick LAN parties...

I love the fact that you have a favourite switch!

Question. I have a home network that's more advanced than your typical house. I started holding back though as I figured when I die my family won't have a clue about all the stuff I have setup. Do you guys ever think about this? I'd hate to leave behind a nightmare for my family members to remove and replace with a regular ISP provided router.

I’ve thought about it, and nobody will care about your/my elaborate setup after we are gone. It will just be replaced by a ISP router without regrets.

My opinion is that your spouse will have to get rid of any other hobby related stuff. If you're a fisherman, she's going to have to find something to do with all the tackle, boat/s, gear.

I know a guy that was a woodworker who had a shop full of well over $20k worth of tools. Poor guy got cancer and died, and his wife had to try to get rid of all of it. Luckily she had some of his woodworking friends who helped her price and sell the stuff. (I got a pretty nice used planer out of the deal)

Also thought about that a lot. The most important is that your people can access your data. My partner and bestie both have LUKS keys on all of my devices.

Maybe do a test run with them to see if they can actually access it.

Got it, sharing the password to my obscure furry midget porn collection with my people

I have Bitwarden set to give my wife access if she requests it and I don't respond in X days

Things generally "just work" so she would have access to everything, and she can figure out what she wants to do. All the passwords are there and all of the configs are fairly easy for stuff she cares about anyway

Wow. That's really an overkill.

Any idea what's the power consumption of all that hardware?

How many hours a month do you spend upgrading or maintaining the network and all other software?

Honestly, I'm not 100% sure. I don't have a way to monitor just the stuff in the rack as the UPS also powers a lot of other stuff in the house. Either way, I've worked to make everything fairly low power, or at least as low power as feasible. The things that use the most power is the disks

I can tell you its less than 800w though, as that's the lowest the UPS goes at night. But that also does include both me and my wifes desktops which stay on 24/7, and an Apple TV, and standby power for all devices etc

It is beyond me that you dont get symetrical connection with fiber in the US.

I am in the US and I do... In fact, I can upgrade to symmetrical 5Gb now with AT&T

What does your current connection cost per month? I get my 500/100 fiber next month for 59€/month. 1000/200 would be 79€, and that would be the fastest you can get :-/

I'm in Colorado and pay $49.95 for 1000/1000 (though i'm grandfathered in and i think it's $69.95 for new users). There's another ISP that offers the same at $70, or i can get 1200/35 cable for about $60.

I can get 2500/2500 for $149 and 10000/10000 for $249 (from my municipal provider) or I can get 6000/6000 for $300 (from the cable provider).

$70/mo for the AT&T Fiber, and $50/mo for the Verizon 5G

200 up isn't too bad, nothing to really cry over. My old connection at my last place was 1000/30! What a joke

Australian here, I get 1000 / 50.. don't ask what it costs to upgrade that upload speed.

Can I ask why you pay for that speed? I'm on 25/10 and I've never felt it was the 25mb download cap that is holding me back.

I was pretty happy with 100/40 but I have 8 TV's often all streaming 4k hdr from various services at once, along with my constant downloads so i had fiber installed and went to 1000, I'm pretty happy now.

I also sync ser ers between home and work so not having to severely throttle that is nice, upstream bandwidth is still awful though.

ahh thats fair, there's never really more than one stream at a time here.

1 more...

Great job on the cabling and the setup! As an Apartment dweller, I hope you don't mind my living vicariously through your setup!

I've been there! Such a hassle. It was great when I moved and was finally able to do what I wanted

You said complete details... So where's your private ssh key and public IP address?

Cool setup btw. Would love to get my hands on such a system.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
DNS Domain Name Service/System
ESXi VMWare virtual machine hypervisor
HTTP Hypertext Transfer Protocol, the Web
HTTPS HTTP over SSL
IP Internet Protocol
NAS Network-Attached Storage
NVMe Non-Volatile Memory Express interface for mass storage
PiHole Network-wide ad-blocker (DNS sinkhole)
SSL Secure Sockets Layer, for transparent encryption
VPN Virtual Private Network
Jargon Definition
Raptor Methane-fueled rocket engine under development by SpaceX

9 acronyms in this thread; the most compressed thread commented on today has 20 acronyms.

[Thread #21 for this sub, first seen 11th Aug 2023, 00:35] [FAQ] [Full list] [Contact] [Source code]

Solid writeup. Good looking setup. I like how you have a great reason for every decision you made.

Crazy overkill for almost everyone, but you're living in the future!

Yeah but can it host PiHole?

It sure can, but so far I've not found much use for it. I set it up to see if it can block YouTube ads in the mobile app, but it can't. Since I already use uBlock Origin, I don't know what I gain

At least from my experience, with a proper blacklist it shuts down a ton more stuff. Not just pure ads, but a ton of tracking and websites/apps phoning home too. You can configure it to be as strict or lenient as you'd like, basically. For me it's nice, because I can just apply it to the entire network, and I don't have to worry about trying to explain how this works to my family

Maybe I'll give it a go again, after all it does have a really nice slick WebUI

Also has the benefit of being a completely local DNS server for all your devices to use. I think you are also able to add custom entries if you wanted to be able to refer to your devices using dns. It also has some caching benefits so there are less DNS requests going out of your home network.

Personally I set up AdGuard Home because it has DNS over HTTPS support out of the box, which means your ISP cannot see your DNS requests. Pihole supports this too, but it requires additional setup.

Check out the Star Trek theme for PiHole! It's one of the default options.

Crazy awesome setup! I noticed you had an enphase inverter next to your electrical meter, I assume for solar panels. Would you mind giving details about that system? What size array do you have and how efficient has it been? How are you monitoring the solar systems output?

Hi OP. If you're reading this, I have a few questions:

  1. You're using the Linode box as the server, on which you forward ports for your services. Am I to assume that you somehow access your homelab via your VPN using the Linode box too? Usually people would access their lab at home directly.
  2. Wouldn't a whitebox build for your NAS save power?
  3. What are you using both switches for? Are you running out of ports?
  4. Since you're running VMWare, are you running VMs for every service? Why not containers?
  5. Even if most of the content on your blog is static, how are you hosting it for it to load so quickly? Are you using some sort of CDN in front of your Linode box to cache the static assets like pictures?

It was great reading about your lab. I'll try and follow your blog on RSS if you have a feed. Thanks.

  1. You’re using the Linode box as the server, on which you forward ports for your services. Am I to assume that you somehow access your homelab via your VPN using the Linode box too? Usually people would access their lab at home directly.

Yes, I also access the lab via the Linode box. I do however have direct VPN access too. The reason for using the Linode box is that for some reason, the speed and latency via the Linode box is far better that directly in. I can only assume its some kind of peering thing. I always connect in via my phone on T-Mobile, so perhaps the connection between T-Mobile and Linode, and the connection between AT&T and Linode, is better than T-Mobile to AT&T Residential? Unsure, all I know is that it works 100x better. And it also means I don't need 2 different connections for the primary and secondary WAN, I can just connected to Linode and it will connect over whatever connection is active

  1. Wouldn’t a whitebox build for your NAS save power?

This really is a whitebox build, it uses very little power. The disks use the most amount of power, which there is no getting around

  1. What are you using both switches for? Are you running out of ports?

The 1Gb switches? yes, I ran out of ports on the Dell, or am very, very close

  1. Since you’re running VMWare, are you running VMs for every service? Why not containers?

Everything that can run in containers already is, on Debian VM's within ESXi

  1. Even if most of the content on your blog is static, how are you hosting it for it to load so quickly? Are you using some sort of CDN in front of your Linode box to cache the static assets like pictures?

I am using CloudFlare in front of it, so that's probably why. But even directly its pretty quick. I guess NVMe storage and decent internet means its fast?

Thanks!

How do you relay your VPN connection over your Linode box? I can understand a direct VPN connection, but I can't understand the networking behind relaying the VPN connection around the Linode box.

Ah, yes CloudFlare is a great proxy/CDN. Thanks

Not OP, but also curious about number 5, I noticed that blog article loaded lickety-split!

Good to hear! I replied above about it, here was my reply

I am using CloudFlare in front of it, so that’s probably why. But even directly its pretty quick. I guess NVMe storage and decent internet means its fast?

Gah, treasure trove of info. Thank you for sharing! How's the garage rack holding up? I'm so tempted to put some servers in my garage but the heat can get excessive.

Zero problems, often times stuff in the house is actually hotter than stuff in the garage funnily enough, even in summer

Fucking amazing writeup, I haven't read it all yet but from what I read there's a lot of good information and inspiration

You. I like you.

Honestly amazing setup. It's more robust than some industrial applications I've seen.

Thanks for such a great write-up. I'll definitely be referring back to it as I upgrade my homelab.

Cheers!

I don't know if you've mentioned this somewhere, but what's the purpose of the GPS on a home server?

Might be for time synchronization in order to not have to rely on a public NTP server.

What is the advantage of self hosting an NTP server?

After all, it just tells time

After all, it just tells time

It's for making your computers clocks be very, very close to each other. Not milliseconds close, but nano seconds. That is more important than one might think, especially for networks.

I'm going to have something similar at home at some point, just need to make a few more cable runs so one GPS can see the sky (= more accurate)

  1. Its cool. Imagine being able to get data from 12 satellites at the same time to get super accurate time, with a $10 GPS board. What a time to be alive!
  2. I'm trying to reduce the amount of stuff I'm relying on the internet for. Time is pretty important, and having a local server solves all that.

Part of me looks at this man as a system God among us puny and unworthy users.

The other part is saying just keep paying for one drive and Netflix. As the saying goes "the first step to powerbill and uptime hell is the simple plex server"

Just wait until they remove your favorite show and jack up the price

Oh I'm already testing plex out on the desktop with a few of the *arr services. It's very neat and satisfying but with power hitting $0.50aud/kW i need to be realistic about what hardware I want to run as investing too heavily early on will take years to pay off.

Still I'm very jealous of what you've got going on.

That is definitely overkill, so you must be an enthusiast.

That's OK. I have a old Dell Poweredge that I use for simply torrenting and backing up everything I ever torrented.

Great setup! but is the thumbnail showing as George Takei for anyone else?

What the heck! Not for me. Got a screenshot?

Sweet setup, what's the reason for the extreme attention to backup power? is the grid really that unreliable where you are?

I've had one unplanned power outage in the last 10 years so it seems like a crazy amount of backup to me.

The grid both is and isn't unreliable. I've not had many random outages, but I have had 2 x day long outages on hot Friday's when they were replacing power poles, which the generator of course kept me through. Working from home, and being in the Texas heat, that would be bad. And I like many people now have really bad power outage PTSD after the 2021 Texas Freeze where we all lost power. I'll never let that happen again! And turning everything off is such a hassle, I want it to to all stay on no matter what. Since I work from home, that adds another layer too. Plus, I just like cool things

Yeah that seems kinda crazy to me too. I've lived in my current house for 8 years and the only time the power has gone out was when a vehicle crashed into one of the distribution boxes by the road. Our power and internet come from the same provider so it was a double whammy for several hours.

But I suppose it depends where you are - i worked at a place that had two independent power feeds from two different cities, massive UPSs to run the datacenter for 10 minutes and then two redundant diesel generators with several months of fuel on site. I still saw that go down twice in my time there.

I've had my power go out around 10 times this year. Gotta love that TX power grid

When we lived in Missouri the power went out twice ish a year, sometimes more. Amazing how fragile that stuff is when nobody invests in the infrastructure for the town.

I have no way to prove it, but I'm 99.9% sure that CenterPoint Energy (Who services this grid around here) leave stuff on its last legs so it gets damaged in a hurricane, and they can claim that juicy FEMA money to fix it.

Round here it's all government run. The city runs power, water, sewer, phone, internet, trash/recycle/compost.

We've got the second fastest internet in the country (and it's free for low income people), our power gets an American Public Power Diamond rating for reliability, we're (mostly) on track for being 100% renewable power by 2030, the city captures and liquifies the methane from the sewage treatment process and uses it to run the garbage trucks (that say "Powered by You" on them) and our rates for all of that are cheaper than commercial providers.

Amazingly we still run into people who live here, know all that and still believe that the government is incapable of running anything well... it's kind of startling.

Still, that makes a bit more sense for why you have a generator and that then pretty much requires you have a UPS - so i get it.

Like the idea of multi-room UPS. Question, once the UPS battery run out during a power outage, is there any other type of power generation (Solar, Propane or gasoline) as a backup (aware of the servers will consume more watts than it can generate)?

Ops got a diesel backup generator, says it takes 10 min to kick in if there's a power outage.

its a Natural Gas generator, and it actually takes 10 SECONDS!

Yes, I have a 27kw Natural Gas standby generator with an ATS. It takes 10 seconds from power failure, to it switching to generator power. So, the UPS just bridges that gap

In the extremely unlikely even the natural gas goes out, I have a 7.2kw Tri-fuel portable generator (Gasoline, NG and Propane) and I keep around 80 gallons of gasoline on hand, and I have an inlet and interlock on the main breaker, so I can switch to that if needed

Loved the UPS article itself. If you wanted to level it up one more time, you could do something like this: https://hackaday.com/2023/07/31/automatic-transfer-switch-keeps-internet-online/

It is a automatic transfer switch, so that in the case of a UPS failure, the power can be transferred to a wall outlet fast enough that you shouldn't experience an outage.

Yeah that's on my to do list, I'm looking for some ATS PDU's for cheap, like the CyberPower PDU20MHVT10AT