Just how secure are the various reverse proxy options?

TedZanzibar@feddit.uk to Selfhosted@lemmy.world – 59 points –

Specifically from the standpoint of protecting against common and not-so-common exploits.

I understand the concept of a reverse proxy and how works on the surface level, but do any of the common recommendations (npm, caddy, traefik) actually do anything worthwhile to protect against exploit probes and/or active attacks?

Npm has a "block common exploits" option but I can't find anything about what that actually does, caddy has a module to add crowdsec support which looks like it could be promising but I haven't wrapped my head around it yet, and traefik looks like a massive pain to get going in the first place!

Meanwhile Bunkerweb actually looks like it's been built with robust protections out of the box, but seems like it's just as complicated as traefik to setup, and DNS based Let's Encrypt requires a pro subscription so that's a no-go for me anyway.

Would love to hear people's thoughts on the matter and what you're doing to adequately secure your setup.

Edit: Thanks for all of your informative replies, everyone. I read them all and replied to as many as I could! In the end I've managed to get npm working with crowdsec, and once I get cloudflare to include the source IP with the requests I think I'll be happy enough with that solution.

24

LE certs can always be "side loaded" by acme.sh or LEbot or whatever, and the reverse proxy restarted to use the new certs. So, the whole "pro subscription to use specific certs" shouldn't be a factor, except a little more work/config (so, money Vs time).

Now for my opinion...

For base security, all it's doing is looking at whatever you tell it to look at in an http request and forward/drop/block as such.
HAProxy is well battle-tested. Nginx is well battle-tested. Traefik and caddy are comparably newer contenders, but considering their adoption they are probably well battle-tested.
Which means, an established reverse proxy is only going to be as secure as the software it's forwarding traffic to.

If there happens to be some mental TLS handshake RCE that comes up, chances are they are all using the same underlying TLS library so all will be susceptible...
But at least an attacker only gets access to the reverse proxy server. Which is why it's worth having that in a locked down isolated VM, ideally built in a way that is extremely easy to rebuild (declarative configs like docker-compose and some scripts, or even something like nixos for an immutable OS).

As for add-ons... Most WAFs only look for things like XSS injection or SQL injection or exploitative HTTP request formats. Very very basic attack vectors that any decent HTTP stack and reasonably built software shouldn't have to even worry.
Any DDOS protection is more likely to blast your network connectivity, which (for self hosting) a WAF isn't going to be able to do anything about.
I'm not sure how good they actually are against a DOS attack that is caused by bugs/inefficiencies in the application. Maybe they monitor for long/increasing response times, and block further requests to them? Might cause a lot of false-positives for your users.

So, the only real benefit - that I see - are zero-day exploit protections.... and that only matters if they are built around near-realtime updates like crowdsec is. I don't know how it compares to cloudflares WAF, tho.
Any zero-day protection that isn't being managed and updated in near-realtime is about as effective as you monitoring news of your installed services/programmes and updating them regularly. Because you are likely to update your WAF and apps when you hear about those, or regular scheduled updates will deal with them before you even learn about them.

I guess there is security in layers, and if layers of security is more important than CPU consumption/response time/requests per second (ie have an abundance of processing, servicing few users, etc) then it might be a no-brainer.

The only other time I can see a generic WAF being useful is if you have rolled your own framework and HTTP stack, and are running your own software. Because, you won't get that right... So might as well have the extra protection of a WAF.

Or, I guess, with really old unsupported software.
But surely there is a newer take or fork of it?

There is also the "am I worth it" factor.
Like, what is your actual threat model?
Defend against the usual script-based attacks (IE low hanging fruit), only expose/forward ports that are actually required, use some sensible security that isolates more vulnerable systems (IE a proxy) from more sensitive (ie a database or storage), and update regularly on stable/lts branches.

Edit:
I just googled bunkerweb.
First we had firewalls. Then we got web application firewalls. Along came next generation firewalls. Now we have Next Generation Web Application Firewalls with paid features like "Pay per protected services" and "Best effort support included"

Maybe I'm just salty

That got a bit long.
Reading more into bunkerweb.

Things like the "limit" feature are going to doink people on cgnat or large corporate networks. I've had security stuff tripped by a company using my software, and it's a PITA cause all the requests from legit users come from only a few IP addresses.

Antibot isn't going to be helpful for things like JS requests, because cookies aren't included by default with fetch requests - so the application needs to be specifically built for this (at which point, do it at an application level so it can scale easier?).
And captcha. For whatever that is worth these days.

Reverse Scan is going to slow down every request (as it scans the remote client for suspicious open ports, so a 500ms delay as default).

Country is just geo-ip.

Bad Behaviour is just rate limiting (although with a 24h ban). Sucks if a few corporate/cgnat users all hit a 404 and suddenly that entire company/ISP's IP is blocked for a day.

This seems like something to use when running a TOR server or something, where security is more important than user experience. Like, every feature seems to punish legit users

If there happens to be some mental TLS handshake RCE that comes up, chances are they are all using the same underlying TLS library so all will be susceptible…

Among common reverse proxies, I know of at least two underlying TLS stacks being used:

  • Nginx uses OpenSSL.
    • This is probably the one you thought everyone was using, as it's essentially considered to be the "default" TLS stack.
  • Caddy uses crypto/tls from the Go standard library (which has its own implementation, it's not just a wrapper around OpenSSL).
    • This is in all likelihood also the case for Traefik (and any other Go-based reverse proxies), though I did not check.

Very useful insights, thanks.

I do currently have external stuff running via a Cloudflare tunnel (which is why I need DNS based LE certs for the internal proxy) but I don't know if it's setup correctly (beyond doing basic reverse proxying) and the admin backend for it feels like massive overkill for a home setup. Plus with Immich I run into the issue of a) dire warnings about it being in active dev and potentially insecure and b) filesize limits making away-from-home backups difficult.

I could well be over thinking the whole thing.

As far as I know you want a web application firewall to block attacks. A reverse proxy is just to proxy requests and doesn't necessarily care if it forwards legitimate traffic or attacks.

What you probably want is a dmz or red/green localnets. A reverse proxy (as others have mentioned) like haproxy or nginx) are extremely unlikely to, themselves, be hacked. But they don't really add security, either.

What does add security is to have a router with a firewall, with one or more red networks, and a green network.

The red network has all of your public-facing servers. They have virtually no external access, and no internal access except to respond. It's even good to have a rule on the router that you can turn on/off that blocks all outbound connections from the red network to the external world. To upgrade a server, turn off the rule, upgrade, and then turn the rule on again. The router only forwards inbound connections from the internet on a specific port, and routes them to the server/servers on the red network(s) on a (possibly different) specific port.

Most ownage-style hacks involve (once compromised) either calling home (can't if the server is not allowed outbound connections) or opening an additional port (who cares, the router will never forward anything to that port).

Then, back up your important info, and keep multiple copies of that info - daily for a week, monthly for a few months, and yearly.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
DHCP Dynamic Host Configuration Protocol, automates assignment of IPs when connecting to a network
DNS Domain Name Service/System
HTTP Hypertext Transfer Protocol, the Web
IP Internet Protocol
SSL Secure Sockets Layer, for transparent encryption
TLS Transport Layer Security, supersedes SSL
VPN Virtual Private Network
nginx Popular HTTP server

7 acronyms in this thread; the most compressed thread commented on today has 12 acronyms.

[Thread #756 for this sub, first seen 20th May 2024, 19:45] [FAQ] [Full list] [Contact] [Source code]

1 more...

To me on the security side of things caddy has a feature I have yet to see anywhere else: default reverse proxy headers.

Got something you want to lock down remote js loading on unless it explicitly requests an override? Default the variable to a locked value. The application can override it with it's own header as necessary.

https://caddyserver.com/docs/caddyfile/directives/header

I think this part should be better left off to the actual tool that does a more promising job on security, like a firewall.

I have yet to see anything I can run myself that works as well as cloudflare. Still, not exactly self hosted.

Yeah I'm running a Cloudflare tunnel for external access (which is why I need DNS based LE certs), but that's another thing that I don't really know what it's doing beyond basic reverse proxying.

I have a country-based whitelist for where my Immich instance can be accessed from but I find the Zero Trust admin backend to be massive overkill for my needs, and it doesn't help that they've recently moved everything around so none of the guides out there point to the right places anymore!

Btw, you can do http based LE certificates through cloudflare. They just proxy the ACME URLs.

Hmm I must be doing something wrong then because it doesn't work for me.

So, is public accessibility actually required?
Does it need to be exposed to the public internet?

Why not use wireguard (or another VPN)? Even easier is tailscale.
If you are hand selecting users (IE, doesn't actually need to be publicly accessible), then VPN is the most secure and just run a reverse proxy for ease & certs.
Or set up client certificate authentication, so only users that install a certificate issued by you can connect to the service (dunno how that works for 3rd party apps to immich)

Like I asked, what is your actual threat model?
What are your requirements?
Is public accessibility actually required?

If it was just me, or if Tailscale wasn't such an insatiable battery leech then I'd absolutely do that but the wife (and kids) acceptance factor plays a big role, and they're never going to accept having to toggle a separate service on and off to get to their photos.

Maybe I'm being overly paranoid but I work in IT and see the daily, near constant barrage of port scans and login attempts to our VPN service and it has an effect!

I use WireGaurd, it’s set to on demand for any network or cellular data (so effectively always on), no DNS records (I just use public DNS providing private range IP addresses). It doesn’t make any sort of dent in my battery life. Also, only the wiregaurd network traffic is routed through it, so if my server is down the phone/laptop’s internet continues to work. I borrowed my wife’s phone and laptop for 15 minutes to set it up, and now no one has to think about it.

Thanks for the suggestion. I spent a good hour or two trying to make Wireguard work for me last night but failed. If I set it to only apply to Immich, nothing else would have Internet access at all. Likewise if I set the peer IP range to just my LAN subnet.

After pulling my hair out for a while I gave up and uninstalled.

The peer range shouldn’t be your LAN, it should be a new network range, just for WireGaurd. Make sure that the server running Immich is part of the WireGaurd network.

My phone and laptop see three networks: the internet, the lan (192.168.1.0/24, typically) and WireGaurd (10.30.0.0/16). I can anonymize and share my WireGaurd config if that would help.

Yes please, I might revisit it with a fresh pair of eyes.

Here are a few more details of my setup:

Components:

  • server
  • clients (phone/laptop)
  • domain name (we'll call it custom.domain)
  • home router
  • dynamic DNS provider

The home router has WireGuard port forwarded to server, with no re-mapping (I'm using the default 51820). It's also providing DHCP services to my home network, using the 192.168.1.0/24 network.

The server is running the dynamic DNS client (keeping the dynamic domain name updated to my public IP), and I have a CNAME record on the vpn.custom.domain pointing to the dynamic DNS name (which is an awful random string of characters). I also have server.custom.domain with an A record pointing to 10.30.0.1. All my DNS records are in public DNS (so no need to change the DNS settings on the computer or phone or use DNS overrides with WireGuard.)

Immich config:

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    ports:
      - target: 3001
        published: 2283
        host_ip: 10.30.0.1
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - immich

WireGuard is configured using wg-quick (/etc/wireguard/wg0.conf):

[Interface]
Address = 10.30.0.1/16
PrivateKey = 
ListenPort = 51820

[Peer]
PublicKey = 
AllowedIPs = 10.30.0.12/32

[Peer]
PublicKey = 
AllowedIPs = 10.30.0.11/32

Start WireGuard with systemctl enable --now wg-quick@wg0.

Phone WireGuard configuration (iOS):

[Interface]
Name = vpn.custom.domain

Private Key = 
Public Key = 

Addresses = 10.30.0.12/32
Listen port = 
MTU = 
DNS servers = 

[Peer]
Public Key = 
Pre-shared key = 
Endpoint = vpn.custom.domain:51820
Allowed IPs = 10.30.0.0/16
Persistent Keepalive = 25

[On Demand Activation]
Cellular = On
Wi-Fi = On
SSIDs = Any SSID

This connection is then left always enabled, and comes on whenever my phone has any kind of network connection.

My laptop (running Linux), is also using wg-quick (/etc/wireguard/wg0.conf):

[Interface]
Address = 10.30.0.14
PrivateKey = 

[Peer]
PublicKey = 
Endpoint = vpn.custom.domain:51820
AllowedIPs = 10.30.0.0/16

My wife's window's laptop is configured using the official WireGuard windows app, with similar settings.

No matter where we are (at home, on a WiFi hotspot, or using cellular data) we access Immich over the VPN: http://server.custom.comain:2283/.

Let me know if you have any further questions.

Thanks, I'll muse over this when I next get the chance!