[not solved] I think I messed something up in the router settings but cannot understand what... now I have no internet connection

Linux@lemmy.ml – 22 points – 5 months ago

Hello! I (tried, at least) converted an old laptop to a Debian home server, and I was trying to set up duckdns.org and to enable port forwarding on my router. internet connection was working, I installed packages, docker, immich, etc, and then suddenly (I don't know exactly when) it refuses to connect to the internet. It does connect to local addresses (I can ssh into it) but ping google.com and any other internet-involving command fail. I had set up a rule on the router to forward port 80 to the device's port 80, but I then removed the rule and it still does not connect to the internet. I rebooted the router but nothing changed. Any idea what could be? the router is a Vodafone router.

I changed the hostname to debianserver but on the router it is still written debian. Also, it's the only device with unknown ipv6

thanks in advance!

EDIT: I rebooted again the server, and now ipv6 is not unknown anymore, and the hostname is correct. however, it still does not connect to the internet

EDIT 2:

only one device (debianserver) has this problem, other devices work as before

EDIT 3:

I don't know if it's useful or not, but if I boot a live debian USB in the server internet works

~~SOLUTION:~~ aaaaand no it does not work, after restarting docker it seems to work because all the brodges are yet to be created and it takes some time, after like 30 seconds it does not work as before :(

edit the file /lib/systemd/system/docker.service
append the flag --bip=192.168.3.1/24 to ExecStart=....
systemctl daemon-reload
systemctl restart docker

docker was the fucker that messed everything up and made me lose a few hours!

EDIT 4:

it seems that ip route flush 0/0 restores the internet connectivity until reboot... I don't know what does this means but can be a temporary workaround I guess? I really have no idea how to solve this

FINAL EDIT: I gave up. I removed debian and installed fedora, and now it all works like a charm

Does ping 8.8.8.8 work? (To check if it's DNS)

What's ip route show say? (Just to try to narrow down whether it's an issue with the server's config or the router's)

What's traceroute 8.8.8.8 display?

thanks for the answer!

ping 8.8.8.8 fails, and I don't have traceroute installed (and no internet to install it)

tubbadu@debianserver:~$ ip route show
0.0.0.0 dev veth3492bf7 scope link
0.0.0.0 dev vethc1bf668 scope link
0.0.0.0 dev vethb41fd7e scope link
0.0.0.0 dev veth2e39932 scope link
0.0.0.0 dev veth68451d9 scope link
default dev veth3492bf7 scope link
default dev vethc1bf668 scope link
default dev vethb41fd7e scope link
default via 192.168.1.1 dev enp1s0
169.254.0.0/16 dev veth68451d9 proto kernel scope link src 169.254.210.75
169.254.0.0/16 dev veth2e39932 proto kernel scope link src 169.254.242.12
169.254.0.0/16 dev vethb41fd7e proto kernel scope link src 169.254.185.90
169.254.0.0/16 dev vethc1bf668 proto kernel scope link src 169.254.225.22
169.254.0.0/16 dev veth3492bf7 proto kernel scope link src 169.254.123.220
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-56cf32fc7cde proto kernel scope link src 172.18.0.1
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.9
192.168.1.1 dev enp1s0 scope link

This is kind of a nutty network config. It looks like docker is setting up extra default routes, which I could easily see fouling everything up. As a first experiment (warning, this may ruin your networking until the next reboot):

ip route flush 0/0
ip route add default via 192.168.1.1 dev enp1s0

... and see if that makes things work (start with ping 192.168.1.1 and ping 8.8.8.8). If that solves the problem, then I think something about your docker config is adding stuff to your networking that's causing the problem; maybe remove/disable docker completely and then re-add docker things one at a time to see where the problem comes in.

okay, I thought to have solved the problem but I was wrong, here I go again. When I docker compose up -d the immich server (the only one I have installed) all those routes are created, and apparently some of them conflicts with something else and now my host has no internet connection. however it seems that ip route flush 0/0 solves the problem until the reboot, which is strange. the other command returns RTNETLINK answers: File exists

Hm. Yeah, that's weird. The default routes you're seeing should basically never exist, so it sounds like there's some kind of manual network config happening inside the Docker container that's creating a broken network.

What does docker network inspect [network] say for each of your Docker networks (substituting each Docker network for "[network]")? What's the network section of docker-compose.yml look like?

Try removing all the superfluous default routes.

My guess is that you have Docker configured incorrectly. Its internal IP range probably overlaps with your real network, so all requests are routed to Docker. Uninstall docker and reboot the server. If that works, reinstall docker and properly configure its internal networking.

I just figured this out! adding a --bip=192.168.3.1/24 to the systemd service solves this! [reference]

I understand you might be a bit stressed out, but it's very hard to make head or tails of what you're describing.

E.g. What do you mean by "no internet connection"? Does it mean just the old laptop, all devices on the network or the router itself?

thanks for the answer! Sorry I didn't specified very well: only the server has problems, all other devices work as before

I bet it has something to do with the LAN DHCP setup, since you have to set a static IP for the server...

Update - That won't work at all if it is indeed the issue. Let me think for a sec on how to pull external ip by doing a IP wget...

wget -qO - icanhazip.com

it waits like 10 seconds and then exits without any output

Yeah, I realized after the fact. This should work: wget 104.18.114.97, and you should get a ERROR 403: Forbidden if your server has any internet connection.

tubbadu@debianserver:~$ wget 104.18.114.97
--2024-01-18 21:35:38--  http://104.18.114.97/
Connecting to 104.18.114.97:80... failed: No route to host.

really really thanks for the help!

Please try wget https://104.18.114.97, if this one goes through I'd think the problem could be related to a faulty forwarding of regular internet traffic (port 80). If that fails as well I'd guess it's to due with the DHCP/Static IP's and involves your router. I'm absolutely clueless about Vodafone routers though.

Edit: Any connection would show "The certificate's owner does not match hostname ‘104.18.114.97’"

Edit: Any connection would show “The certificate’s owner does not match hostname ‘104.18.114.97’”

sorry I don't think I understand, where should this be written?

Here I've tried wget https://104.18.114.97. It's able to connect to the IP on port 443, but shows an error message since I'm not using a domain name. But at least I know I've reached the server.

so this means that the problem is actually the server? or do I misunderstand?

wget https://104.18.114.97

same as before:

tubbadu@debianserver:~$ wget https://104.18.114.97
--2024-01-18 21:52:29--  https://104.18.114.97/
Connecting to 104.18.114.97:443... failed: No route to host.

Something is completely shutting your server off from the internet, despite it having full LAN access. The only time I've run into this exact issue was when I misconfigured the firewall on a server, effectively only allowing for local connections. I simply started over by reinstalling Debian, wiping all my mistakes. But it could also be a setting on your router, and without you knowing what changes you made it's hard to give any reasonable advice.

These are just shots in the dark, and other might offer better solutions but I'd try;

Boot the laptop into a live session directly from USB. All settings are default. Test again, either wget or maybe ping a website. If it works, it's the server setup and I'd start over. If not...
Try reversing all changes on the router, give the server a different static IP.
Back up the router configuration to a file, consider making notes of important changes, reset the router and try again. If it STILL doesn't work you can restore the important settings. If it works, you can reimplement the settings from your notes. (unless we're talking manually imported VPN certificates and similar lol)

Sorry I wasn't able to help you out, I'm hosting from home and it's a fantastic thing when things

I don’t know if it’s useful or not, but if I boot a live debian USB in the server internet works

Haha yes, as mentioned. The issue is a setting on your laptop server installation. The simplest thing is just reinstalling and starting over.

The issue is a setting on your laptop server installation. The simplest thing is just reinstalling and starting over.

this is probably a good news I think, because it's only tedious but easy to solve If there's a way to do it without having to reinstall immich on docker would be awesome, but perhaps it's faster to reinstall everything than to look for a way to do this

thank you very very very VERY much for your help!!!!!!!!!

Happy to help, sorry I can't offer a simpler solution that avoids reformatting. It's probably doable if you want to do more fine grain troubleshooting but I've never found it to be worth the time.

For my home self-hosting I also prefer pre-made scripts exactly to avoid having to manually set up everything. It's such a slog and particularly if I have to restart from scratch. What's the goal of your project, perhaps something like Yunohost or even DietPi could save you a lot of work and get straight to the fun of self-hosting?

What’s the goal of your project?

I have my brother's old laptop with the screen broken, so I thought to convert it to a server for my huge pictures library I have on my external ssd installing immich, it's been a while since I'm reading things about this and now I finally have the chance to do it! I finished reinstalling now and internet works fine. I'll run a ping linux.org after every single step so if this happens again I'll at least know who's the guilty!

thanks again for your time and help!

You mentioned you changed firewall rules for that device. Any chance you have set outbound rule instead of inbound rule?

Anyway, what's the output of ip route?

Anyway, what’s the output of ip route?

tubbadu@debianserver:~$ ip route
0.0.0.0 dev veth60f206e scope link
0.0.0.0 dev veth48a6716 scope link
0.0.0.0 dev vethdf7381d scope link
0.0.0.0 dev veth59e46aa scope link
0.0.0.0 dev vethac4830e scope link
default dev veth60f206e scope link
default dev veth48a6716 scope link
default dev vethdf7381d scope link
default dev veth59e46aa scope link
default via 192.168.1.1 dev enp1s0
169.254.0.0/16 dev vethac4830e proto kernel scope link src 169.254.241.201
169.254.0.0/16 dev veth59e46aa proto kernel scope link src 169.254.97.105
169.254.0.0/16 dev vethdf7381d proto kernel scope link src 169.254.212.49
169.254.0.0/16 dev veth48a6716 proto kernel scope link src 169.254.26.74
169.254.0.0/16 dev veth60f206e proto kernel scope link src 169.254.212.242
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-56cf32fc7cde proto kernel scope link src 172.18.0.1
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.9
192.168.1.1 dev enp1s0 scope link

You mentioned you changed firewall rules for that device. Any chance you have set outbound rule instead of inbound rule?

All I did was doing this and press Save:

(now I rewrote it just to take the screenshot)

after internet stopped working I edited it to set the port to 443, and then I removed the rule. there are no rules now

Is there a reset button somewhere on the router? Most of them have something like this in order to reset them to factory settings. If not, google for your device name and factory reset, maybe it’s something like „press button while turning on“ etc. I’d try something like that

Yes there is in the web UI a factory reset button, but I'd rather not do this because it has some settings by other people

So, several people share the router and can modify settings...
That's a recipe for trouble.

I do external support for small businesses. When I work, configure etc. sometning, I change the password. I write the password on a piece of paper and put it in an envelope. Then I sign across the sealed flap. That envelope is handed to the customer. If I cease to work for them or something happens to me, they can open it and retrive the password. If something goes belly up, I ask for the envelope. If the envelope has been tampered, I wash my hands, and chrge handsomly to solve the issue.

I'd reset it, and then make someone responsible for doing things to the router, that way everyone knows what's going on.

whel this is clever XD

Go to the outer status page. The router should display whether it has an internet connection to your provider. If no, then your router/modem has no credentials or another issue preventing access.

If it shows as working, then you can narrow it down to incorrect DNS and IP routing. Perhaps dynamic IP allocation is set to off or another configuration error or bug, in which case you might need to reset all the router settings. Then, is it only broken for a single end device?

thanks for the reply! Sorry it isn't very clear from the post, but yes only one device (debianserver) has this problem (no internet connection, but yes local network connection), all other devices works as before. I'll update the original post to clarify this