connection/certificate errors from DSM. Package center, QC, and much more unavailable. (Synology ds1621+)

MonkCanatella@sh.itjust.works to Selfhosted@lemmy.world – 5 points –

Update (FIXED)

a dev from synology fixed it by running 'update-ca-certificates.sh' from the /etc/ssl/certs folder as root.

Not sure if he somehow moved that file into my nas manually if that's included functionality. either way it was a really simple fix that only took them like 3 weeks to address.

---- Original post -----

A couple days ago I randomly received this notification email: "System is busy and unable to deliver the diagnostic data. Please try again later." there's no info in the body. When I received the email I checked my DSM dashboard and it was unreachable, but my docker ccontainers were still running and I was able to SSH to the machine.

I tried to reset it and it didn't reset for around 20 minutes (i think maybe something to do with virtual machine manager because after logging back in and restarting, this was holding up the restart process). I restarted by holding it down until it powered down and started up again

It has been exhibiting odd behavior: I cannot access the package manager, Security advisor, quickconnect, support center, push service under notifications, sign in on the notification > email page on control panel, it does nothing, synology account section of the control panel, active insight, DSM update, all are giving me errors that imply some broken connection, certificate, or networking issue. I tried to update a docker container and I can't access the registry. It's giving me an error: "Error response from daemon: Get "https://registry-1.docker.io/v2/": x509: certificate signed by unknown authority" Trying to access synology photos from my phone also gives an invalid certificate error

I have made no changes to my router or system or anything.

I can ping all the services here from SSH: https://kb.synology.com/en-global/DSM/tutorial/What_websites_does_Synology_NAS_connect_to_when_running_services_or_updating_software

I'm worried that I somehow got attacked. I've been trying to figure out how to connect to this thing with my double nat situation which has made it impossible to access from outside without tailscale. I just don't understand what's happening. My worst fear is that maybe someone hacked in and modified my dsm install to mess with it or something. IDK.

I reached out to synology support a couple days ago but they responded with the most generic tech support questions:

you attempted to access DSM using various devices or web browsers? Are there any indications of hardware-related issues?

Are there any third-party applications or packages installed on your Synology device that might be affecting its performance?

Can you access the Synology device's interface directly, or is the problem limited to accessing DSM?

Have you encountered any recent power outages or disruptions that might have influenced the current situation?

I saw on this thread some people having errors that sound very similar but they all got them resolved around the same time

3

Sorry you're going through that OP. I manage around a dozen large business Synology servers, and this right here is why I cringe when I see it be so often recommended for home use. I appreciate the value they bring for home labs with all their out of the box bells and whistles, but when they fail, they fail really hard.

For your use case, your best bet is to ssh in and start digging through the logs. You'll be missing a lot of basic tools when you ssh in, like a text editor, and there is probably no package manager to install anything from the shell. I remember last time I had to do that I had to install some stuff by downloading directly. I believe I was also able to mount a remote drive from the shell and copy the logs over at one point.

Anyway, the log files should at least give you plenty of error messages and at least point you in the right direction for working with support. Good luck.

I checked out log center and there wasn't nearly enough information in those logs. Where would I find

FYI I can still access the DSM interface, but it gives errors on lots of things that I know need to get sorted. I will look into the logs. Do you have any pointers on what I might want to check for, or ways I could ensure I'm not compromised in any way? You mentioned you fixed yours by downloading some stuff directly, do you happen to remember more about that? I really appreciate the help and the sympathy. I seriously couldn't sleep the night it happened purely out of stress/anxiety :/

I found something that says (removed bunch of random identification shit from url vars)

Failed to exec curl command, with url: https://update7.synology.com/autoupdate/v2/getList , with reason: error setting certificate file: /etc/ssl/certs/ca-certificates.crt

I SSHed into /etc/ssl/certs and that doesn't exist. What does exist is a bunch of files like "b23h893.0" ""fb3j58d.0", etc. using ls -l I see that they have lrwxrwxrwx permissions on all of them and they point to stuff like affirmtrust, comodo cert, amazon_root, etc. nothing that makes any sense to me.

checking certificates from the control panel, all certificates are assigened to the cert generated by my quick connect url. I have 3 in total, one with my quick connect url, one with .direct.synology.me, and and that's just 'synology'. They all have a lock with a green checkmark, assuming that means they're legit but idk. What may be causing issues is that I can't use my quick connect. It's glitchy - I can't enable it because I get a connection error, but I'm wondering if that's cause of some certificate or maybe it's due to the error I showed above - that setting up the quick connect is throwing an error because it wants to write to some certificate that doesn't exist.

EDIT:

I found a (with the single quote around it) 'NetLock_Arany_=Class_Gold=_Főtanúsítvány.crt' in my /usr/shares/ca-certificates/mozilla . So ca-certificates exists but not where this updater thing is looking.