Using healthchecks.io and ntfy.sh to wake you up if your services are down

binwiederhier@discuss.ntfy.sh to Selfhosted@lemmy.world – 28 points –

cross-posted from: https://discuss.ntfy.sh/post/3200

I use ntfy (on another instance) + healthchecks.io to wake me up at night when ntfy.sh is down (crazy inception, right?). It's the "poor man's PagerDuty" if you will. It works amazingly.

Here's how I set it up:

  • I signed up healthchecks.io (free plan) and configured a project for "ntfy.sh" with a "ntfy" integration, i.e. publish to ntfy.example.com/<secret> with max priority
  • I have two different hosts execute small "integration ntfy.sh tests" and only ping healthchecks.io if they succeed. If they don't healthchecks.io will publish to ntfy.example.com/<secret>
  • In the ntfy Android app, I subscribe to ntfy.example.com/<secret>, enable "Keep alerting on highest priority", and make it override DND (do not disturb) for this topic.

Now when ntfy.sh goes down, the integration tests in the cronjobs will fail, and so healthchecks.io will not be pinged, which will trigger it to publish to ntfy.example.com/<secret> and let my phone consistently ring until I acknowledge it.

(Disclamer: I am the maintainer of ntfy. Hope posting this is fine. Happy to answer questions; I also have a brand new ntfy community, feel free to join)

24

Because of this thread, I signed up to healthchecks.io and configured it with gotify - similar to ntfy, I guess. Have a cron job checking my Lemmy instance is returning 200 every 15 minutes and then pinging healthchecks.

Gotify is great, but it can't override DND and wake you up at night. I suppose not many people want to do that anyway, haha. Only crazy me wants to be on call forever and all the time.

Why can't it override DND? In the notifications settings you set that for each type of message.

Wrong choice of words. What I meant was that you cannot have your alert ring consistently AFAIK. ntfy has an option to play the alert forever until you ACK it, which is very useful when you're sleeping or in a loud environment.

Maybe Gotify can do that too; though I was under the impression that it cannot. I've never used it so I don't know. 🙃

I self-host and use uptime-kuma to notify me over Telegram, works :tm: tho healthchecks.io for backups are great, use them with borgmatic.

I just have to say, I really love ntfy.

Years ago I used an app called simplepush, even bought the premium key. Installed it half a year ago and now they've changed to a yearly subscription. Fair enough. So I thought about throwing something together in python just to replicate the functionality just for my personal use... but luckily I searched and found ntfy first.

Really appreciate the work.

Thank you for the kind words. Feel free to recommend ntfy to your (nerd) friends if you like, haha.

Oh wow it's the man himself! Have to say you've put together an amazing piece of software! It has become an amazing tool in my alerting infrastructure. Thank you! for the wonderful project

lmao i won't lie, your meme looks like one of the recent reddit ads for a saas service xD

Except that you can selfhost both healthchecks.io and ntfy.sh; and you can't selfhost reddit or any of the services they advertise, haha

it's true :3 ntfy has been on my to-do to test out in my home lab. thank you for your service!

I thought the post was timely since the alert triggered today. The ntfy server was DDoS'd today and I got alerted pretty much instantly. It was quite nice.

must feel extra godly to see all your hard work perform exactly as hoped 💙