If you have the time to spare (a few weeks perhaps, if coming from zero) to experiment and read, Prometheus and Grafana offers a lot and can be really flexible. I use a pretty simple bash script that scrapes my desired https endpoints and writes out the results to a file Prometheus (node-exporter) understands, and from there I can write alert rules in Grafana to fire off notices by email or slack.

I did a 4 node Pi4 kubernetes cluster for about 5 years. The learning experience was priceless. I think most notable was learning to do proper multiarch container builds to support arm and x86_64. That being said, about half a year ago I decided to try condensing it all into two n100 nuc-like clones and keep one pi as the controller. For me and my apps and use cases there was no going back. Performance gains were substantial and in this regard I think I was hobbling myself after the educational aspect plateaued.