I went down this very same twisty road a while back with rootless Podman. I tried several of the solutions you mentioned. None of them worked. The actual working solution I finally settled on was using Proxy Protocol to pass the original client IP from the host into a container. In my particular case, I’m running a very basic HAProxy config on the host that’s talking Proxy Protocol to Traefik running in a container. And it works great; actual client IPs show up in the logs as expected.

In your particular case, you could probably run HAProxy on the host and have that talk Proxy Protocol to Caddy running in a container.

You can do calendar and contacts separate from email. Try Radicale. I’ve been using it for years.

Another container-based alternative in that space is Mailu.