We all migrate to smaller websites try not to post outside drawing attention just to hide from the “Ai” crawlers. The internet seems dead except for the few pockets we each know existed away from the clankers
We all migrate to smaller websites try not to post outside drawing attention just to hide from the “Ai” crawlers. The internet seems dead except for the few pockets we each know existed away from the clankers
But an IP can have multiple websites and even not return anything on plain IP access. How do crawlers find out about domains and unlinked subdomains? Do they even?
@kossa @dual_sport_dork If you’re using HTTPS, which is by and large the norm nowadays, then every domain is going to be trivially discoverable via certificate transparency logs: https://social.cryptography.dog/@ansuz/115592837662781553
Thanks for the link!
thinking about this, wouldn’t the best way to hide a modern websie be something along getting a wildcard domain cert (can be done with LE with DNS challenge), cnaming the wildcard to the root domain and then hosting the website on a random subdomain string ? am I missing something
I do something something like this using wildcard certs with Let’s Encrypt. Except I go one step further because my ISP blocks incoming data on common ports so I end up using an uncommon port as well.
I’m not hosting anything important and I don’t need to always access to it, it’s mostly just for fun for myself.
Accessing my site ends up looking like
https://randomsubdomain.registered-domain-name.com:4444/My logs only ever show my own activity. I’m sure there are downsides to using uncommon ports but I mitigate that by adjusting my personal life to not caring about being connected to my stuff at all times.
I get to have my little hobby in my own corner of the internet without the worry of bots or AI.
Every SSL certificate is publicly logged(you can see these logs e. g. under crt.sh) and you might be able to read DNS records to find new (sub)domains. The modern internet is too focused on being discoverable and transparent to make hiding an entire service(domain + servers) feasible. But things like example.com/dhusvsuahavag8wjwhsusiajaosbsh are entirely unfindable as long as they are not linked to
Random subdomain on wildcard certificate, IP written in the host file to mitigate DNS records, only given by word-to-mouth 😅.
Nobody said the uncrawled dark forest would be comfortable.