everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

  • kcweller@feddit.nl
    link
    fedilink
    English
    arrow-up
    11
    ·
    9 hours ago

    Nephentes that shit. Poison every scraper until they start respecting robot.txt. Purposefully use llm.txt to trap the fuckers.

      • groet@feddit.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        2 hours ago

        You most likely pay for a maximum CPU capacity and your Server cant go above that no matter what you run. Your vps provider doesn’t care what that CPU power is used for.

        However with limited resources, the tarpit might use up all CPU power and the rest of the webserver will crawl to a halt.