How to Block Some of the Bots

15 points | by Bender an hour ago

12 comments

  • fxtentacle an hour ago

    I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.

      Bender 42 minutes ago

      When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.

  • binaryturtle an hour ago

    410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks!

    Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)

      Bender an hour ago

      Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not.

      People can see a few headers here [2]

      [1] - https://caniuse.com/?search=sec-fetch-mode

      [2] - https://nochan.net/.env

      gruez 27 minutes ago

      I didn't even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn't even get past the tls handshake.

        ErroneousBosh 19 minutes ago

        I mean that just might be the dying gasp of a puddle of scorched fibreglass and boiling copper where the server used to be.

          Bender 17 minutes ago

          Mostly harmless(c)

              load average: 0.00, 0.01, 0.00
  • ajsnigrutin 8 minutes ago

    There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.

      Bender a few seconds ago

      [delayed]

  • iririririr 28 minutes ago

    most (all?) of those will 100% block valid traffic too

      Bender 18 minutes ago

      I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.

        ferngodfather 3 minutes ago

        > Obvious proxy is obvious.

        > if ($http_x_forwarded_for) {....

        This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn't know they could turn it off).