Jump to content

Community Status Updates


Photo

carlsson

I wonder if a crawling bot realizes it is not welcome after a while of being served HTTP 403.
Apr 21 2017 6:28 AM
  • Reaperman's Photo
    Reaperman
    You could just tell it to its face with a robots.txt.
    Apr 21 2017 6:55 AM
  • carlsson's Photo
    carlsson
    True, I can try that and see if the bot even bothers about robots.txt. Not all of them do, in particular the more intruisive ones.
    Apr 21 2017 7:33 AM
  • Reaperman's Photo
    Reaperman
    Yeah, I suppose 'no trespassing' signs only work on honest folk.
    Apr 21 2017 8:08 AM