Maybe its only needed for AI review as opposed to generic google search results? Just realized one of the website we made already has a robots.txt, we were just unaware about. Thanks for sharing
I find this a bit hard to believe that they refuse to index it at all if the robots.txt doesn't exist. Would love to see more data or confirmation on this.
Maybe its only needed for AI review as opposed to generic google search results? Just realized one of the website we made already has a robots.txt, we were just unaware about. Thanks for sharing
I find this a bit hard to believe that they refuse to index it at all if the robots.txt doesn't exist. Would love to see more data or confirmation on this.