3 points | by flyingcoder an hour ago
2 comments
How do you know it's facebook's crawler if it going through residential proxies?
It had meta in the user-agent headers, it’s trivial to spoof but I’ve read of many others who are seeing the same thing. It’s not just them though, it’s also parallel web systems, and then a bunch of others who aren’t identifying themselves.
How do you know it's facebook's crawler if it going through residential proxies?
It had meta in the user-agent headers, it’s trivial to spoof but I’ve read of many others who are seeing the same thing. It’s not just them though, it’s also parallel web systems, and then a bunch of others who aren’t identifying themselves.