Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.
I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
Since my grade-school mascot/team was "The Trojans" and my adopted hometown is firmly rooted in pre-Christian Greek culture and mythology, I am currently studying the Trojan War, and I may observe that "Trojan" is an extremely perspicacious brand-name for condoms, more than the average guy would expect.
Is it ethical to scrape when a site has explicitly blocked bots? I know a fair number of people who run small sites who are already considering closing them down because the bots are relentlessly hammering their sites and driving up hosting costs.
I think it really depends on how the bot is run. If the bot is replacing me navigating there manually, absolutely. If it’s sucking up content to rip off and make someone else billions of dollars, no.
Why do you feel entitled to navigate to a website you neither own nor pay for via a method the owner expressly forbids?
It's most definitely unethical.
odd name; shouldn't it be named after something that stealthily infiltrates fortresses?
fellow contributor on this.
we thought of it the other way round: your automation is the fortress, and every bot-detector trying to fingerprint it is the siege.
also most good burglar names were taken on PyPI
in my defence, i did lobby hard for "GuyWhoWavesYouThroughTheGate"
"Doorman"
"Of course... Why didn't I think of that?"
Trojan Horse probably wouldn't make a very good name for a browser.
Since my grade-school mascot/team was "The Trojans" and my adopted hometown is firmly rooted in pre-Christian Greek culture and mythology, I am currently studying the Trojan War, and I may observe that "Trojan" is an extremely perspicacious brand-name for condoms, more than the average guy would expect.