They generally do, because very few people bother to block them anyway. Complying with robots.txt is a good way to show in court that you did put effort into complying with the usual standards, without it ever impacting the useful information you’re scraping.
At the executive level, no I don’t think they care or pay attention, but considering both have said “here’s how to block our crawler,” I do hope that that some mistreated developer did actually program a check in to the crawler. I still think it’s worth doing, even though I don’t fully trust them.
you think either of those companies pays attention to robots.txt? its not legally binding or anythjng
They generally do, because very few people bother to block them anyway. Complying with robots.txt is a good way to show in court that you did put effort into complying with the usual standards, without it ever impacting the useful information you’re scraping.
At the executive level, no I don’t think they care or pay attention, but considering both have said “here’s how to block our crawler,” I do hope that that some mistreated developer did actually program a check in to the crawler. I still think it’s worth doing, even though I don’t fully trust them.