Yup. The robots.txt file is not only meant to block robots from accessing the site, it’s also meant to block bots from accessing resources that are not interesting for human readers, even indirectly.
For example, MediaWiki installations are pretty clever in that by default, /w/
is blocked and /wiki/
is encouraged. Because nobody wants technical pages and wiki histories in search results, they only want the current versions of the pages.
Fun tidbit: in the late 1990s, there was a real epidemic of spammers scraping the web pages for email addresses. Some people developed wpoison.cgi
, a script whose sole purpose was to generate garbage web pages with bogus email addresses. Real search engines ignored these, thanks to robots.txt. Guess what the spam bots did?
Do the AI bros really want to go there? Are they asking for model collapse?
I recommend one of my favourite CRPGs of all time: Neverwinter Nights - for the modern hassle-free experience, get the Enhanced Edition. The first single-player campaign is pretty meh by Bioware standards, but the expansion packs (included in the NWNEE) are pretty great. Heard a lot of good about the premium modules (a few of the original premium modules come with NWNEE, the rest are available as DLC).
The official campaigns are set in Forgotten Realms, the same D&D setting as BG3, but you really don’t need to worry about diving headlong into horrors. More fantasy vibes and less visceral stuff. (the second expansion pack is a bit more in the direction of subterranean spooks, but not, like, excessively so.)
However, the real big strength of NWN was not the campaigns. It was deliberately designed for player-created adventure modules created with the included Aurora Toolset. There’s loads of them and some of them had really great production values and writing. They’re currently hosted at Neverwinter Vault and NWNEE also has a custom content browser (though the latter doesn’t have much stuff). Custom modules also have a whole bunch of genres and settings, as expected.
Oh and it’s a game from 2002 so it runs on any ol’ potato. (Well the EE needs a vaguely modernish machine, but not anything unreasonable.)