just because the (corporate) internet works this way now, doesn’t mean it should
The web worked this way before there was a large corporate presence. Scraping was common during the blogosphere period and
robots.txt
was the solution everyone at the time agreed on and that’s been the standard ever since.I’m happy that you’re comfortable with this model, but I don’t want people who operate like this to intrude on the spaces we’re building to get away from it
We’re not intruding on this space. We’ve been in the fediverse for just as long or longer; the fediverse has been scrapable since 2008.
We’re not intruding on this space. We’ve been in the fediverse for just as long or longer; the fediverse has been scrapable since 2008.
Totally. And while it was scrapable, and scraped a lot, I wish there had been a lot more systematic public scraping of the “federated social web” (as it was called before the terrible name “fediverse” was adopted) back then - I had a lot of public conversations on identi.ca and StatusNet which I wish I could still see, but they now exist only in a bunch of private databases I don’t have access to. 😢