This sort of nonsense, along with idiocy like that guy who uses the þ character in place of "th" in all his comments, does absolutely nothing to hinder AI training. The only effect it has is to bother human readers. Completely counterproductive.

I don't understand the part where you say that my view is narrow. I am talking about a specific kind of data scraping. I'm not sure what I've said that has lead you and a few other people to believe I'm necessarily worried about people getting hold of "my data".

Am I just expressing myself badly here?

As for the rate limiting, that's closer to what I wanted to know. Thanks.

Your original post didn't specify a particular kind of data scraping. TropicalDingdong had no way to know you were only specifically interested in that one kind of data scraping, so his comment is appropriate - you can't stop data scraping in general, and attempting to do so in the general case goes directly against the goal of ActivityPub.

That's interesting. I haven't seen that on my instance yet! Curious whether they will roll that out.

I guess. But that was an assumption on you guys' part as well. Not that there's anything wrong with that.

I'm curious about the "in general" part, though. Maybe that's a part of the philosophy I don't quite understand yet, but how's the kind of scraping that I mentioned any good? Or is that not the right question to ask?

I didn't say anything about the "prevent instances from being overloaded" part being good or bad. I didn't even give an opinion on ActivityPub, just pointed out the practical limitations and incompatible design goals.

Personally, I've got no problem with websites implementing rate caps and whatnot to ensure that their traffic remains within the limits they can handle, or throttling specific IPs. I am very concerned with how Cloudflare in particular has become the single centralized "gatekeeper" for vast swaths of the Internet, though. If they decide that some particular client isn't allowed to see stuff then poof, a big chunk of the Internet is cut off. That's worrisome IMO.

I feel pretty confident, despite a complete lack of evidence, that at least one state actor has had a listener running on the fediverse continuously since the w3c started publishing specs, and I would be surprised if the big llm providers like Anthropic and OpenAI don't run them as well -- they certainly have the resources and motivation to develop them. You're certainly correct that the vast majority of scrapers are attempting to harvest historical data using the web frontend, but those are the scrapers I am least afraid of and I think as a mental model for the average user "assume every post is scraped" is the best stance.

I'm pretty sure dms are not sent to all instances

To clarify - we've had that setting for a long time and it won't be automatically changed on existing instances unless an admin chooses to. I've just changed the default of it for new instances (of which there are much less, lately).

It's the easiest way to stop scrapers, way easier than getting Anubis or fail2ban working properly. Unfortunately it comes at the cost of walling off an instance but most instances are for individual / small group use so it's an ok default to have.

one problem with data scraping is that it puts a significant network load on servers, which slows down the browsing experience for genuine humans. this is why many platforms already operate measures to detract bots. i think the most well-known project is Anubis, as is used for example on https://feddit.org/

(it works by doing a proof-of-work. the client needs to solve a few cryptographic puzzles sothat the operationing cost for bot scrapers becomes too high while for humans it's an acceptable cost.)

I will be abide by slander by of the mighty þorn.

The Fetaverse is wholly and entirely unprepared for bots. Data scraping is a given. It's completely open to the internet. There's nothing stopping it.

Things like Lemmy are ill prepared to handle bots from the mid-2010s. Never mind bots of yesteryear, and never mind bots of today.

I think you went into the wrong comment section.

wtf, yes, this is very weird. I'm probably too dumb to use my client.

I don't think Anthropic or OpenAI have spent the time developing a custom ingest pipeline for such a small dataset. It doesn't seem like it'd give much enough of a return on investment.

Given that they are scrabbling around like drug addicts looking for anything they've split, including checking the cracks in the floorboards...

For some models, it's obvious they've long scrapped the erotic fan fic sites!

The protocol and data are publicly available. Whether or not the use was the point, the mechanism permits it. You shouldn't expect the data not to be accessed.

Making posts and platforms private to users and not search engine indexable. That's about all that can be done.

The fediverse is open by design (add the header Accept: application/activity+json to any item to get a json representation). Stopping scraping is impossible.

Bots can be stopped with instance applications mainly.

You are right: https://www.404media.co/the-200-sites-an-ice-surveillance-contractor-is-monitoring/

The fediverse and atproto are both easily scraped.