NodeBB 4.13: activitypub/fetchPublicKey Rate limit exceeded errors.
-
Starting with 4.13, I've started receiving rate limit errors for ActivityPub public key fetching attempts. As a result, my World feed has drawn to a crawl and I am receiving almost no content from Lemmy servers or larger Mastodon instances.
[4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 1] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 2] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 3] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 4] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 5](IP addresses removed manually)
-
Starting with 4.13, I've started receiving rate limit errors for ActivityPub public key fetching attempts. As a result, my World feed has drawn to a crawl and I am receiving almost no content from Lemmy servers or larger Mastodon instances.
[4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 1] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 2] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 3] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 4] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 5](IP addresses removed manually)
@Kichae are the IP addresses ones you control? It may be triggering the rate limit when the incorrect IP address is proxied in.
I experienced this too, and it was due to Anubis not passing the right IP through. I just disabled it for now <img class="not-responsive emoji" src="https://community.nodebb.org/assets/plugins/nodebb-plugin-emoji/emoji/customizations/6854442b-3161-4485-a9c4-ea0977c622e7-abf762c960d25b9d.gif?v=c5868b2e003" title=":think:" />
Does that sound like something similar? The rate limiting was added to guard against unchecked use of the affected endpoint
-
Starting with 4.13, I've started receiving rate limit errors for ActivityPub public key fetching attempts. As a result, my World feed has drawn to a crawl and I am receiving almost no content from Lemmy servers or larger Mastodon instances.
[4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 1] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 2] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 3] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 4] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 5](IP addresses removed manually)
Nope. I currently only have this one forum running on ActivityPub, and all of my sites are running on a single IP. Some of these point back to smaller Mastodon servers, most of them don't seem to have URLs associated with them, so I'm not sure who it is.
-
Nope. I currently only have this one forum running on ActivityPub, and all of my sites are running on a single IP. Some of these point back to smaller Mastodon servers, most of them don't seem to have URLs associated with them, so I'm not sure who it is.
@Kichae hmm, perhaps the fetchPublicKey IP limiter isn't bit too aggressive.
See if you can modify this line. Try changing 60 to something higher like 180
-
Starting with 4.13, I've started receiving rate limit errors for ActivityPub public key fetching attempts. As a result, my World feed has drawn to a crawl and I am receiving almost no content from Lemmy servers or larger Mastodon instances.
[4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 1] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 2] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 3] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 4] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 5](IP addresses removed manually)
Bumping it up to 180 has given me a larger variety of Mastodon sources coming through. Lemmy still seems silent, though.
-
Bumping it up to 180 has given me a larger variety of Mastodon sources coming through. Lemmy still seems silent, though.
@Kichae Lemmy may take a little while longer because there's a cooloff period they won't send content until your server begins responding again.
The IP limit rejections caused those Lemmy servers to temporarily mark yours as down. Lemmy queues everything for later delivery.
-
Starting with 4.13, I've started receiving rate limit errors for ActivityPub public key fetching attempts. As a result, my World feed has drawn to a crawl and I am receiving almost no content from Lemmy servers or larger Mastodon instances.
[4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 1] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 2] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 3] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 4] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 5](IP addresses removed manually)
Cool! I'll keep an eye out for that, then. 180 seems to be working increasingly well for Mastodon servers now ad North America wakes up.
-
Starting with 4.13, I've started receiving rate limit errors for ActivityPub public key fetching attempts. As a result, my World feed has drawn to a crawl and I am receiving almost no content from Lemmy servers or larger Mastodon instances.
[4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 1] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 2] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 3] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 4] [4566/1203025] - [33mwarn[39m: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 5](IP addresses removed manually)
I had to push the value up to 240 to make the errors stop totally. I haven't noticed any meaningful difference in my World feed after doing this, but my error log's much cleaner now.
I'm still not seeing movement on the Lemmy side of things. I went to check the Lemmy forums I'm subscribed to here, to see if there's a similar issue there, and... it's been 18 days since the last Lemmy post was received by com.nbb from the communities I'm subscribed to. These might be separate issues...
-
I had to push the value up to 240 to make the errors stop totally. I haven't noticed any meaningful difference in my World feed after doing this, but my error log's much cleaner now.
I'm still not seeing movement on the Lemmy side of things. I went to check the Lemmy forums I'm subscribed to here, to see if there's a similar issue there, and... it's been 18 days since the last Lemmy post was received by com.nbb from the communities I'm subscribed to. These might be separate issues...
@Kichae understood, I'll have to take a closer look.
There is definitely an issue, some relays have stopped working but I'm sure the relay is fine.
Stay tuned...
-
I had to push the value up to 240 to make the errors stop totally. I haven't noticed any meaningful difference in my World feed after doing this, but my error log's much cleaner now.
I'm still not seeing movement on the Lemmy side of things. I went to check the Lemmy forums I'm subscribed to here, to see if there's a similar issue there, and... it's been 18 days since the last Lemmy post was received by com.nbb from the communities I'm subscribed to. These might be separate issues...
@Kichae there's definitely a federation issue with Lemmy here...
This dashboard shows ActivityPub.Space is just over three weeks behind lemmy.world. Failure count is ~64, which means Lemmy stopped sending stuff over.
I upped my public key fetch limiter value to 512 as well, and will see if this addresses the issue.
-
@Kichae there's definitely a federation issue with Lemmy here...
This dashboard shows ActivityPub.Space is just over three weeks behind lemmy.world. Failure count is ~64, which means Lemmy stopped sending stuff over.
I upped my public key fetch limiter value to 512 as well, and will see if this addresses the issue.
@kichae@community.nodebb.org can you try again with the latest
developcommits?A couple things:
- You can now set the rate limit value to 0 to disable it completely
- The logic was a mix of both a db-backed mutex lock and a rate limiter, which was overkill. I think what happened was Lemmy (or other popular instances) sent many activities all at once, causing the rate-limiter to trip, even though the value was well under the threshold.
The default is now updated to 256 but if you've overridden it your value stays. I'll keep an eye on Lemmy federation to see if that was the issue...
If I am right, then within the next 6-24 hours, I should be receiving a flurry of Lemmy content as it speedruns through the backlog.
-
As it turns out I was not right, at least not entirely <img class="not-responsive emoji" src="https://activitypub.space/assets/plugins/nodebb-plugin-emoji/emoji/android/1f61b.png?v=c1f554d6ce2" title="
" /> -
Yeah, I'm not seeing anything behaving differently on develop. My logs are super clean now, though.
-
Yeah, I'm not seeing anything behaving differently on develop. My logs are super clean now, though.
@kichae@community.nodebb.org hopefully this will be something that we'll see results on within a day.
Once Lemmy decides your server is "down", it won't send anything for awhile. How long that is I don't know offhand.
-
Heckin' yeah. Update to the latest
developagain.
For me,There it goes... 5 million activities an hour hahahlemmy.worldhasn't started re-sending activities yet, but it will...
-
Better, but it's not over yet:
Error rates have crept back up, so there's more debugging to do.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login