I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
I want to give special thanks to the members of our community who have
dedicated their time and efforts to track down the perpetrators of this
attack.
Although this fake abuse incident had minimal impact on the network --
temporarily taking only a few relays offline -- it has been a
frustrating issue for many relay operators. However, I want to reassure
everyone that this disruption had no effect on Tor users whatsoever.
We're incredibly fortunate to have such a skilled and committed group of
relay operators standing with Tor.
Thank you all for your resilience, ongoing support and for making the
Tor network possible by running relays.
We should expect some more days of fallout, while mistaken abuse
complaints are still being processed by various hosters. That is, if
you get a complaint from your hoster tomorrow, be sure to check the
timestamp before worrying that there is some new variant of the attack.
That said, everybody please do keep watch for some future variation of
this attack. All the attack needs is a hosting provider that doesn't do
egress filtering, i.e. that lets its users pretend to be anybody anywhere
on the internet. Those hosting providers are supposed to be gone from
the world decages ago, but well, the world is flawed in many ways and
this isn't the worst of them. At least if it happens again soon,
many people understand the attack now and they will be ready to track
it down quickly again.
--Roger
···
On Thu, Nov 07, 2024 at 03:49:37PM -0300, gus wrote:
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
Would you please expand on that a bit please? Was it a single server, a
network of them, one provider or multiple of them, etc...?
I doubt this was the work of a single person simply because they were
bored. I'm assuming we should still keep a lookout for
them to simply rent a bunch of more servers and continue.
By the way, I just received two more abuse reports an hour ago regarding
scans that happened on Nov. 6 so this might hopefully be before the stop
of the attacks.
Thank you
Enkidu
···
On 11/7/2024 1:49 PM, gus wrote:
Hello everyone,
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
I want to give special thanks to the members of our community who have
dedicated their time and efforts to track down the perpetrators of this
attack.
Although this fake abuse incident had minimal impact on the network --
temporarily taking only a few relays offline -- it has been a
frustrating issue for many relay operators. However, I want to reassure
everyone that this disruption had no effect on Tor users whatsoever.
We're incredibly fortunate to have such a skilled and committed group of
relay operators standing with Tor.
Thank you all for your resilience, ongoing support and for making the
Tor network possible by running relays.
Gus
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
We should expect some more days of fallout, while mistaken abuse
complaints are still being processed by various hosters.
You called it. Mere minutes ago, Hetzner forwarded another complaint,
for a grand total of 9 (yes, nine, what a gruesome level of abuse)
spoofed connection attempts over the course of November 5 and 6.
The destination addresses were part of the known class C subnets already
reported here, and the source of the complaint were of course the
tireless dolts at watchdogcyberdefense.com. Unsurprisingly, I can't tell
if Hetzner is not done processing old complaints, or if the complaining
party is still generating fresh mail based on their accumulated backlog.
Apart from that: My thanks to everybody who helped clamping down on this.
-Ralph
···
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
Thank-you for you efforts, and for the efforts of the anonymous contributors! And let me second the motion requesting (much) more information about the perps.
Do we know the full impact though? The vast majority of relay operators seem not to be on the mailing list. What are the actual numbers on how many relays went dark in that period? I think this is a number that would be good to know. Marie lost 10 IONOS VPSs in one shot, and only two are back. Another 10 or so IONOS servers went dark at that same time and are still not back.
More than the number of servers lost, it was shown that it's quite possible to discredit with an IP spoof. Given that the effect of this should have been exactly zero, I'd (unfortunately) call their operation surprisingly successful.
Information and education are the best weapons against any sort of discredit attack. I recommend an official educational blog entry from the project if (when?) this happens again in the future. Or was there one and I'm just not aware of it? This is valuable if nothing else to reassure relay operators that the project has their backs as much as possible and is willing to go to bat for them.
Marie, if you're still on the list, do you want to speak toward your efforts to get your shut down servers back? You are, to my knowledge, the person who lost the most in one shot to this.
···
On 2024-11-07 14:49, gus wrote:
Hello everyone,
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
I want to give special thanks to the members of our community who have
dedicated their time and efforts to track down the perpetrators of this
attack.
Although this fake abuse incident had minimal impact on the network --
temporarily taking only a few relays offline -- it has been a
frustrating issue for many relay operators. However, I want to reassure
everyone that this disruption had no effect on Tor users whatsoever.
We're incredibly fortunate to have such a skilled and committed group of
relay operators standing with Tor.
Thank you all for your resilience, ongoing support and for making the
Tor network possible by running relays.
Gus
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
Thank-you for you efforts, and for the efforts of the anonymous contributors! And let me second the motion requesting (much) more information about the perps.
Do we know the full impact though? The vast majority of relay operators seem not to be on the mailing list. What are the actual numbers on how many relays went dark in that period? I think this is a number that would be good to know. Marie lost 10 IONOS VPSs in one shot, and only two are back. Another 10 or so IONOS servers went dark at that same time and are still not back.
More than the number of servers lost, it was shown that it's quite possible to discredit with an IP spoof. Given that the effect of this should have been exactly zero, I'd (unfortunately) call their operation surprisingly successful.
Information and education are the best weapons against any sort of discredit attack. I recommend an official educational blog entry from the project if (when?) this happens again in the future. Or was there one and I'm just not aware of it? This is valuable if nothing else to reassure relay operators that the project has their backs as much as possible and is willing to go to bat for them.
Marie, if you're still on the list, do you want to speak toward your efforts to get your shut down servers back? You are, to my knowledge, the person who lost the most in one shot to this.
On 2024-11-07 14:49, gus wrote:
Hello everyone,
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
I want to give special thanks to the members of our community who have
dedicated their time and efforts to track down the perpetrators of this
attack.
Although this fake abuse incident had minimal impact on the network --
temporarily taking only a few relays offline -- it has been a
frustrating issue for many relay operators. However, I want to reassure
everyone that this disruption had no effect on Tor users whatsoever.
We're incredibly fortunate to have such a skilled and committed group of
relay operators standing with Tor.
Thank you all for your resilience, ongoing support and for making the
Tor network possible by running relays.
Gus
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
This rule will also count SYN-ACKs sent from your own server to bots trying to connect to your SSH on port 22.
To get the right count for the SYN-ACKs coming back from the spoofed packets, you’ll want to exclude your own IP address. You can do that like this:
tcp sport 22 tcp flags syn,ack / syn,ack ip saddr != 172.16.254.1 counter
Just swap out 172.16.254.1 with the IP address of your Tor relay.
···
On 8/11/24 03:14, Red Oaive via tor-relays wrote:
That was in five minutes.
On 2024-11-08 03:03, Red Oaive wrote:
Thank-you for you efforts, and for the efforts of the anonymous contributors! And let me second the motion requesting (much) more information about the perps.
Do we know the full impact though? The vast majority of relay operators seem not to be on the mailing list. What are the actual numbers on how many relays went dark in that period? I think this is a number that would be good to know. Marie lost 10 IONOS VPSs in one shot, and only two are back. Another 10 or so IONOS servers went dark at that same time and are still not back.
More than the number of servers lost, it was shown that it's quite possible to discredit with an IP spoof. Given that the effect of this should have been exactly zero, I'd (unfortunately) call their operation surprisingly successful.
Information and education are the best weapons against any sort of discredit attack. I recommend an official educational blog entry from the project if (when?) this happens again in the future. Or was there one and I'm just not aware of it? This is valuable if nothing else to reassure relay operators that the project has their backs as much as possible and is willing to go to bat for them.
Marie, if you're still on the list, do you want to speak toward your efforts to get your shut down servers back? You are, to my knowledge, the person who lost the most in one shot to this.
On 2024-11-07 14:49, gus wrote:
Hello everyone,
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
I want to give special thanks to the members of our community who have
dedicated their time and efforts to track down the perpetrators of this
attack.
Although this fake abuse incident had minimal impact on the network --
temporarily taking only a few relays offline -- it has been a
frustrating issue for many relay operators. However, I want to reassure
everyone that this disruption had no effect on Tor users whatsoever.
We're incredibly fortunate to have such a skilled and committed group of
relay operators standing with Tor.
Thank you all for your resilience, ongoing support and for making the
Tor network possible by running relays.
Gus
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
This rule will also count SYN-ACKs sent from your own server to bots trying to connect to your SSH on port 22.
To get the right count for the SYN-ACKs coming back from the spoofed packets, you’ll want to exclude your own IP address. You can do that like this:
tcp sport 22 tcp flags syn,ack / syn,ack ip saddr != 172.16.254.1 counter
Oops, I sent that email before my morning coffee kicked in! You don’t need to worry about excluding your own IP address in the input chain. But definitely make sure to exclude the IPs of other Tor relays listening on port 22. That could be why you’re seeing those counters go up.
···
On 8/11/24 08:47, tor-relays+tor-relays@queer.cat wrote:
On 8/11/24 03:14, Red Oaive via tor-relays wrote:
Just swap out 172.16.254.1 with the IP address of your Tor relay.
That was in five minutes.
On 2024-11-08 03:03, Red Oaive wrote:
Thank-you for you efforts, and for the efforts of the anonymous contributors! And let me second the motion requesting (much) more information about the perps.
Do we know the full impact though? The vast majority of relay operators seem not to be on the mailing list. What are the actual numbers on how many relays went dark in that period? I think this is a number that would be good to know. Marie lost 10 IONOS VPSs in one shot, and only two are back. Another 10 or so IONOS servers went dark at that same time and are still not back.
More than the number of servers lost, it was shown that it's quite possible to discredit with an IP spoof. Given that the effect of this should have been exactly zero, I'd (unfortunately) call their operation surprisingly successful.
Information and education are the best weapons against any sort of discredit attack. I recommend an official educational blog entry from the project if (when?) this happens again in the future. Or was there one and I'm just not aware of it? This is valuable if nothing else to reassure relay operators that the project has their backs as much as possible and is willing to go to bat for them.
Marie, if you're still on the list, do you want to speak toward your efforts to get your shut down servers back? You are, to my knowledge, the person who lost the most in one shot to this.
On 2024-11-07 14:49, gus wrote:
Hello everyone,
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
I want to give special thanks to the members of our community who have
dedicated their time and efforts to track down the perpetrators of this
attack.
Although this fake abuse incident had minimal impact on the network --
temporarily taking only a few relays offline -- it has been a
frustrating issue for many relay operators. However, I want to reassure
everyone that this disruption had no effect on Tor users whatsoever.
We're incredibly fortunate to have such a skilled and committed group of
relay operators standing with Tor.
Thank you all for your resilience, ongoing support and for making the
Tor network possible by running relays.
Gus
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
Are you sure that it has been effectively shut down? We're still
receiving spoofed packets with IP addresses of Tor relays set as source
after this message has been posted. We've also received more "reports"
from the same newbies after this message was posted.
Our traps even see packets with the IP addresses of Tor relays that are
in the same subnet.
So far we've been able to trace this to a certain peer, we'll be
monitoring.
···
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
Unfortunately I lost one relay (Runaz3) that was hosted on dataclub.eu today. They didn´t even send me a message so I could explain to them what is happening. They just shut down my server without any notice. My second server hosted at ATW is in danger as well.
···
On 11/8/24 2:08 PM, marie wrote:
My efforts to get them back are/where pretty low, its not much effort for me to set up new relays. The support also didnt gave me much information, so i just created new Relays at Strato, but they are in the same Datacenter as the Ionos ones. Im now checking out other providers for more relays. Maybe it was also some combination of other factors why they shut down my servers, i had like 13 of the 1€/Month ones, could be that it looked like abuse to them.
On 08.11.24 08:03, Red Oaive wrote:
Thank-you for you efforts, and for the efforts of the anonymous contributors! And let me second the motion requesting (much) more information about the perps.
Do we know the full impact though? The vast majority of relay operators seem not to be on the mailing list. What are the actual numbers on how many relays went dark in that period? I think this is a number that would be good to know. Marie lost 10 IONOS VPSs in one shot, and only two are back. Another 10 or so IONOS servers went dark at that same time and are still not back.
More than the number of servers lost, it was shown that it's quite possible to discredit with an IP spoof. Given that the effect of this should have been exactly zero, I'd (unfortunately) call their operation surprisingly successful.
Information and education are the best weapons against any sort of discredit attack. I recommend an official educational blog entry from the project if (when?) this happens again in the future. Or was there one and I'm just not aware of it? This is valuable if nothing else to reassure relay operators that the project has their backs as much as possible and is willing to go to bat for them.
Marie, if you're still on the list, do you want to speak toward your efforts to get your shut down servers back? You are, to my knowledge, the person who lost the most in one shot to this.
On 2024-11-07 14:49, gus wrote:
Hello everyone,
I'm writing to share that the origin of the spoofed packets has been
identified and successfully shut down today, thanks to the assistance
from Andrew Morris at GreyNoise and anonymous contributors.
I want to give special thanks to the members of our community who have
dedicated their time and efforts to track down the perpetrators of this
attack.
Although this fake abuse incident had minimal impact on the network --
temporarily taking only a few relays offline -- it has been a
frustrating issue for many relay operators. However, I want to reassure
everyone that this disruption had no effect on Tor users whatsoever.
We're incredibly fortunate to have such a skilled and committed group of
relay operators standing with Tor.
Thank you all for your resilience, ongoing support and for making the
Tor network possible by running relays.
Gus
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
...as long as you're not on the part of the internet that has censored
that IP address, that is.
--Roger
···
On Fri, Nov 08, 2024 at 11:14:54AM -0400, tor-relays+tor-relays@queer.cat wrote:
But
definitely make sure to exclude the IPs of other Tor relays listening on
port 22. That could be why you’re seeing those counters go up.
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
The rule is on the source port = 22, not the destination port = 22. Incoming bot connections will not have a sport = 22.
It is also in a chain hooked only to input packets and will not trigger on outgoing packets.
~# nft list table ip accounting
table ip accounting {
chain input {
type filter hook input priority filter; policy accept;
...
tcp sport 22 tcp flags == 0x12 counter packets 210 bytes 12360
}
My ssh service is anyway behind knockd, so my machine will never send out SYN-ACKS. The knockd ssh rule ssh is reject so it will only send out RSTs.
Also, these have to be coming from more than one source. The byte count is not an even multiple of the number of packets, meaning that there are almost assuredly different sources with different stack configurations.
I assess the rule is correctly configured to detect only incoming syn-acks and that I am seeing SYN-AKCs from multiple machines that were targetted with SYNs spoofing of my IP.
I am seeing this behavior on a friend's VPS with newly created relay. None of my more public-facing VPSs that are not involved in Tor are seeing these.
I would encourage everyone to add the above table and rule so we can assess how much SYN spoofing is still going on. So far spoofing seems now reduced in intensity but still occuring. But my data points are few so more data points and from more established servers than mine would be valuable.
Oaive
···
On 2024-11-08 08:47, tor-relays+tor-relays@queer.cat wrote:
This rule will also count SYN-ACKs sent from your own server to bots trying to connect to your SSH on port 22.
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
I can confirm that the attack has not stopped and that we continue to
monitor spoofed packets with Tor relay's IP addresses including the
addresses of relays that are at our network.
This continues to trigger the sending of reports from the same amateurs.
···
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
Hi! Can you send me (off-list) the details of what you are seeing?
I see several possible scenarios:
(1) The attack stopped in some places but not in others. Or more
specifically, some addresses are no longer being targeted but others
still are.
(2) The attackers moved to some new host and started up the attack again,
but only to some addresses. Or, some new attacker heard about all the
excitement and decided to give it a go.
(3) You are misreading your packets and actually it is more benign than
you think or otherwise we can find an expected explanation for what you
are seeing.
#1 seems unlikely. #2 is definitely possible and we should look for
evidence that it has happened, so we can pull in our friends and allies
to do their work again. I am hoping for #3.
Thanks,
--Roger
···
On Sun, Nov 10, 2024 at 03:15:59AM -0000, tor-operator@urdn.com.ua wrote:
I can confirm that the attack has not stopped and that we continue to
monitor spoofed packets with Tor relay's IP addresses including the
addresses of relays that are at our network.
This continues to trigger the sending of reports from the same amateurs.
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
It’s possible that the attack was filtered upstream, and since you’re closer to the attacker, you might still be seeing those spoofed packets. Also, if you’re noticing spoofed packets coming from your own network, it could indicate a deeper issue. Have you checked if reverse path filtering is enabled?
···
On 9/11/24 23:15, tor-operator@urdn.com.ua wrote:
I can confirm that the attack has not stopped and that we continue to
monitor spoofed packets with Tor relay's IP addresses including the
addresses of relays that are at our network.
This continues to trigger the sending of reports from the same amateurs.
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
A few notes. I don't know if I have missed it but I don't recall seeing bridges mentioned in this discussion.
I too have gotten an abuse message/info/alert from my hosting provider (Nov 8, 03:20 hrs) and I have an OBFS4 BRIDGE, no middle or exit node. And it has always been a bridge, from the initial installation/deploy 5+ years ago.
My server was noted as being "blocked in Russia" earlier on the relay search tor metrics page, I have noted that this info have been removed from the page, I don't know if that is due to the server not being blocked (unlikely?) or the info have been removed from all pages, due top false positives etc(?).
This leads me to wonder if this "DOS attack" is being orchestrated from Russia somehow?
A tor op
···
On Sunday, November 10th, 2024 at 9:36 AM, Roger Dingledine <arma@torproject.org> wrote:
On Sun, Nov 10, 2024 at 03:15:59AM -0000, tor-operator@urdn.com.ua wrote:
> I can confirm that the attack has not stopped and that we continue to
> monitor spoofed packets with Tor relay's IP addresses including the
> addresses of relays that are at our network.
>
> This continues to trigger the sending of reports from the same amateurs.
Hi! Can you send me (off-list) the details of what you are seeing?
I see several possible scenarios:
(1) The attack stopped in some places but not in others. Or more
specifically, some addresses are no longer being targeted but others
still are.
(2) The attackers moved to some new host and started up the attack again,
but only to some addresses. Or, some new attacker heard about all the
excitement and decided to give it a go.
(3) You are misreading your packets and actually it is more benign than
you think or otherwise we can find an expected explanation for what you
are seeing.
#1 seems unlikely. #2 is definitely possible and we should look for
evidence that it has happened, so we can pull in our friends and allies
to do their work again. I am hoping for #3.
Thanks,
--Roger
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
Hi! Can you send me (off-list) the details of what you are seeing?
Done.
The last observation was made Nov. 9 at 11:49 UTC, that is after it was
announced the attacker was shut down.
We no longer see the packets, but we continue to receive reports from
the same mentioned amateurs, the last one is dated 12 Nov 2024 07:57:06
+0800. All mentioned addresses are those of Tor relays, and the
destination port is still ssh.
This could be simple brute force attacks, but since the reporter blocks
the connections, that seems unlikely. Perhaps the attacker tuned the
attack to a list of networks that are known for triggering reports.
(3) You are misreading your packets and actually it is more benign
than you think or otherwise we can find an expected explanation for
what you are seeing.
No misreading; the attack is benign anyway, the problem is really
with the fools that take these reports seriously.
···
_______________________________________________
tor-relays mailing list -- tor-relays@lists.torproject.org
To unsubscribe send an email to tor-relays-leave@lists.torproject.org
At least if it happens again soon,