Hi everyone,
Hello TinCan!
dgoulet here, from the Tor network team. First, thanks for this report, very
useful!
We are currently working on several fixes for this ongoing BEGIN_DIR DDoS badly
affecting the network. Next release (soon!) will most likely have them in.
It will be a fun one because to be effective, we will need most of the relays
to upgrade so to avoid degradation.
I have couple questions for you.
The attack started targeting my relay yesterday at 12:00 UTC, 15 hours after
the relay achieved the HSDir flag. This time I had implemented a rate cap (on
version 0.4.9.14) for BEGIN_DIR which allowed me to try out a few different
things. The rate limiter has a configurable rate/burst limit and two
configurable defense types:
Either
- Refuse the stream by sending END_STREAM_REASON_NOTDIRECTORY back to the offender
Or
- Close the offending circuit by sending -END_CIRC_REASON_RESOURCELIMIT back to the offender
At the beginning of the attack I was limiting BEGIN_DIR requests to 200/s and
refusing anything beyond that. The attacker was spawning 300 circuits per
second on average and hitting me with ~55,000 BEGIN_DIR requests per second
on average. After 8 hours of being attacked the relay logged "1647224804
BEGIN_DIR rejected". During these first 8 hours, the relay remained
responsive, with memory usage staying flat at or under 1000MB, main thread
CPU usage 20-80% and traffic 200-850 Mbps with a bandwidth ratio of around
2:1 down:up. CPU usage was highly correlated with the amount of traffic being
moved. There were also occasional latency spikes while the bandwidth was
close to the Gbit line rate.
Are you able to tell me if these BEGIN_DIR are for onion service descriptors as
in your relay being an HSDir? Or they are normal regular BEGIN_DIR requests
from a client asking for other directory information (ex: consensus)?
Also, that limit you put in is global to all circuits or per circuit?
At 8 hours into the attack I reloaded Tor with the other defense type (close
BEGIN_DIR circuits) and lowering the limit to 100 requests per second.
Bandwidth usage immediately dropped to 80-100 Mbps in both directions and CPU
usage plummeted to <10%. The attack now changed its strategy and started
creating circuits at an increasing rate, topping out at 1500 circuits per
second, still no match for 12 Ryzen cores though. At the time of writing the
attacker is still spawning around 1000 circuits per second and the main
thread is at 10% of one core with the circuit work being spread out over the
rest of the cores.
The circuit rate here is wild! We have DoS defenses in place for rate limiting
circuit creation per IP address so are these 1000/sec circuits coming from 1000
different addresses?
Note here that if they come from tor clients, you will have a single
**unauthenticated channel** (tor channel) per client IP address. This would
imply they are a single hop request to your relay so client -> your relay. And
thus generic directory requests such as consensus fetch for instance.
If they come from a relay, it would be on an **authenticated** channel and this
means these BEGIN_DIR request come from a multi-hop circuit implying onion
service descriptor get/post.
I'm giving you these details because if you point a LLM at your 500MB debug
logs ;), it is possible to tell if the channel those circuits are on is
authenticated vs not.
The difference is very important here because in both cases, we will apply
different fixes. And it would be very useful to learn what you are seeing.
Thanks a lot for your help! If you want to chat, I'm "dgoulet" on our #tor-dev
IRC channel which you can access via Matrix as well.
Cheers!
David
···
On 22 Oct (09:00:27), TinCan via tor-relays wrote:
--
Fwj9yfk1MF4rtih0k2x/SemjPPhGxtNXDVaAx9n7TgQ=