Proof of Work (PoW) defense for Onion Service is released

Support Onion Services
1

We are thrilled to announce that the Proof of Work (PoW) protection for Onion Services is now available for general adoption with the tor 0.4.8.4 stable release.

If you’re an Onion Service operator, your feedback will help us identify issues with this new protection and ensure its reliability.

What is the PoW protection for Onion Service?

Proof of Work (PoW) is a cryptographic mechanism where a computing system can prove to another that they have performed some computational effort.

The Proof of Work (PoW) defense for Onion Services is a way to protect against Denial of Service (DoS) attacks by prioritizing, when under stress, clients that have proved to the service that they performed a number of resource-intensive operations.

It’s a way to prioritize verified effort (but not a way to verify users), which means attackers would have trouble launching many requests to an Onion Service, but users will possibly have resources to do their legitimate requests.

In other words, Onion Services may be configured to offer a Client Puzzle if they’re under heavy load, and to prioritize incoming client connections containing solutions to the puzzle.

For an overview of this new protection, check it’s blog post.

For a deeper explanation about how it works, check the PoW FAQ and Proposal 327.

How to try PoW

If you operate an Onion Service and believe that it may be subject to high traffic or even a DoS attack, you may help Tor by giving feedback about the PoW protection.

To setup the PoW protection, please follow the steps outlined at the Onion Services DoS Guidelines page. This involves:

  1. Using a GPL-covered C Tor binary version 0.4.8.4 onwards (your software distribution may already provide it or you might need to compile it yourself).

  2. Enable the protection for each of your Onion Services with HiddenServicePoWDefensesEnabled 1.

  3. Monitor your services with MetricsPort (be careful to not expose this port publicly) and tools like Prometheus and Grafana.

  4. Tune HiddenServicePoWQueueRate and HiddenServicePoWQueueBurst for each Onion Service as needed.

During DoS attacks, you might also want to increase verbosity on your logs for a short while to help understanding what’s going on. To do that, use a Log configuration like this:

Log info file /var/log/tor/info.log

Submit your feedback

For general questions about PoW, you can leave a comment in this post, or start a new thread.

If you believe that you have found a non-security issue, submit your feedback at the Tor GitLab repository for technical reports. Include a clear description of the problem, your Tor logs, steps to reproduce it, and any relevant details.

In the other hand, if you think you found a security issue, follow the procedure at the Security Policy page in order to report it privately.

Be careful with the data you share in your bug reports (MetricsPort data or log files). When in doubt, don’t share it at first and ask for help on how to clean them.

By testing PoW and reporting any issues, bugs, or suggestions, you will contribute significantly to refining its performance and optimizing its capabilities. Your participation will not only benefit the Tor community but also help advance the Internet freedom community.

5 Likes
2

Hi, excited to see this get released!

I saw that 0.4.8.4 packages have been uploaded to deb.torproject.org, but they don’t appear to have been compiled with the required --enable-gpl option (based on looking at the tor --version output). Is Tor planning to provide GPL-covered Debian packages or are we on our own for that? I’m asking specifically for SecureDrop, which uses the deb.torproject.org packages but would like to enable this.

1 Like
3

GPL-covered packages will be soon available at deb.torproject.org:

2 Likes
4

so POWdefense in version 4.8.7 is auto enabled by default when compiled - no further config lines is mandatory in torrc - is it correct understanding ? it will triggered if n when any DDOS happens
becos i got errors when try insert HiddenServicePoWDefensesEnabled 1 in torrc

5

Do I understand correctly, that for PoW to work, the client also has to be updated? If so, what is stopping the attacker from using an older version?

6

PoW doesn’t help me in action
I’m under GET attack, when PoW is enabled they can take my onion address down (Onion site disconnected) but when PoW is disabled my onion works fine but my nginx crashes
The number of GET requests I get is around 3000 requests per second

7

You can check here if your tor binary has the PoW defense compiled.

Then, if your tor binary has the PoW defense compiled, you still still need to use HiddenServicePoWDefensesEnabled for each Onion Service you want to protect.

8

I’m under GET attack, when PoW is enabled they can take my onion address down (Onion site disconnected) but when PoW is disabled my onion works fine but my nginx crashes

I would suggest you to tune both HiddenServicePoWQueueRate and HiddenServicePoWQueueBurst to fit your existing resources.

You could try to set these values to the highest possible that does not crash your NGINX, and also reducing the proportion of clients that aren’t solving the puzzle with enough effort (and hence getting an “Onion site disconnected” error).

9

Recent clients having PoW enabled would be able to solve puzzles, while older clients won’t.

If an attacker is using an older client, without PoW, they can still try to attack onionsites, but once PoW is enable and activated (i.e, suggested effort is greater than zero), then these attacker’s requests will have lower priority in the queue than those requests coming with PoW puzzle solutions.

In summary, for PoW to be effective against attackers, legitimate clients need to use recent clients with PoW support, but for attackers it should not matter whether they’re using recent clients or not, as their attacks tends to be either ineffective or have an increasing computational cost, with lower returns.

1 Like
10

They will have lower priority but will still incur significant overhead for the service. The whole PoW defense mechanism is fundamentally flawed if there’s no way to say “no puzzle => immediate reject”. The ongoing DDOS is a proof that the mechanism doesn’t work. Hell, I cannot even reach the Tor Project’s GitLab right now to provide some more context.

11

@wyVt78pWBTpJnYt please note that setups using Onionbalance still don’t support PoW fully yet: it will benefit from rate limiting, but currently have no way to communicate the suggested effort back to users.

Full PoW support in Onionbalance is discussed here.

1 Like
12

I stopped onionbalance and changed it to a simple one threaded tor and under attack tor crashes with these errors


[warn] Possible replay detected! An INTRODUCE2 cell with the same ENCRYPTED section was seen 2 seconds ago. Dropping cell.
[notice] Failed to find node for hop #1 of our path. Discarding this circuit.
[warn] Giving up on launching a rendezvous circuit to [scrubbed] for hidden service [scrubbed]
[notice] Our circuit 0 (id: 315713) died due to an invalid selected path, purpose Hidden service: Connecting to rendezvous point. This may be a torrc configuration issue, or a bug.
[err] tor_assertion_failed_(): Bug: src/feature/hs/hs_metrics.c:202: hs_metrics_update_by_ident: Assertion ident_pk failed; aborting. (on Tor 0.4.8.7 )
[err] Bug: Tor 0.4.8.7: Assertion ident_pk failed in hs_metrics_update_by_ident at src/feature/hs/hs_metrics.c:202: . Stack trace: (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(log_backtrace_impl+0x5a) [0x55d10288287a] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(tor_assertion_failed_+0x14c) [0x55d10288d99c] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(hs_metrics_update_by_ident+0xd2) [0x55d10299ce22] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(circuit_build_failed+0x301) [0x55d102908dc1] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(circuit_close_all_marked+0x11f) [0x55d1028f708f] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(+0x6ea9d) [0x55d102809a9d] (on Tor 0.4.8.7 )
[err] Bug:     /lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x20fde) [0x7f2e228f3fde] (on Tor 0.4.8.7 )
[err] Bug:     /lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x52f) [0x7f2e228f487f] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(do_main_loop+0xe1) [0x55d10280adc1] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(tor_run_main+0x195) [0x55d102806705] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(tor_main+0x3e) [0x55d102802c9e] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(main+0x1d) [0x55d10280284d] (on Tor 0.4.8.7 )
[err] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f2e22377083] (on Tor 0.4.8.7 )
[err] Bug:     /home/username/tor-0.4.8.7/src/app/tor(_start+0x2e) [0x55d1028028ae] (on Tor 0.4.8.7 )
13

Thanks again for reporting, @wyVt78pWBTpJnYt.

Sounds like you hit in a C Tor bug. Could you open an issue here? That may be easier for Tor developers to interact and fix.

14

@alphagoo The “no pluzzle solution => no place in the queue” approach was considered during the PoW defense development, but it was ruled out by considering that effort updates would automatically regulate the queue. More context would be helpful to decide whether an additional config option should be added to exclude clients offering no puzzle when effort is greater than zero.

The ongoing DoS is still being investigated, but does not seem to be the case it’s bypassing PoW. Instead, it may be exploiting some unrelated bug.

1 Like