2 relay per IP limit / how to use more bandwidth

As far as I understand, no.

Yes, but this is a long answer so bear with me.

I have asked a very similar question in the last tor relay operator meetup a short while ago. Spinning up high-bandwidth nodes outside Europe (especially in Asia and on the US West Coast) appears to be a challenge. No matter how much computing power and/or bandwidth I threw at the problem, the results were underwhelming.

The response I got is that it has to do with network latency. A large portion of the network is located in Europe with the 10 largest AS by Consensus Weight making up just over 50% alone all being in Europe (source: OrNetStats | OrNetStats ). Even if you have tons of bandwidth, it will be difficult to to get good utilization if you are “far away” from a majority of the network.

Why is that the case? Because of the way the network measures it’s relays. Long story short: It’s old and assuming of the way the internet was 10+ years ago, but it is working. I understand that there are some improvements on the way, but it will take time (years).

How did I deal with it? I first tried the same as you - more relays per host. I got a /28 IPv4 address block for each server, ran 2 relays per IP and got “decent” overall utilization across all of them. Not great, not terrible.

What has worked much better for me is to run the relays on a fast host in Europe for a couple of weeks, getting a high consensus and advertised bandwidth rate while doing so, and then transferring them back to my server in the US. If the US server is able to handle the increased load, it will pretty much stay there.

Example (server has a 10 Gbit/s link):

 eno1  /  daily

          day        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     2021-12-12     3.07 TiB |    3.10 TiB |    6.17 TiB |  627.84 Mbit/s
     2021-12-13     3.23 TiB |    3.28 TiB |    6.52 TiB |  663.39 Mbit/s
     2021-12-14     4.03 TiB |    4.08 TiB |    8.11 TiB |  825.78 Mbit/s
     2021-12-15     4.23 TiB |    4.29 TiB |    8.52 TiB |  867.33 Mbit/s
     2021-12-16     3.04 TiB |    3.10 TiB |    6.13 TiB |  624.52 Mbit/s
     2021-12-17     3.07 TiB |    3.11 TiB |    6.18 TiB |  629.40 Mbit/s
     2021-12-18     3.02 TiB |    3.05 TiB |    6.07 TiB |  618.05 Mbit/s
     2021-12-19     3.32 TiB |    3.34 TiB |    6.65 TiB |  677.30 Mbit/s
     2021-12-20     3.56 TiB |    3.58 TiB |    7.14 TiB |  727.36 Mbit/s
     2021-12-21     3.41 TiB |    3.43 TiB |    6.85 TiB |  697.23 Mbit/s
     2021-12-22     4.16 TiB |    4.18 TiB |    8.34 TiB |  848.84 Mbit/s
     2021-12-23     6.67 TiB |    6.71 TiB |   13.38 TiB |    1.36 Gbit/s
     2021-12-24     6.51 TiB |    6.55 TiB |   13.06 TiB |    1.33 Gbit/s
     2021-12-25     6.74 TiB |    6.78 TiB |   13.52 TiB |    1.38 Gbit/s
     2021-12-26     8.35 TiB |    8.41 TiB |   16.76 TiB |    1.71 Gbit/s
     2021-12-27    10.01 TiB |   10.11 TiB |   20.12 TiB |    2.05 Gbit/s
     2021-12-28     9.77 TiB |    9.87 TiB |   19.64 TiB |    2.00 Gbit/s
     2021-12-29    10.59 TiB |   10.67 TiB |   21.26 TiB |    2.16 Gbit/s
     2021-12-30    11.36 TiB |   11.39 TiB |   22.75 TiB |    2.32 Gbit/s
     2021-12-31    10.97 TiB |   11.05 TiB |   22.02 TiB |    2.24 Gbit/s
     2022-01-01    12.45 TiB |   12.56 TiB |   25.01 TiB |    2.55 Gbit/s
     2022-01-02    13.08 TiB |   13.21 TiB |   26.29 TiB |    2.68 Gbit/s
     2022-01-03    13.06 TiB |   13.19 TiB |   26.25 TiB |    2.67 Gbit/s
     2022-01-04    12.72 TiB |   12.86 TiB |   25.58 TiB |    2.60 Gbit/s
     2022-01-05    13.25 TiB |   13.40 TiB |   26.65 TiB |    2.71 Gbit/s
     2022-01-06    12.34 TiB |   12.47 TiB |   24.80 TiB |    2.53 Gbit/s

=> The steady-state I could achieve by running a ton of relays on one US-based machine was somewhere around 6 to 8 TB per day.
=> On Dec 22nd and on Dec 26th I transferred some high-bandwidth fingerprints from my European hosts to this server. Each time, I could see a big jump in utilization. First it doubled to around 13 TB /day and then it jumped the 2nd time to over 20 TB/day. Since the server can handle the load, it stayed.

If you have only one public IP and transfer two high-consensus & bandwidth relays, I’d expect you’d be able to push up to 1.0 Gbit/s - assuming everything else checks out. At that point, you will probably be limited by the single-core performance of your Xeon CPU. Tor is mostly single-threaded.

5 Likes