Hi, here are the meetup notes.
The next Tor Relay Operator Meetup will happen on November 19, 19:00 - 20:30 UTC.
Thanks everyone for joining us!
Gus
Relay Operator Meetup - 2022-10-22 @ 19 UTC (online)
* Recap of the agenda
* Announcements:
* End Of Life: relays running tor 0.4.6.x.
Some weeks ago we rejected 300 or 400 relays that are running this
end-of-life version. As soon as a Tor version series goes end-of-life,
there is a risk there are unfixed security vulnerabilities. The general
policy is to try to contact everyone, give them a couple of weeks to
update. We got some hundred to upgrade, but also a fair amount didn't.
If they contact us to explain that they upgraded, we remove them
from the blocklist. The rejections we do here are only by relay
fingerprint, not by IP address, so if they spin up a new relay they auto
come back into the network.
There's a tradeoff here because the network is under load and Tor is
actively being useful to people in Russia and Iran, so we aren't yet
being too aggressive.
One other exception is moria1, Roger's directory authority, which is
still running an older version because bug
Connection DDoS defenses never applied to DirPort so dir auths still impacted (#40622) · Issues · The Tor Project / Core / Tor · GitLab remains open.
Servers – Tor Metrics is where you can see
the short drop.
* Tor and Snowflake in Iran
In Sept, many protests began in Iran. Iran is active in censorship
not just of destination websites and protocols but also of circumvention
tools like Tor / Orbot.
People use Tor a lot in Iran over the past month. Vanilla Tor
connections are not working, so you need to use a pluggable transport
like obfs4 or Snowflake.
Snowflake is a cool new pluggable transport where everybody can be a
proxy in an easy way. Just install a browser extension, or install the
headless proxy.
Many new Snowflake volunteers in Sept: mid Sept 30k proxies, now
110k proxies!
Snowflake was working well, users were happy, volunteer proxies
happy too.
...Until Oct 4, when we saw a huge drop in connections. Just this
past week we have finally understood how the block worked. The next
Orbot gets around the block.
Tor Browser for computers was not effected by this block.
Read more:
- Snowflake proxy metrics:
Sources – Tor Metrics
- Tor censorship in Iran:
Tor censorship in Iran (#96) · Issues · The Tor Project / Anti-censorship / Team · GitLab
- User counts from Iran:
Graphs of user counts from Iran since the onset of shutdowns
- Net4people thread - "Unexplained drop in Snowflake client polls and bandwidth, testers wanted":
https://github.com/net4people/bbs/issues/131
* Bridge: obfs4proxy package update
#1021911 - obfs4proxy: preserve user capability overrides on upgrade - Debian Bug report logs
It's available on debian backports, so to get it you need to enable
backports. It's not yet decided whether we will kick out un-updated
bridges eventually.
Survey: how many people here are running bridges, running Debian,
and using backports?
Several people who use Debian say that no they don't use
backports
* Tor Weather rewrite status
Tickets:
- Set up machine/VM to run Tor Weather on (#40893) · Issues · The Tor Project / TPA / TPA team · GitLab
- Home · Wiki · The Tor Project / Network Health / Tor-Weather · GitLab
Google Summer of Code project to revive Tor Weather, which is a
tool to notify relay operators of downtime or other issues with their relays.
The project is over, and the student has done well.
Q: Sometimes we get alerts already from some tor-weather thing.
That one is run by another person right?
A: Yes, that one is run by third-party volunteers somewhere on the
internet.
* State of DoS attack:
* Provide a recommended set of iptables/nftables rules to help in case of DoS attacks
See: Provide a recommended set of iptables/nftables rules to help in case of DoS attacks (#40093) · Issues · The Tor Project / Community / Support · GitLab
* Provide a set of commands that would gather data which could help you understand the attack
There are several different kinds of overload going on.
The first one visible on the overload graph started mid-July and caused
a huge spike in overloaded non-exit relays. The performance impact was
limited at that point.
The other one that started in mid Sept mostly affected exit relays. It
is unfortunately visible in our performance metrics. Tor is slower
because of it. 
One challenge we recognize here is that both users and relay operators
lack visibility into what is going on and what we're doing to handle it.
We worry that the attackers are watching our irc discussions and that
means if we are more quiet in our work then it's harder for our
community to follow along.
Q: Did proof-of-work get implemented to mitigate ddos attacks?
A: prop327: Implement PoW over Introduction Circuits (#40634) · Issues · The Tor Project / Core / Tor · GitLab has a
working proof-of-concept of the proof-of-work idea. but much work
remains.
But also remember that it only aims to resolve one of the several types
of attacks we are seeing.
This is a top priority, even though there isn't as much visibility as
we'd like, and even though developers are distracted with funded work.
* New Network Health project
There is a new funded project, woo! The idea is:
1, improve health of the Tor network
2, combat malicious relays.
3, relay dynamics of the network
We've been working the past years on removing and rejecting relays,
but the next better step is to look to the community and see how we can
steer and improve the relay community.
We have our "expectations for relay operators" document, and we want
to start from there and continue to do more things like it.
- Expectations for Relay Operators · Wiki · The Tor Project / Community / Team · GitLab
- https://gitlab.torproject.org/tpo/community/relays/-/issues/18
- Project: Sponsor 112 : Combating Malicious Relays · The Tor Project · GitLab
The objective of the community side of this project is to *find
solutions*. For example, should we demand a valid ContactInfo on relays?
There are genuine disagreements about what to require and what to
expect. We need to understand what the community is expecting so we can
make good choices.
Another objective: trying to understand the network relay dynamics
better. What kind of relays join and leave? What counts as 'normal'
there so we have a chance to recognize anomalies?
We lack tools to automatically analyze and understand these
dynamics, so we need to re-do the metrics infrastructure to have more
useful database structures, so we can do the right queries easily.
Once we have that we can start annotating relays and relay groups.
Do we know the operator of that relay? Of that group of relays? Does it
seem similar to that other group of relays that we know something about?
Ideally we will support annotations from several different angles,
to start recording what we know and what we don't know about the
network.
Links:
- OrNetRadar 2.0 OrNetRadar | Monitoring the Tor Network for new Relay Groups and Events
- OrNetStats | OrNetStats
- Visualize and track impact of customizable relay groups (#40001) · Issues · The Tor Project / Network Health / Metrics / Relay Search · GitLab
Looking at network health *tools* for automatically understanding
e.g. exit blocking.
Lastly, there are many ways that relays can deceive the network, and
we want to understand those better. For example there are known ways
that relays can inflate the amount of traffic they get from clients, and
there are known designs to make that harder but they are still in the
research phase.
We will probably do some of our new changes directly in Arti, the
rust Tor rewrite.
The project is 22 months of work and 6 months of monitoring
afterward, so we are excited!
* Next Tor Relay Operator Meetup
* November 19 at 19:00 - 20:30 UTC
* Relay operator meetup at rC3? (no rC3 planned yet)
Plan: let's do next one Nov 19, and then if RC3 happens Leibi will
organize a meet-up there too.
* Q&A -- write all of your questions here!
- Any plans for something like TorConf? virtual is okay, at least users
can mass donate on the live stream. Also keep a monero wallet too and
expect some donations to come 
Yes! There will be a "State of the Onion" streaming event Nov 9 (core
Tor teams) and Nov 16 (community orgs). Stay tuned for the announcement.
- Are we sure those iptables rules don't produce false positives? e.g.
onion services?
This is a great question and one that has been worrying me too.
Especially since many people are deploying many different rules, it
seems likely that some people are over-blocking. All the more reason to
centralize on a recommended good set, in that ticket.
- Would it make sense to send abuse reports to the handfull of ISPs that
are the source of the DoS? If yes how to generate Logs for the abuse
report?
Do any of you have contacts at OVH or Hetzner who could help us from
inside the ISPs? That is, do you have a friend who works there? We could
use some better community contacts here.
Also, be very careful with your logs and your reporting, because some
of those might be (probably are) legit Tor clients and you could be
putting them at risk.
- Any plans to lower DoSConnectionMaxConcurrentCount ? (asking because
of
reports that relays not obeying DoSConnectionMaxConcurrentCount (#40636) · Issues · The Tor Project / Core / Tor · GitLab)
(Currently it is at DoSConnectionMaxConcurrentCount=50 in the
consensus) What this param does is: if at any point one IP address has
50 connections open to you, you drop all future connections from it for
the next hours.
Complicated, but I guess the simplistic answer is "no there are no
plans at this moment." Let's continue on that ticket!
- Are relay to relay connections mutually authenticated? Can I tell
which relay connects to my exit without looking at the source IP? Can
I as an exit operator differentiate client from relay connections?
Yes they are mutually authenticated, and Tor knows which are which but
it isn't easy to check at the system (e.g. netstat) level. If you're
interested in looking at it at the code level, Roger or others can help
after this meetup. See how we use nodelist_probably_contains_address()
for a start.
- Please work on the moderation queue of
gitlab/anonticket.onionize.space - new notes remain "pending for
review" for a long time recently
Gus will take a look at it and see how he can help. Thanks for the
reminder!
- Outbound flooding via exits is rather effective to kill even large
exits, currently this does not look like an attack against exits and
is easy to mitigate - single destination IP only. Any idea on how to
deal with outbound flooding to random destinations?
We've been pondering doing rate limiting *per circuit* at exits, i.e.
if a single circuit makes too many exit requests, or too many requests
to a specific destination, we stop that circuit from being able to make
more. That defense would work for that particular exit at that moment,
but would it result in the same total number of exit streams happening,
just spread out over more exits and also more circuit creation pain?
This question is why we haven't built that feature yet.
- Does Snowflake always route to Tor after it goes through the initial
proxy? Just worried about abuse reports on my home network if not.
Yes, Snowflake routes everything into Tor. You are not an exit node
and don't need to worry about that.
- I run a obfs4 bridge and the users by country are russia --> US -->
Iran, why is US above Iran?
We aren't sure! One possibility is that plenty of people in the US are
using bridges too. But we are also beginning to suspect that our geoip
databases are wrong, especially about Iran. That is, those US users
might actually be users in countries that the geoip databases don't care
about.
- To fix the Iran censorship: so the standalone snowflake does not need
an update ?
Correct, for this particular block, we only needed to update the
client. But please do keep your standalone proxy up to date in general!
- still no deb packages for snowflake?
Meskio has been working on that package. But I think it is not yet
available? There is one for FreeBSD but not Linux?
- why no progress on
Enable IP_BIND_ADDRESS_NO_PORT if supported (!579) · Merge requests · The Tor Project / Core / Tor · GitLab
- I'll ping dgoulet on Monday, I guess this fell through the cracks,
sorry --GeKo
- What is the state of the new family feature?
It is still in the general plans, but nobody is working on it
currently.
- Will arti 2.0 make IPv6-only relays possible?
First answer, arti 2.0 is in the distant future, there are no funders
for it and thus no timeframe and no specifics.
Second answer, one of the goals of Arti is to make it easier for us to
improve Tor. So in the glorious future once we have a funder for
relay-support-in-Arti and a specific plan and then we have done that
plan, then yes I hope we will proceed to do all of the other pending Tor
fixes we have been needing. But it will be a while.
- Is there a new board member?
No, not yet. We have many good candidates and we are still working
through them. Last I heard, we plan to pick more than one.
- If i have a stand-alone machine and want to help Iran is obfs4 or
snowflake better?
(what are the benefits of snowflake/obfs4 for a dedicated machine)
A: A standalone snowflake is a good pick at this point, since it will
reach more users more scalably. Both of them are useful in general, and
which is more useful could change from one week to the next depending on
blocking by the censor.
- Team Cymru's threat monitor collects lots of information that also
cover Tor IPs. Do they collect information on relays?
We talked to them and asked exactly this question, and they said no in
a way that made sense to us. They definitely have a mess on their hands
in terms of public relations, and also they do seem to have some
genuinely dangerous products. But they don't seem to focus on Tor in
particular in any way, and also there are some individuals there who
value Tor and don't *want* to have their products attack it. So, we
should continue being wary, and also remember there are many other
threat intelligence databases out there that seem at least as scary.
On Sat, Oct 22, 2022 at 03:29:44PM -0300, gus wrote:
Hello,
We're meeting today, October 22, at 19UTC (in 30 minutes).
See you soon!
Gus
On Mon, Oct 10, 2022 at 05:46:09PM -0300, gus wrote:
> Dear relay operators,
>
> The next Tor relay operator meetup will happen on Saturday,
> October 22 at 19.00 UTC.
>
> ## Where
>
> BigBlueButton: Tor Relay Operator Meetup
>
>
> ## Agenda (WIP)
>
> * Announcements
> * Tor and Snowflake in Iran
> * EOL rejection
> * State of DoS attack
> * New network health project
> * Q&A
> * Next Tor Relay Operator Meetup
>
> Meeting pad: Riseup Pad
>
> Everyone is free to bring up additional questions or topics at the
> meeting itself.
>
> ## Registration
>
> No need for a registration or anything else, just use the room-link
> above. We will open the room 10 minutes before so you can test your mic
> setup.
>
> Please share with your friends, social media and other mailing lists!
>
> cheers,
> Gus
> --
> The Tor Project
> Community Team Lead
--
The Tor Project
Community Team Lead
--
The Tor Project
Community Team Lead