The bridge is running on this device with OpenWrt firmware. After a week or so the tmp space in RAM with a size of 494 MB fills up completely. This is annoying because it stops other applications from working as they can’t use tmp space anymore.
Every time this happens I restart tor and clean up the tmp space. Turns out that especially cached-descriptors and the diff-cache folder inside /tmp/lib/tor grow very fast. Only two days after a restart cached-descriptors already uses 127.2M and diff-cache 95.6M.
I’ve been lowering MaxMemInQueues without any success. It looks like the tmp space gets full even more quickly. Currently MaxMemInQueues is set to 112 MB which should be more than enough given the tmp space size of 494 MB.
Is MaxMemInQueues the only way to handle this problem or is there a better way?
AFAIK there are two ways of approaching this issue. Tweaking TOR and configuring linux. Feel free to comment if you think you can help with the first, I think the latter offers at least these options:
Assign a limited tmpfs space to TOR
Restart TOR regularly
Delete diff-cache files
Option 1 is easy to implement:
mount -t tmpfs -o size=350M tmpfs /tmp/lib/tor
Pro: TOR doesn’t affect other applications anymore
Con: TOR keeps crashing
Option 2 is probably the easiest to implement and the only one that really solves the issue. The only question is: how frequently?
On option 3, looking at their timestamp the oldest files in /tmp/lib/tor/diff-cache just seem to be lingering around. It looks like TOR just didn’t clean them up. Anyone knows if removing them harms TOR?
I tried pruning the /tmp/lib/tor/diff-cache/ folder by removing files older than one day:
find /tmp/lib/tor/diff-cache -type f -mtime 1 -exec rm {} \;
but almost five days on Tor starts complaining about their absence:
Apr 12 19:13:43.000 [warn] Unable to unlink "/var/lib/tor/diff-cache/1000" while removing file: No such file or directory
Apr 12 19:13:43.000 [warn] Unable to unlink "/var/lib/tor/diff-cache/1001" while removing file: No such file or directory
Apr 12 20:38:35.000 [warn] Unable to unlink "/var/lib/tor/diff-cache/1002" while removing file: No such file or directory
Apr 12 21:47:34.000 [warn] Unable to unlink "/var/lib/tor/diff-cache/1008" while removing file: No such file or directory
Apr 12 22:48:37.000 [warn] Unable to unlink "/var/lib/tor/diff-cache/1005" while removing file: No such file or directory
Clearly Tor uses more tmp space than what is available causing it to fill and render useless for other applications (and itself).
So my question is: now what? This is a bug, right?
The question is rather, why your /tmp is a tmpfs. It shouldn’t be.
Did you modify this or is it a default on OpenWRT firmware?
Can you post the output of the command df?
I didn’t verify this, but I’m sure that MaxMemInQueues doesn’t involve data written to the file system. And tmpfs is a file system. So changing MaxMemInQueues won’t have an impact in your case. Also this is not a bug in tor then.
The best solution is to create a swap file as I already mentioned, so the OS can swap out data used less frequently from RAM to flash memory. Have you tried it? If it is not possible on your firmware, then change /tmp to be ext4 or something other than tmpfs and retry then.
Edit:
Can you explain why your tor’s DataDirectory location is /tmp/lib/tor? It should be /var/lib/tor. Try to change this first.
I agree. MaxMemInQueues in /etc/tor/torrc successfully controls RAM usage.
But tmpfs is also in RAM and is insufficient for all the files Tor tries to store. At least on this 1 GB RAM router. And there is no mechanism to control this.
I haven’t because on an embedded device the swap file would be in flash memory (USB flash or SD card in this case) which would make it painfully slow. And it wouldn’t solve the tmpfs problem.
/var/lib/tor and /tmp/lib/tor are the same on OpenWrt because of a symbolic link from /var to /tmp in the / folder.
I tried moving /var/lib/tor to /root which is an ext4 fs on SD card and is big enough. This works flawlessly.
So why is /var/lib/tor installed in tmpfs? I suspect they do this to protect the flash memory. It has a limited number of write cycles. Tor stores a considerable number of files in /var/lib/tor several times every hour, 24 hours a day. There is no HDD so this would quickly wear out the flash memory and reduce life expectancy. IMHO this is a OpenWrt port problem.
Therefore the question remains if there is a better solution than:
DataDirectory should be on permanent storage. It is not intended to be in RAM. You will lose all the information your bridge has written so far on power loss and whenever you shutdown or restart your device.
So why don’t you use it this way? When the SD card gets broken (don’t expect it to be soon, it will take a long time), you could simply replace it with a new one and the internal storage wouldn’t be affected.
Use
AvoidDiskWrites 1
in torrc (if you don’t already) to reduce the amount of writes to flash memory.
it is not the magic wand. It doesn’t reduce disk space or the number of files created. According to the few pages that describe this option it should especially affect the way Tor behaves in the event of hibernation.
This router does not have internal storage. The firmware runs off the SD card. So the problem is not the cost of a new SD card, it’s a crashed production router waiting for the firmware to be installed onto a new SD card.
BTW, don’t underestimate the risk of SD cards getting corrupt. It happened in the same router last year and it was an unpleasant experience.
Long story short. Having to choose between moving /var/lib/tor to SD card and increasing the risk of its corruption by doing so or leaving it in tmpfs and:
Thank you for taking the time to help. Here is torrc:
# https://community.torproject.org/relay/setup/bridge/archlinux/
# https://2019.www.torproject.org/docs/tor-manual.html.en
## Tor opens a socks proxy on port 9050 by default
## Bind to this address:port too for local application connections:
SocksPort 192.168.0.1:9100
HTTPTunnelPort 192.168.0.1:9200
BridgeRelay 1
ExitRelay 0
Exitpolicy reject *:*
Log notice file /var/log/tor/notices.log
DataDirectory /var/lib/tor
User tor
# Replace "TODO1" with a Tor port of your choice. This port must be externally
# reachable. Avoid port 9001 because it's commonly associated with Tor and
# censors may be scanning the Internet for this port.
# ORPort TODO1
ORPort 29013 IPv4Only
BandwidthRate 1 MBytes # Throttle traffic to 8 Mbps
BandwidthBurst 2 MBytes # But allow bursts up to 16 Mb
ServerTransportPlugin obfs4 exec /usr/bin/obfs4proxy
# Replace "TODO2" with an obfs4 port of your choice. This port must be
# externally reachable and must be different from the one specified for ORPort.
# Avoid port 9001 because it's commonly associated with
# Tor and censors may be scanning the Internet for this port.
# ServerTransportListenAddr obfs4 0.0.0.0:TODO2
ServerTransportListenAddr obfs4 0.0.0.0:29014
# Local communication port between Tor and obfs4. Always set this to "auto".
# "Ext" means "extended", not "external". Don't try to set a specific port
# number, nor listen on 0.0.0.0.
ExtORPort auto
# Replace "<address@email.com>" with your email address so we can contact you if
# there are problems with your bridge. This is optional but encouraged.
ContactInfo <tor@xxxxxxxxxx>
# Pick a nickname that you like for your bridge. This is optional.
Nickname FortunaBridge
# Limit memory usage
MaxMemInQueues 160 MB
# Additional Settings for Speed and Security
AvoidDiskWrites 1
MaxConsensusAgeForDiffs N minutes|hours|days|weeks
When this option is nonzero, Tor caches will not try to generate
consensus diffs for any consensus older than this amount of time.
If this option is set to zero, Tor will pick a reasonable default
from the current networkstatus document. You should not set this
option unless your cache is severely low on disk space or CPU. If
you need to set it, keeping it above 3 or 4 hours will help clients
much more than setting it to zero. (Default: 0)
