Relay Started Crashing (Line unexpectedly reached at workerthread_new)

Support Relay Operator
1

Weird one here.

I’ve been running a relay for years. I recently upgraded my tor container from bullseye to bookworm(GitHub - silvenga-docker/tor: Yet another tor image, updated weekly.).

Then, about five days later, it started crashing.

image
image1462×321 98.1 KB

At first I thought this was another DDOS attack on the network, but looking at the logs, the relay is crashing on startup:

tor-tor-1  | Sep 23 16:46:06.000 [notice] Configured hibernation.  This interval began at 2024-09-01 00:00:00; the scheduled wake-up time was 2024-09-01 00:00:00; we expect to exhaust our quota for this interval around 2024-10-01 00:00:00; the next interval begins at 2024-10-01 00:00:00 (all times local)
tor-tor-1  | Sep 23 16:46:06.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
tor-tor-1  | Sep 23 16:46:07.000 [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6.
tor-tor-1  | Sep 23 16:46:07.000 [notice] Configured to measure statistics. Look for the *-stats files that will first be written to the data directory in 24 hours from now.
tor-tor-1  | Sep 23 16:46:07.000 [notice] Bootstrapped 0% (starting): Starting
tor-tor-1  | Sep 23 16:46:08.000 [notice] Starting with guard context "default"
tor-tor-1  | Sep 23 16:46:08.000 [warn] tor_bug_occurred_(): Bug: ../src/lib/evloop/workqueue.c:356: workerthread_new: This line should not have been reached. (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug: Tor 0.4.8.12: Line unexpectedly reached at workerthread_new at ../src/lib/evloop/workqueue.c:356. Stack trace: (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(log_backtrace_impl+0x57) [0x558e40ce1dd7] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_bug_occurred_+0x169) [0x558e40cecd29] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(threadpool_new+0x1dc) [0x558e40e2ab4c] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(cpuworker_init+0xb8) [0x558e40db8a98] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(run_tor_main_loop+0xd0) [0x558e40c639c0] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_run_main+0x1e5) [0x558e40c63fa5] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_main+0x59) [0x558e40c60329] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(main+0x19) [0x558e40c5fee9] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f566d43124a] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f566d431305] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(_start+0x21) [0x558e40c5ff31] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [err] Can't launch worker thread.
tor-tor-1  | Sep 23 16:46:08.000 [warn] tor_bug_occurred_(): Bug: ../src/lib/evloop/workqueue.c:522: threadpool_start_threads: This line should not have been reached. (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug: Tor 0.4.8.12: Line unexpectedly reached at threadpool_start_threads at ../src/lib/evloop/workqueue.c:522. Stack trace: (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(log_backtrace_impl+0x57) [0x558e40ce1dd7] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_bug_occurred_+0x169) [0x558e40cecd29] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(threadpool_new+0x221) [0x558e40e2ab91] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(cpuworker_init+0xb8) [0x558e40db8a98] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(run_tor_main_loop+0xd0) [0x558e40c639c0] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_run_main+0x1e5) [0x558e40c63fa5] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_main+0x59) [0x558e40c60329] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(main+0x19) [0x558e40c5fee9] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f566d43124a] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f566d431305] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(_start+0x21) [0x558e40c5ff31] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] tor_bug_occurred_(): Bug: ../src/lib/evloop/workqueue.c:566: threadpool_new: This line should not have been reached. (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug: Tor 0.4.8.12: Line unexpectedly reached at threadpool_new at ../src/lib/evloop/workqueue.c:566. Stack trace: (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(log_backtrace_impl+0x57) [0x558e40ce1dd7] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_bug_occurred_+0x169) [0x558e40cecd29] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(threadpool_new+0x252) [0x558e40e2abc2] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(cpuworker_init+0xb8) [0x558e40db8a98] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(run_tor_main_loop+0xd0) [0x558e40c639c0] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_run_main+0x1e5) [0x558e40c63fa5] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(tor_main+0x59) [0x558e40c60329] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(main+0x19) [0x558e40c5fee9] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f566d43124a] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f566d431305] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 16:46:08.000 [warn] Bug:     tor(_start+0x21) [0x558e40c5ff31] (on Tor 0.4.8.12 )
tor-tor-1  |
tor-tor-1  | ============================================================ T= 1727109968
tor-tor-1  | Tor 0.4.8.12 died: Caught signal 11
tor-tor-1  | tor(+0xeac9a)[0x558e40ce1c9a]
tor-tor-1  | tor(threadpool_register_reply_event+0x23)[0x558e40e2ad03]
tor-tor-1  | tor(threadpool_register_reply_event+0x23)[0x558e40e2ad03]
tor-tor-1  | tor(cpuworker_init+0xc9)[0x558e40db8aa9]
tor-tor-1  | tor(run_tor_main_loop+0xd0)[0x558e40c639c0]
tor-tor-1  | tor(tor_run_main+0x1e5)[0x558e40c63fa5]
tor-tor-1  | tor(tor_main+0x59)[0x558e40c60329]
tor-tor-1  | tor(main+0x19)[0x558e40c5fee9]
tor-tor-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f566d43124a]
tor-tor-1  | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f566d431305]
tor-tor-1  | tor(_start+0x21)[0x558e40c5ff31]

I’m not quite sure how to proceed with debugging this. Nothing on the host has changed in that time.

My configuration (I’m not using a torrc)

tor \
--Address <removed> \
--ORPort "443 IPv4Only" \
--DirPort "80 IPv4Only" \
--ContactInfo <removed> \
--Nickname <removed> \
--AccountingMax 8TB \
--ExitRelay 0 \
--SocksPort 0 \
--DataDirectory /data \
--User tor \
--MetricsPort 0.0.0.0:9035 \
--MetricsPortPolicy "accept 0.0.0.0" \
--NumCPUs 2
2

Seems like the number of CPUs is not being recognized correctly.
Try to set NumCPUsdirective manually to the number of CPU cores available on your system.

3

Thanks, I’ve updated my relay’s flags to include --NumCPUs 2 (also tried with NumCPUs set to 8, which is the number of vCPU’s assigned to this VM).

I think it’s crashing with the same stack:

tor-tor-1  | Sep 23 22:32:32.000 [warn] tor_bug_occurred_(): Bug: ../src/lib/evloop/workqueue.c:356: workerthread_new: This line should not have been reached. (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug: Tor 0.4.8.12: Line unexpectedly reached at workerthread_new at ../src/lib/evloop/workqueue.c:356. Stack trace: (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(log_backtrace_impl+0x57) [0x55aacc4a2dd7] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_bug_occurred_+0x169) [0x55aacc4add29] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(threadpool_new+0x1dc) [0x55aacc5ebb4c] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(cpuworker_init+0xb8) [0x55aacc579a98] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(run_tor_main_loop+0xd0) [0x55aacc4249c0] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_run_main+0x1e5) [0x55aacc424fa5] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_main+0x59) [0x55aacc421329] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(main+0x19) [0x55aacc420ee9] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f8b606b924a] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f8b606b9305] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(_start+0x21) [0x55aacc420f31] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [err] Can't launch worker thread.
tor-tor-1  | Sep 23 22:32:32.000 [warn] tor_bug_occurred_(): Bug: ../src/lib/evloop/workqueue.c:522: threadpool_start_threads: This line should not have been reached. (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug: Tor 0.4.8.12: Line unexpectedly reached at threadpool_start_threads at ../src/lib/evloop/workqueue.c:522. Stack trace: (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(log_backtrace_impl+0x57) [0x55aacc4a2dd7] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_bug_occurred_+0x169) [0x55aacc4add29] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(threadpool_new+0x221) [0x55aacc5ebb91] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(cpuworker_init+0xb8) [0x55aacc579a98] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(run_tor_main_loop+0xd0) [0x55aacc4249c0] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_run_main+0x1e5) [0x55aacc424fa5] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_main+0x59) [0x55aacc421329] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(main+0x19) [0x55aacc420ee9] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f8b606b924a] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f8b606b9305] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(_start+0x21) [0x55aacc420f31] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] tor_bug_occurred_(): Bug: ../src/lib/evloop/workqueue.c:566: threadpool_new: This line should not have been reached. (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug: Tor 0.4.8.12: Line unexpectedly reached at threadpool_new at ../src/lib/evloop/workqueue.c:566. Stack trace: (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(log_backtrace_impl+0x57) [0x55aacc4a2dd7] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_bug_occurred_+0x169) [0x55aacc4add29] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(threadpool_new+0x252) [0x55aacc5ebbc2] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(cpuworker_init+0xb8) [0x55aacc579a98] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(run_tor_main_loop+0xd0) [0x55aacc4249c0] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_run_main+0x1e5) [0x55aacc424fa5] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(tor_main+0x59) [0x55aacc421329] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(main+0x19) [0x55aacc420ee9] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f8b606b924a] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f8b606b9305] (on Tor 0.4.8.12 )
tor-tor-1  | Sep 23 22:32:32.000 [warn] Bug:     tor(_start+0x21) [0x55aacc420f31] (on Tor 0.4.8.12 )
tor-tor-1  |
tor-tor-1  | ============================================================ T= 1727130752
tor-tor-1  | Tor 0.4.8.12 died: Caught signal 11
tor-tor-1  | tor(+0xeac9a)[0x55aacc4a2c9a]
tor-tor-1  | tor(threadpool_register_reply_event+0x23)[0x55aacc5ebd03]
tor-tor-1  | tor(threadpool_register_reply_event+0x23)[0x55aacc5ebd03]
tor-tor-1  | tor(cpuworker_init+0xc9)[0x55aacc579aa9]
tor-tor-1  | tor(run_tor_main_loop+0xd0)[0x55aacc4249c0]
tor-tor-1  | tor(tor_run_main+0x1e5)[0x55aacc424fa5]
tor-tor-1  | tor(tor_main+0x59)[0x55aacc421329]
tor-tor-1  | tor(main+0x19)[0x55aacc420ee9]
tor-tor-1  | /lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f8b606b924a]
tor-tor-1  | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f8b606b9305]
tor-tor-1  | tor(_start+0x21)[0x55aacc420f31]

(This is after I’ve rebooted to update the kernel)

4

Here is issue link which relates to Sandbox option.
Looks like you are not using it, but maybe this link will give some ideas anyway:

5

Yeah, I did see that one, and yeah, I’m not using any kind of hardened sandbox - I treat the VM as the security boundary.

It’s just odd that is started happening five days after upgrading - no other package was upgraded on the guest in that time, and no updates on the host either.

It almost looks kernel related - I can reproduce it on multiple hosts running 5.4, but works on 6.5 kernels. (I’m assuming libc versions don’t really matter across container boundaries)

6

I’ve located the reason for the SIGSEGV (signal 11). I will post a fix for it on Tor Project’s GitLab. This will prevent Tor from crashing, but won’t solve your issue unfortunately. The warnings will remain.

What you can do is attach strace and post the output here.

7

I’ll do that! Thanks.

8

Success!

More debugging later, I was running into random errors that just made no sense within the container. The issue appears to be around a new seccomp flag that was needed by the newer libc version and my old Docker runtime wasn’t allowing it. The runtime was originally pinned for legacy reasons, and it was just stuck in my automation.

Upgrading Docker resolved the issue from the relay from starting, the kernel wasn’t related at all (it just happened that all the older kernels were using the same pinning automation). It appears to be purely related to upgrading to bookworm, with it’s newer libc 2.36 using a new syscall, along with using a bad version of Docker.

Sorry about the PEBCAK problem. I’m guessing I missed something when building the timeline which shot me down the wrong path.

1 Like