Timed out waiting for client to open data channel

I’m getting a lot of these in the last 3 days and it seems to start the day I had a brownout. Brownout as in power flickers. The internet goes down but not the server since it is on a UPS. I kill the the proxy then restart the modem then the router then restart the proxy from the command line.

I doubt the brownout or IP change. I checked.

INFO

nohup /home/ME/snowflake/proxy/proxy -nat-retest-interval 0 -ephemeral-ports-range "65497:65534" -capacity 9 -unsafe-logging -verbose >> /home/ME/snowflake.log 2>&1 &

Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-136-generic x86_64)

2025/05/01 02:36:38 error polling broker: dial tcp: lookup snowflake-broker.torproject.net on 127.0.0.53:53: server misbehaving
2025/05/01 02:36:38 ----> Brownout so my IP changed.

2025/05/01 02:55:46 snowflake-proxy 2.9.2 (103278d6*)
2025/05/01 02:55:48 NAT Type measurement: unknown -> unrestricted

Those messages jumped from the 450s to over 1000 per day
Am I doing something wrong except nothing has changed on that server. Or is this just the way it works.

What other info can I supply if there is a problem?

1 Like

This might be explained by the fact that you limir the capacity. Before the internet goes out, your proxy is operating at capacity, having some long-standing client connections. They they all go out, and the proxy starts polling more frequently.
Just a hypothesis though.

I suggest using the -metrics flag to get more detailed stats.

But 1000 failed connections per day doesn’t seem extraordinary to me. It’s slightly less than 1 per minute.

1 Like

I eliminated the internet failure as a cause. This year I lost power twice for a about 1 hour and did not see a spike in my timeouts although I did have to restart the server.

Tested -metrics on another server (without a connection on purpose) and I see the output on port 9999 so it might shine a light with more info.

You need maxmind GeoLite2-Country.mmdb, which I already use, but no clue about mmdb-convert.py to convert it to a csv file. Maybe it’s that 12 year old script I see.

Maybe you are right and it’s an overreaction and there is no problem with 1000 timeouts/day but here must be something for it to spike like that. I see no drop in traffic.

For anyone else reading this and wanting to try -metrics, you now need a code to download that maxmind file.

edited later:
Is there anyway to know if the content of http://localhost:9999/internal/metrics lives in the file system somewhere?

I’ll just recompile with the latest GO and latest proxy v2.11.0 - 2025-03-18 and get back.
It can’t hurt, in theory.

1 Like

It’s been a few days and these timeouts are back to my pre internet failure levels. I find it hard to believe that version 2.11.0 is the reason.
I mistakenly associated the timeouts with my internet failure and deduced something was wrong.
They were a fluke and had I checked this stat 3 days before I would have noticed nothing or 3 days from now I would simply have seen a temporary spike.

I’m marking this solved.

The solution is that there is no solution because there was no problem. Why clients fail to connect is unknown. This is just how it works.

The positive is that I am now at the latest version.

2 Likes

I believe the GeoIP DB is not required. The metrics will still work, but without per-country stats.

Snowflake uses Prometheus for these metrics, but I don’t know where it stores the data.

1 Like

Yes you are correct, geoip is not required as I found out. Forgot to install them. It loads the file when you start the proxy as I found out when I did install it and restarted the proxy but without the -metrics parameter as I saw nothing in there which I could use.

TKS for the help

I think the metrics are shown from memory when it starts the web server on localhost:9999

2 Likes

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.