Anyone seeing big difference in Snowflake proxy utilization after upgrade to 2.10.1?

Support Censorship Circumvention
, ,
1

I saw this post:

It claims that Snowflake proxy usage went to the moon after an upgrade from 2.9.2 to 2.10.x. However, I personally do not see a commit that could explain such a huge difference. Here is comparison of the 2 versions. Apart from a bunch of dependency updates, I singled out these commits:

But IMO none of them should be that impactful.

So, the question is: does anyone else see huge differences in traffic / utilization after the upgrade from 2.9.2 to 2.10.x?

2

Are your proxies by any chance now “Unrestricted NAT” and before they were recognized as “Restricted” ?

Here the update(s) of Snowflake proxies don’t show any breaking changes in the metrics.

3 Likes
3

I do not see changes that should affect NAT checking on the proxy side. Also I would expect the traffic difference to be much higher (though let’s keep in mind that they posted aggregate metrics for several proxies).

4

Hi,

I’m from Unredacted.

I don’t recall seeing this in the logs, and I believe they were unrestricted before. None of our snowflake proxies are running behind NAT, and they’re all on dedicated IPs.

2 Likes
5

Not much data collected yet and difference is not huge, but I see it:

$ grep "completed\|starting" /d/snowflake/log.txt | tail -49
2024/11/14 19:54:19 In the last 1h0m0s, there were 61 completed connections. Traffic Relayed ↓ 1047545 KB, ↑ 166131 KB.
2024/11/14 20:54:19 In the last 1h0m0s, there were 45 completed connections. Traffic Relayed ↓ 1974041 KB, ↑ 344890 KB.
2024/11/14 21:54:19 In the last 1h0m0s, there were 35 completed connections. Traffic Relayed ↓ 1656270 KB, ↑ 125317 KB.
2024/11/14 22:54:19 In the last 1h0m0s, there were 19 completed connections. Traffic Relayed ↓ 897964 KB, ↑ 98472 KB.
2024/11/14 23:54:19 In the last 1h0m0s, there were 41 completed connections. Traffic Relayed ↓ 439852 KB, ↑ 52340 KB.
2024/11/15 00:54:19 In the last 1h0m0s, there were 39 completed connections. Traffic Relayed ↓ 755063 KB, ↑ 55532 KB.
2024/11/15 01:54:19 In the last 1h0m0s, there were 34 completed connections. Traffic Relayed ↓ 999981 KB, ↑ 87807 KB.
2024/11/15 02:54:19 In the last 1h0m0s, there were 33 completed connections. Traffic Relayed ↓ 695931 KB, ↑ 47144 KB.
2024/11/15 03:54:19 In the last 1h0m0s, there were 26 completed connections. Traffic Relayed ↓ 799956 KB, ↑ 50434 KB.
2024/11/15 04:54:19 In the last 1h0m0s, there were 52 completed connections. Traffic Relayed ↓ 971213 KB, ↑ 70468 KB.
2024/11/15 05:54:19 In the last 1h0m0s, there were 55 completed connections. Traffic Relayed ↓ 473815 KB, ↑ 56776 KB.
2024/11/15 06:54:19 In the last 1h0m0s, there were 46 completed connections. Traffic Relayed ↓ 659812 KB, ↑ 91565 KB.
2024/11/15 07:54:19 In the last 1h0m0s, there were 39 completed connections. Traffic Relayed ↓ 1616590 KB, ↑ 108245 KB.
2024/11/15 08:54:19 In the last 1h0m0s, there were 27 completed connections. Traffic Relayed ↓ 780198 KB, ↑ 106271 KB.
2024/11/15 09:54:19 In the last 1h0m0s, there were 44 completed connections. Traffic Relayed ↓ 541960 KB, ↑ 95502 KB.
2024/11/15 10:54:19 In the last 1h0m0s, there were 61 completed connections. Traffic Relayed ↓ 578765 KB, ↑ 59398 KB.
2024/11/15 11:54:19 In the last 1h0m0s, there were 55 completed connections. Traffic Relayed ↓ 712707 KB, ↑ 75116 KB.
2024/11/15 12:54:19 In the last 1h0m0s, there were 46 completed connections. Traffic Relayed ↓ 749133 KB, ↑ 177799 KB.
2024/11/15 13:54:19 In the last 1h0m0s, there were 39 completed connections. Traffic Relayed ↓ 1003583 KB, ↑ 127254 KB.
2024/11/15 14:54:19 In the last 1h0m0s, there were 45 completed connections. Traffic Relayed ↓ 506639 KB, ↑ 59460 KB.
2024/11/15 15:54:19 In the last 1h0m0s, there were 53 completed connections. Traffic Relayed ↓ 1548249 KB, ↑ 136643 KB.
2024/11/15 16:54:19 In the last 1h0m0s, there were 65 completed connections. Traffic Relayed ↓ 1838542 KB, ↑ 165871 KB.
2024/11/15 17:54:19 In the last 1h0m0s, there were 45 completed connections. Traffic Relayed ↓ 1438460 KB, ↑ 179130 KB.
2024/11/15 18:54:19 In the last 1h0m0s, there were 55 completed connections. Traffic Relayed ↓ 1677574 KB, ↑ 168384 KB.
2024/11/15 19:54:19 In the last 1h0m0s, there were 67 completed connections. Traffic Relayed ↓ 1303920 KB, ↑ 185785 KB.
2024/11/15 20:54:19 In the last 1h0m0s, there were 46 completed connections. Traffic Relayed ↓ 1318863 KB, ↑ 137266 KB.
2024/11/15 21:54:19 In the last 1h0m0s, there were 61 completed connections. Traffic Relayed ↓ 976954 KB, ↑ 105706 KB.
2024/11/15 22:54:19 In the last 1h0m0s, there were 49 completed connections. Traffic Relayed ↓ 1400569 KB, ↑ 131866 KB.
2024/11/15 23:54:19 In the last 1h0m0s, there were 45 completed connections. Traffic Relayed ↓ 1481322 KB, ↑ 98709 KB.
2024/11/16 00:54:19 In the last 1h0m0s, there were 28 completed connections. Traffic Relayed ↓ 606246 KB, ↑ 47642 KB.
2024/11/16 01:54:19 In the last 1h0m0s, there were 13 completed connections. Traffic Relayed ↓ 22339 KB, ↑ 7261 KB.
2024/11/16 02:54:19 In the last 1h0m0s, there were 22 completed connections. Traffic Relayed ↓ 574203 KB, ↑ 60168 KB.
2024/11/16 03:54:19 In the last 1h0m0s, there were 27 completed connections. Traffic Relayed ↓ 691077 KB, ↑ 64850 KB.
2024/11/16 04:54:19 In the last 1h0m0s, there were 26 completed connections. Traffic Relayed ↓ 838664 KB, ↑ 79757 KB.
2024/11/16 05:54:19 In the last 1h0m0s, there were 38 completed connections. Traffic Relayed ↓ 1092028 KB, ↑ 102901 KB.
2024/11/16 06:54:19 In the last 1h0m0s, there were 63 completed connections. Traffic Relayed ↓ 2907563 KB, ↑ 216042 KB.
2024/11/16 07:54:19 In the last 1h0m0s, there were 49 completed connections. Traffic Relayed ↓ 731014 KB, ↑ 97379 KB.
2024/11/16 08:54:19 In the last 1h0m0s, there were 62 completed connections. Traffic Relayed ↓ 1384940 KB, ↑ 125384 KB.
2024/11/16 09:54:19 In the last 1h0m0s, there were 61 completed connections. Traffic Relayed ↓ 758305 KB, ↑ 79709 KB.
2024/11/16 10:54:19 In the last 1h0m0s, there were 51 completed connections. Traffic Relayed ↓ 668823 KB, ↑ 99071 KB.
2024/11/16 11:54:19 In the last 1h0m0s, there were 52 completed connections. Traffic Relayed ↓ 1160182 KB, ↑ 169377 KB.
2024/11/16 12:42:54 Proxy starting
2024/11/16 13:42:54 In the last 1h0m0s, there were 48 completed connections. Traffic Relayed ↓ 546363 KB (151.77 KB/s), ↑ 69781 KB (19.38 KB/s).
2024/11/16 14:42:54 In the last 1h0m0s, there were 57 completed connections. Traffic Relayed ↓ 1392034 KB (386.68 KB/s), ↑ 139644 KB (38.79 KB/s).
2024/11/16 15:42:54 In the last 1h0m0s, there were 74 completed connections. Traffic Relayed ↓ 1654149 KB (459.49 KB/s), ↑ 133668 KB (37.13 KB/s).
2024/11/16 16:42:54 In the last 1h0m0s, there were 58 completed connections. Traffic Relayed ↓ 2355710 KB (654.36 KB/s), ↑ 191297 KB (53.14 KB/s).
2024/11/16 17:42:54 In the last 1h0m0s, there were 59 completed connections. Traffic Relayed ↓ 1938685 KB (538.52 KB/s), ↑ 193443 KB (53.73 KB/s).
2024/11/16 18:42:54 In the last 1h0m0s, there were 60 completed connections. Traffic Relayed ↓ 2138684 KB (594.08 KB/s), ↑ 177815 KB (49.39 KB/s).
2024/11/16 19:42:54 In the last 1h0m0s, there were 51 completed connections. Traffic Relayed ↓ 2086345 KB (579.54 KB/s), ↑ 175947 KB (48.87 KB/s).
1 Like
6

Just a snapshot, I recently checked how many concurrent clients my proxy was serving, it was about 20, now it’s about 30.

1 Like
7

Well, probably off-topic, but I do have dedicated cloud instances, were NAT detection failed for some (local?) reasons:

8

As they said in Unredacted: "@WofWca No worries, not annoying at all. Haven't…" - Mastodon, this commit is probably also ruled out as a possible cause.

9

Possible, but with the behavior outlined in the issue, I’d expect an opposite effect: reduced load after a restart.

10

For my info, how do you check concurrent clients?

11

In verbose logging my proxy shows a counter that increases when a client connects and decreases when a client disconnects.

12

Is this standard or some mode you made? It’s not that 1 hour message then.

13

Here is updated data from my proxy (collected from lines In the last 1h0m0s...).
For me it looks like user count is the same, traffic is slightly higher.
Connections:
snowflake_update_conns
Traffic:
snowflake_update_traff

1 Like
14

I don’t put much faith in the completed connections count of “In the last” message. I use to think it was a client count.

It just states how many connections closed in the last hour. That connection could have opened in the last minute, last hour, last day or, in theory, the last week or last month.

1 Like
15

I see a huge difference between up and down in snowflake:

2024/12/23 05:44:59 In the last 1h0m0s, there were 68 completed connections. Traffic Relayed ↓ 1097981 KB (304.99 KB/s), ↑ 88573 KB (24.60 KB/s).

2024/12/22 10:44:59 In the last 1h0m0s, there were 82 completed connections. Traffic Relayed ↓ 675609 KB (187.67 KB/s), ↑ 71453 KB (19.85 KB/s).
2024/12/22 11:44:59 In the last 1h0m0s, there were 83 completed connections. Traffic Relayed ↓ 1314721 KB (365.20 KB/s), ↑ 128850 KB (35.79 KB/s).
2024/12/22 12:44:59 In the last 1h0m0s, there were 100 completed connections. Traffic Relayed ↓ 2407368 KB (668.71 KB/s), ↑ 216350 KB (60.10 KB/s).
2024/12/22 13:44:59 In the last 1h0m0s, there were 88 completed connections. Traffic Relayed ↓ 1449853 KB (402.74 KB/s), ↑ 115625 KB (32.12 KB/s).
2024/12/22 14:44:59 In the last 1h0m0s, there were 115 completed connections. Traffic Relayed ↓ 2195171 KB (609.77 KB/s), ↑ 145554 KB (40.43 KB/s).
2024/12/22 15:44:59 In the last 1h0m0s, there were 83 completed connections. Traffic Relayed ↓ 1525750 KB (423.82 KB/s), ↑ 113761 KB (31.60 KB/s).
2024/12/22 16:44:59 In the last 1h0m0s, there were 109 completed connections. Traffic Relayed ↓ 1396074 KB (387.80 KB/s), ↑ 120824 KB (33.56 KB/s).
2024/12/22 17:44:59 In the last 1h0m0s, there were 81 completed connections. Traffic Relayed ↓ 1205192 KB (334.78 KB/s), ↑ 118768 KB (32.99 KB/s).
2024/12/22 18:44:59 In the last 1h0m0s, there were 90 completed connections. Traffic Relayed ↓ 3812000 KB (1058.89 KB/s), ↑ 239818 KB (66.62 KB/s).
2024/12/22 19:44:59 In the last 1h0m0s, there were 107 completed connections. Traffic Relayed ↓ 1860924 KB (516.92 KB/s), ↑ 205720 KB (57.14 KB/s).
2024/12/22 20:44:59 In the last 1h0m0s, there were 91 completed connections. Traffic Relayed ↓ 3699650 KB (1027.68 KB/s), ↑ 212282 KB (58.97 KB/s).
2024/12/22 21:44:59 In the last 1h0m0s, there were 97 completed connections. Traffic Relayed ↓ 1681599 KB (467.11 KB/s), ↑ 101739 KB (28.26 KB/s).
2024/12/22 22:44:59 In the last 1h0m0s, there were 57 completed connections. Traffic Relayed ↓ 1817222 KB (504.78 KB/s), ↑ 132525 KB (36.81 KB/s).
2024/12/22 23:44:59 In the last 1h0m0s, there were 20 completed connections. Traffic Relayed ↓ 1078866 KB (299.69 KB/s), ↑ 99306 KB (27.59 KB/s).
2024/12/23 00:44:59 In the last 1h0m0s, there were 38 completed connections. Traffic Relayed ↓ 639518 KB (177.64 KB/s), ↑ 100662 KB (27.96 KB/s).
2024/12/23 01:44:59 In the last 1h0m0s, there were 32 completed connections. Traffic Relayed ↓ 528713 KB (146.86 KB/s), ↑ 51664 KB (14.35 KB/s).
2024/12/23 02:44:59 In the last 1h0m0s, there were 36 completed connections. Traffic Relayed ↓ 599455 KB (166.52 KB/s), ↑ 73039 KB (20.29 KB/s).
2024/12/23 03:44:59 In the last 1h0m0s, there were 64 completed connections. Traffic Relayed ↓ 716890 KB (199.14 KB/s), ↑ 67757 KB (18.82 KB/s).
2024/12/23 04:44:59 In the last 1h0m0s, there were 55 completed connections. Traffic Relayed ↓ 683276 KB (189.80 KB/s), ↑ 68038 KB (18.90 KB/s).
2024/12/23 05:44:59 In the last 1h0m0s, there were 68 completed connections. Traffic Relayed ↓ 1097981 KB (304.99 KB/s), ↑ 88573 KB (24.60 KB/s).
2024/12/23 06:44:59 In the last 1h0m0s, there were 89 completed connections. Traffic Relayed ↓ 1934630 KB (537.40 KB/s), ↑ 130583 KB (36.27 KB/s).
2024/12/23 07:44:59 In the last 1h0m0s, there were 72 completed connections. Traffic Relayed ↓ 1464795 KB (406.89 KB/s), ↑ 115185 KB (32.00 KB/s).
2024/12/23 08:44:59 In the last 1h0m0s, there were 111 completed connections. Traffic Relayed ↓ 2175554 KB (604.32 KB/s), ↑ 130027 KB (36.12 KB/s).
2024/12/23 09:44:59 In the last 1h0m0s, there were 89 completed connections. Traffic Relayed ↓ 2869614 KB (797.12 KB/s), ↑ 180566 KB (50.16 KB/s).
2024/12/23 10:44:59 In the last 1h0m0s, there were 117 completed connections. Traffic Relayed ↓ 892818 KB (248.00 KB/s), ↑ 115713 KB (32.14 KB/s).
2024/12/23 11:44:59 In the last 1h0m0s, there were 112 completed connections. Traffic Relayed ↓ 2115090 KB (587.52 KB/s), ↑ 137564 KB (38.21 KB/s).
2024/12/23 12:44:59 In the last 1h0m0s, there were 135 completed connections. Traffic Relayed ↓ 369218 KB (102.56 KB/s), ↑ 54091 KB (15.03 KB/s).
2024/12/23 13:44:59 In the last 1h0m0s, there were 113 completed connections. Traffic Relayed ↓ 985400 KB (273.72 KB/s), ↑ 142823 KB (39.67 KB/s).
2024/12/23 14:44:59 In the last 1h0m0s, there were 126 completed connections. Traffic Relayed ↓ 1342818 KB (373.00 KB/s), ↑ 134096 KB (37.25 KB/s).
2024/12/23 15:44:59 In the last 1h0m0s, there were 112 completed connections. Traffic Relayed ↓ 2003641 KB (556.57 KB/s), ↑ 143616 KB (39.89 KB/s).
2024/12/23 16:44:59 In the last 1h0m0s, there were 128 completed connections. Traffic Relayed ↓ 1579346 KB (438.71 KB/s), ↑ 141820 KB (39.39 KB/s).

Is this normal - version is snowflake-proxy 2.10.1 (e7a7f41c*) with the dirty patch applied. Running on a server without any NAT.

mrtg shows different:

image
image700×335 5.62 KB

16

Most likely, these lines show numbers from proxy user perspective.
Users are more active at downloading than uploading, that’s why difference appear.

2 Likes
17

Looks normal to me, I think this has always been the case. This shows the traffic from the client’s perspective, if I can put it this way, i.e. GET requests are usually smaller than the responses.

1 Like