Between 2022-10-01 and 2023-03-13, there was a bug in the software deployed at Snowflake bridges that could cause parts of a user’s stream to be overwritten by parts of other users’ streams. Though the Snowflake team believes the privacy risks of the bug are minor, we are treating it as a security bug and therefore making this announcement. The fix is already deployed on the server, and no action is required by users.
The cause
The source of the bug was issue #40199 and specifically commit 839d2218, which was intended as a performance optimization to reduce memory allocation during the large influx of Snowflake users in late September 2022.
The analysis surrounding the performance improvement incorrectly concluded that certain memory buffers were not reused and therefore did not need to be copied. But in fact, the buffers were reused, with the effect that data meant for one user could be unpredictably overwritten by data for another user.
The effect
There is analysis at issue #40260. If our understanding of the bug is correct, it did not reveal plaintext data or IP addresses to other users. The Snowflake bridge is a Tor bridge, not an exit relay, so even a catastrophic failure could not leak users’ plaintext data or what destinations they were accessing. A Snowflake connection consists of a layered set of protocols, and the bug was in a layer that does not have access to IP addresses or any other persistent identifiers.
- IP
- TCP
- HTTPS (WebSocket)
- session/retransmission ← The bug was here
- Tor TLS
- user data
At most, users and proxies would see fragments of other users’ Tor TLS streams. We believe the degree of information disclosure is less than what an eavesdropper listening at the Snowflake bridge’s HTTPS port would see, which we already believe to be safe. The practical effect, more than a loss of privacy, was excessive retransmissions at the session/retransmission layer, and corrupted TLS data resulting in a terminated connection.
Remediation
We reverted the erroneous commit and added a test to check for the erroneous buffer reuse. We deployed the update to both Snowflake bridges on 2023-03-13.
Required action
No action is required by users. The bug was in code that was only used by the Snowflake server, not proxies or clients.
Thanks
The problem was discovered and first investigated by Haz Æ 41 and reported on HackerOne.