[tor-project] GitLab confidential notifications now encrypted!

Summary: GitLab now encrypts outgoing email notifications on
confidential issues, if your key is in LDAP, OpenPGP keys stored in
GitLab will be used soon.

**Table of Contents**

- Announcement
    - Affected users
    - Future work
        - OpenPGP certificates in GitLab
        - OpenPGP signatures
- Background
    - History of the confidential issue handling
    - TPA handling of OpenPGP certificates
    - OpenPGP implementation details

# Announcement

Anyone who has dealt with GitLab confidential issues will know this
message:

A comment was added to a confidential issue and its content was
redacted from this email notification.

If you found that irritating, you're not alone! Rejoice, its time is
coming to an end.

Starting today (around 2025-06-10 19:00UTC), we have deployed a new
encryption system in the GitLab notification pipeline. If your OpenPGP
certificate (or "PGP key") is properly setup in LDAP, you will instead
receive a OpenPGP-encrypted email with the actual contents.

No need to click through anymore!

If your key is not available, nothing changes: you will still get the
"redacted" messages. If you do *not* control your key, yet it's still
valid and in the keyring, you *will* get encrypted email you won't be
able to read.

In any case, if any of those new changes cause any problems or if you
need to send us an OpenPGP certificate (or update it), [file an
issue] or reach out to our [usual support channels].

[usual support channels]: support · Wiki · The Tor Project / TPA / TPA team · GitLab
[file an issue]: Sign in · GitLab

We also welcome constructive feedback on the implementation, relieved
thanks and other comments, either here, through the above support
channels, or in the [discussion issue].

[discussion issue]: Encrypt confidential emails (TPA-RFC-89) (#151) · Issues · The Tor Project / TPA / Gitlab · GitLab

## Affected users

Any GitLab user subscribed to confidential issues and who is
interested in not getting "redacted" emails from GitLab.

## Future work

### OpenPGP certificates in GitLab

Right now, only "LDAP keys" (technically, the OpenPGP certificates
`account-keyring.git` project) are considered for encryption.

Only mail delivered to `@torproject.org` are considered as well.

In the future, we hope to implement a GitLab API lookup that will
allow other users to upload OpenPGP certificates through GitLab to
use OpenPGP encryption for outgoing mail.

This has not been implemented yet because implementing the current
backend was vastly easier, but we still hope to implement the GitLab
backend.

### OpenPGP signatures

Mails are currently encrypted, without signature, which is [actually
discouraged]. We are considering signing outgoing mail, but this
needs to be done carefully because we must handle yet another secret,
rotation, expiry and so on.

[actually discouraged]: Guidance on End-to-End E-mail Security

This means, among other things, that the OpenPGP messages do not
provide any sort of authentication that the message really comes from
GitLab. It's still entirely possible for an attacker to introduce
"fake" GitLab notifications through this system, so you should still
consider notifications to be advisory. The source of truth here is the
GitLab web interface.

OpenPGP signatures were seen as not absolutely necessary for a first
implementation of the encryption system, but may be considered in the
future. Note that we do *not* plan on implementing signatures for
*all* outgoing mail at the time.

# Background

## History of the confidential issue handling

GitLab supports "confidential issues" that are accessible only to the
issue creator and users with the "reporter" role on a project. It is
used to manage security-sensitive issues and any issue that contains
privately identifiable information (PII).

When someone creates or modifies an issue on GitLab, it sends a
notification to users watching the issue. Unfortunately, those
notifications are sent by email without any sort of protection. This
is a long-standing issue in GitLab (e.g. [gitlab-org/gitlab#19056],
2017) that doesn't seem to have gotten any interest upstream.

[gitlab-org/gitlab#19056]: Allow notification emails to be encrypted with PGP/GPG (#19056) · Issues · GitLab.org / GitLab · GitLab

We realized this problem shortly after the GitLab migration, in 2020
([tpo/tpa/gitlab#23]), at which time it wasn't clear what we could
do about it.

[tpo/tpa/gitlab#23]: Confidential issues and comments on them are sent out in the clear (#23) · Issues · The Tor Project / TPA / Gitlab · GitLab

But a few years later (September 2022), Micah actually bit the bullet
and started work on patching GitLab itself to at least *identify*
confidential issues with a special header.

He also provided a prototype filtering script that would *redact* (but
not encrypt!) messages on the way out, which anarcat improved on and
deployed in production. That was deployed in October 2023 and there
were actual [fireworks] to celebrate this monumental change, which
has been working reliably for almost two years at this point.

[fireworks]: Confidential issues and comments on them are sent out in the clear (#23) · Issues · The Tor Project / TPA / Gitlab · GitLab

## TPA handling of OpenPGP certificates

We acknowledge our handling of OpenPGP keys (or "certificates") is far
from optimal. Key updates require manual work and the whole thing is
pretty arcane and weird, even weirder than what OpenPGP actually is,
if that's even possible. We have an issue to address that technical
debt ([tpo/tpa/team#29671]) and we're considering this system to be
legacy.

[tpo/tpa/team#29671]: evaluate possible options for OpenPGP keyring maintenance (#29671) · Issues · The Tor Project / TPA / TPA team · GitLab

We are also aware that the keyring is severely out of date and
[requires a serious audit].

[requires a serious audit]: audit account-keyring (#31214) · Issues · The Tor Project / TPA / TPA team · GitLab

The hope, at the moment, is we can ignore that problem and rely on the
GitLab API for users to provide key updates for this system, with the
legacy keyring only used as a fallback.

## OpenPGP implementation details

Programmers might be interested to know this was implemented in an
existing Python script, by encrypting mail with a SOP interface
([Stateless OpenPGP]), which simplified OpenPGP operations
tremendously.

[Stateless OpenPGP]: Stateless OpenPGP Command Line Interface

While SOP is not yet an adopted standards and implementations are
completely solid yet, it has provided a refreshing experience in
OpenPGP interoperability that actually shows great promise in the
standard and its future.

PGP/MIME is another story altogether: that's still an horrible mess
that required crafting MIME parts by hammering butterflies into
melting anvils with deprecated Python blood. But that's just a normal
day at the TPA office, don't worry, everything was [PETA] approved.

[PETA]: People for the Ethical Treatment of Animals - Wikipedia

The implementation is available in TPA's [fabric-tasks] repository,
currently as [merge request !40] but will be merged into the main
branch once GitLab API support is implemented.

[fabric-tasks]: The Tor Project / TPA / Fabric Tasks · GitLab
[merge request !40]: implement confidential issue notifications through legacy account-keyring lookups (!40) · Merge requests · The Tor Project / TPA / Fabric Tasks · GitLab

Follow the progress on this work in the [discussion issue].

···

--
Antoine Beaupré
torproject.org system administration

2 Likes

Finally! you know how long I've been grumbling about those pesky censored notifications :slight_smile:

Bless you and everybody else working on this <3

···

--
ma1

On 11/06/25 03:42, Antoine Beaupré via tor-project wrote:

Summary: GitLab now encrypts outgoing email notifications on
confidential issues, if your key is in LDAP, OpenPGP keys stored in
GitLab will be used soon.

**Table of Contents**

- Announcement
     - Affected users
     - Future work
         - OpenPGP certificates in GitLab
         - OpenPGP signatures
- Background
     - History of the confidential issue handling
     - TPA handling of OpenPGP certificates
     - OpenPGP implementation details

# Announcement

Anyone who has dealt with GitLab confidential issues will know this
message:

A comment was added to a confidential issue and its content was
redacted from this email notification.

If you found that irritating, you're not alone! Rejoice, its time is
coming to an end.

Starting today (around 2025-06-10 19:00UTC), we have deployed a new
encryption system in the GitLab notification pipeline. If your OpenPGP
certificate (or "PGP key") is properly setup in LDAP, you will instead
receive a OpenPGP-encrypted email with the actual contents.

No need to click through anymore!

If your key is not available, nothing changes: you will still get the
"redacted" messages. If you do *not* control your key, yet it's still
valid and in the keyring, you *will* get encrypted email you won't be
able to read.

In any case, if any of those new changes cause any problems or if you
need to send us an OpenPGP certificate (or update it), [file an
issue] or reach out to our [usual support channels].

  [usual support channels]: support · Wiki · The Tor Project / TPA / TPA team · GitLab
  [file an issue]: Sign in · GitLab

We also welcome constructive feedback on the implementation, relieved
thanks and other comments, either here, through the above support
channels, or in the [discussion issue].

  [discussion issue]: Encrypt confidential emails (TPA-RFC-89) (#151) · Issues · The Tor Project / TPA / Gitlab · GitLab

## Affected users

Any GitLab user subscribed to confidential issues and who is
interested in not getting "redacted" emails from GitLab.

## Future work

### OpenPGP certificates in GitLab

Right now, only "LDAP keys" (technically, the OpenPGP certificates
`account-keyring.git` project) are considered for encryption.

Only mail delivered to `@torproject.org` are considered as well.

In the future, we hope to implement a GitLab API lookup that will
allow other users to upload OpenPGP certificates through GitLab to
use OpenPGP encryption for outgoing mail.

This has not been implemented yet because implementing the current
backend was vastly easier, but we still hope to implement the GitLab
backend.

### OpenPGP signatures

Mails are currently encrypted, without signature, which is [actually
discouraged]. We are considering signing outgoing mail, but this
needs to be done carefully because we must handle yet another secret,
rotation, expiry and so on.

  [actually discouraged]: Guidance on End-to-End E-mail Security

This means, among other things, that the OpenPGP messages do not
provide any sort of authentication that the message really comes from
GitLab. It's still entirely possible for an attacker to introduce
"fake" GitLab notifications through this system, so you should still
consider notifications to be advisory. The source of truth here is the
GitLab web interface.

OpenPGP signatures were seen as not absolutely necessary for a first
implementation of the encryption system, but may be considered in the
future. Note that we do *not* plan on implementing signatures for
*all* outgoing mail at the time.

# Background

## History of the confidential issue handling

GitLab supports "confidential issues" that are accessible only to the
issue creator and users with the "reporter" role on a project. It is
used to manage security-sensitive issues and any issue that contains
privately identifiable information (PII).

When someone creates or modifies an issue on GitLab, it sends a
notification to users watching the issue. Unfortunately, those
notifications are sent by email without any sort of protection. This
is a long-standing issue in GitLab (e.g. [gitlab-org/gitlab#19056],
2017) that doesn't seem to have gotten any interest upstream.

  [gitlab-org/gitlab#19056]: Allow notification emails to be encrypted with PGP/GPG (#19056) · Issues · GitLab.org / GitLab · GitLab

We realized this problem shortly after the GitLab migration, in 2020
([tpo/tpa/gitlab#23]), at which time it wasn't clear what we could
do about it.

  [tpo/tpa/gitlab#23]: Confidential issues and comments on them are sent out in the clear (#23) · Issues · The Tor Project / TPA / Gitlab · GitLab

But a few years later (September 2022), Micah actually bit the bullet
and started work on patching GitLab itself to at least *identify*
confidential issues with a special header.

He also provided a prototype filtering script that would *redact* (but
not encrypt!) messages on the way out, which anarcat improved on and
deployed in production. That was deployed in October 2023 and there
were actual [fireworks] to celebrate this monumental change, which
has been working reliably for almost two years at this point.

  [fireworks]: Confidential issues and comments on them are sent out in the clear (#23) · Issues · The Tor Project / TPA / Gitlab · GitLab

## TPA handling of OpenPGP certificates

We acknowledge our handling of OpenPGP keys (or "certificates") is far
from optimal. Key updates require manual work and the whole thing is
pretty arcane and weird, even weirder than what OpenPGP actually is,
if that's even possible. We have an issue to address that technical
debt ([tpo/tpa/team#29671]) and we're considering this system to be
legacy.

  [tpo/tpa/team#29671]: evaluate possible options for OpenPGP keyring maintenance (#29671) · Issues · The Tor Project / TPA / TPA team · GitLab

We are also aware that the keyring is severely out of date and
[requires a serious audit].

  [requires a serious audit]: audit account-keyring (#31214) · Issues · The Tor Project / TPA / TPA team · GitLab

The hope, at the moment, is we can ignore that problem and rely on the
GitLab API for users to provide key updates for this system, with the
legacy keyring only used as a fallback.

## OpenPGP implementation details

Programmers might be interested to know this was implemented in an
existing Python script, by encrypting mail with a SOP interface
([Stateless OpenPGP]), which simplified OpenPGP operations
tremendously.

  [Stateless OpenPGP]: Stateless OpenPGP Command Line Interface

While SOP is not yet an adopted standards and implementations are
completely solid yet, it has provided a refreshing experience in
OpenPGP interoperability that actually shows great promise in the
standard and its future.

PGP/MIME is another story altogether: that's still an horrible mess
that required crafting MIME parts by hammering butterflies into
melting anvils with deprecated Python blood. But that's just a normal
day at the TPA office, don't worry, everything was [PETA] approved.

  [PETA]: People for the Ethical Treatment of Animals - Wikipedia

The implementation is available in TPA's [fabric-tasks] repository,
currently as [merge request !40] but will be merged into the main
branch once GitLab API support is implemented.

  [fabric-tasks]: The Tor Project / TPA / Fabric Tasks · GitLab
  [merge request !40]: implement confidential issue notifications through legacy account-keyring lookups (!40) · Merge requests · The Tor Project / TPA / Fabric Tasks · GitLab

Follow the progress on this work in the [discussion issue].

_______________________________________________
tor-project mailing list -- tor-project@lists.torproject.org
To unsubscribe send an email to tor-project-leave@lists.torproject.org

_______________________________________________
tor-project mailing list -- tor-project@lists.torproject.org
To unsubscribe send an email to tor-project-leave@lists.torproject.org

2 Likes

[...]

Starting today (around 2025-06-10 19:00UTC), we have deployed a new
encryption system in the GitLab notification pipeline. If your OpenPGP
certificate (or "PGP key") is properly setup in LDAP, you will instead
receive a OpenPGP-encrypted email with the actual contents.

To clarify this (because at least one person stumbled upon this), you
actually *can't* "properly setup your PGP key in LDAP" yourself. This is
normally done by TPA as part of your onboarding.

If it's not setup correctly, you need to file an issue with TPA. Send us
your new public key, and we'll take it from there!

This only applies to people with @torproject.org accounts. For others,
we're (again) looking at providing support for OpenPGP keys hosted in
GitLab instead.

See also:

···

On 2025-06-10 21:42:10, Antoine Beaupré via tor-project wrote:

--
Antoine Beaupré
torproject.org system administration
_______________________________________________
tor-project mailing list -- tor-project@lists.torproject.org
To unsubscribe send an email to tor-project-leave@lists.torproject.org

1 Like

Short update...

Summary: GitLab now encrypts outgoing email notifications on
confidential issues, if your key is in LDAP, OpenPGP keys stored in
GitLab will be used soon.

[...]

## Future work

### OpenPGP certificates in GitLab

Right now, only "LDAP keys" (technically, the OpenPGP certificates
`account-keyring.git` project) are considered for encryption.

Only mail delivered to `@torproject.org` are considered as well.

In the future, we hope to implement a GitLab API lookup that will
allow other users to upload OpenPGP certificates through GitLab to
use OpenPGP encryption for outgoing mail.

This has not been implemented yet because implementing the current
backend was vastly easier, but we still hope to implement the GitLab
backend.

I have walked back this idea. GitLab API lookups are vastly more
complicated than just using the current keyring. At this point, I
consider, again, that the best course of action to implement this is to
natively implement this in GitLab and stop bolting on hacks on top of
it.

### OpenPGP signatures

Mails are currently encrypted, without signature, which is [actually
discouraged]. We are considering signing outgoing mail, but this
needs to be done carefully because we must handle yet another secret,
rotation, expiry and so on.

[actually discouraged]: Guidance on End-to-End E-mail Security

This means, among other things, that the OpenPGP messages do not
provide any sort of authentication that the message really comes from
GitLab. It's still entirely possible for an attacker to introduce
"fake" GitLab notifications through this system, so you should still
consider notifications to be advisory. The source of truth here is the
GitLab web interface.

OpenPGP signatures were seen as not absolutely necessary for a first
implementation of the encryption system, but may be considered in the
future. Note that we do *not* plan on implementing signatures for
*all* outgoing mail at the time.

OpenPGP signatures were, however, implemented. The signing key is
available through WKD in "direct mode", and is attached to this message.

I have resolved the related issue about this work, but feedback is still
welcome!

A.

···

On 2025-06-10 21:42:10, Antoine Beaupré via tor-project wrote:

--
Antoine Beaupré
torproject.org system administration

1 Like