Summary: a proposal to limit the retention of GitLab CI data to 1 year
# Background
As more and more Tor projects moved to GitLab and embraced its continuous integration features, managing the ensuing storage requirements has been a challenge.
We regularly deal with near filesystem saturation incidents on the GitLab server, especially involving CI artifact storage, such as tpo/tpa/team#41402 and recently, tpo/tpa/team#41861
Previously, [TPA-RFC-14] was implemented to reduce the default artifact retention period from 30 to 14 days. This, and CI optimization of individual projects has provided relief, but the long-term issue has not been definitively addressed since the retention period doesn't apply to some artifacts such as job logs, which are kept indefinitely by default.
[TPA-RFC-14]: https://gitlab.torproject.org/tpo/tpa/team/-/wikis/tpa-rfc-14-gitlab-artifacts
# Proposal
Implement a daily GitLab maintenance task to delete CI pipelines older than 1 year in *all* projects hosted on our instance. This will:
* Purge old CI pipeline and job records for the GitLab database
* Delete associated CI job artifacts, even those "kept" either:
* When [manually prevented from expiring] ("Keep" button on CI job pages)
* When they're the [latest successful pipeline artifact]
* Delete old CI job log artifacts
[manually prevented from expiring]: Job artifacts · Jobs · Ci · Help · GitLab
[latest successful pipeline artifact]: Job artifacts · Jobs · Ci · Help · GitLab
## Goals
This is expected to significantly reduce the growth rate of CI-related storage usage, and of the GitLab service in general.
## Affected users
All users of GitLab CI will be impacted by this change.
But more specifically, some projects have "kept" artifacts, which were manually set not to expire. We'll ensure the concerned users and projects will be notified of this proposal. GitLab's documentation has the [instructions to extract this list of non-expiring artifacts](Job artifact troubleshooting for administrators | GitLab).
## Timeline
Barring the need to further discussion, this will be implemented on Monday, December 9th.
## Costs estimates
### Hardware
This is expected to reduce future requirements in terms of storage hardware.
### Staff
This will reduce the amount of TPA labor needed to deal with filesystem
saturation incidents.
# Alternatives considered
A "CI housekeeping" script is already in place, which scrubs job logs daily in a hard-coded list of key projects such as c-tor packaging, which runs an elaborate CI pipeline on a daily basis, and triage-bot, which runs it CI pipeline on a schedule, every 15 minutes.
Although it has helped up until now, this approach is not able to deal with the increasing use of personal fork projects which are used for development.
It's possible to define a different retention policy based on a project's namespace. For example, projects under the `tpo` namespace could have a longer retention period, while others (personal projects) could have a shorter one. This isn't part of the proposal currently as it could violate the principle of least surprise.
# References
* Discussion ticket: tpo/tpa/team#41874
* [Make It Ephemeral: Software Should Decay and Lose Data](Make It Ephemeral: Software Should Decay and Lose Data | Armin Ronacher's Thoughts and Writings)
···
_______________________________________________
tor-project mailing list -- tor-project@lists.torproject.org
To unsubscribe send an email to tor-project-leave@lists.torproject.org