[tor-project] minutes from the sysadmin meeting

Mailing Lists tor-project
1

Roll call: who’s there and emergencies

onionoo-backend running out of disk space (tpo/tpa/team#41343)

Dashboard cleanup

Normal per-user check-in:

General dashboards:

Nextcloud roadmap / spreadsheet.

Overall, it seems we are as you would expect when returning from a
rather chaotic vacation. Backlog is large, but things seem to be under
control.

We added SVN back on the roadmap after one too many tickets asking for
setup.

Metrics of the month

  • hosts in Puppet: 89, LDAP: 89, Prometheus exporters: 166
  • number of Apache servers monitored: 37, hits per second: 626
  • number of self-hosted nameservers: 6, mail servers: 10
  • pending upgrades: 1, reboots: 0
  • average load: 0.69, memory available: 3.58 TiB/4.98 TiB, running processes: 424
  • disk free/total: 53.19 TiB/126.72 TiB
  • bytes sent: 403.47 MB/s, received: 269.04 MB/s
  • planned bullseye upgrades completion date: 2024-08-02
  • GitLab tickets: 196 tickets including…
    • open: 0
    • icebox: 163
    • needs information: 5
    • backlog: 13
    • next: 9
    • doing: 4
    • needs review: 2
    • (closed: 3301)

Upgrade prediction graph lives at:

Now also available as the main Grafana dashboard. Head to
https://grafana.torproject.org/, change the time period to 30 days,
and wait a while for results to render.

Number of the month: 42

34 machines were upgraded from bullseye to bookworm in the two first
days of last week! We calculated this was an average of 20 minutes per
host to upgrade.

The trick, of course, is that things often break after the upgrade,
and that “fixing” time is not counted here. That said, last estimate
for this was one hour per machine, and we’re doing a whole fleet
upgrade every 2-3 years, which means about ten hours of work saved per
year.

But the number of the month is, of course, 42, as we now have an equal
number of bookworm and bullseye machine, after the upgrade. And that
number is, naturally, 42.

See also xkcd: Is It Worth the Time? which, interestingly, we fall out of
scope of.