Hi everyone,
TPA held its first meeting of the year, and those are the minutes. I’ll
take the opportunity to wish everyone a happy new year, if you’re into
that kind of calendar. I know it’s not the most obvious thing to do
right now, but I wish you can find hope this year.
Roll call: who’s there and emergencies
- anarcat
- kez
- lavamind
No emergencies.
Holidays debrief
Holidays went fine, some minor issues, but nothing that needed to be
urgently dealt with (e.g. 40569, 40567, commit, runner
bug). Rotation worked well.
anarcat went cowboy and setup two new nodes before the holidays, which
is not great because it’s against our general “don’t launch on a
friday”. (It wasn’t on a friday, but it was close enough to the
holidays to be a significant risk.) Thankfully things worked out fine:
one of the runners ended up failing just as lavamind was starting work
again last week. (!)
2021 roadmap review
sysadmin
We did a review directly in the wiki page. Notable changes:
- jenkins is marked as completed, as rouyi will be retired this week
(!) - the blog migration was completed!
- we consider we managed to deal with the day-to-day while still
reserving time for the unexpected (e.g. the rushed web migration
from Jenkins to GitLab CI) - we loved that team work and should plan to do it again
- we were mostly on budget: we had an extra 100EUR/mth at hetzner for
a new Ganeti node in the gnt-fsn cluster, and extra costs
(54EUR/mth!) for the Hetzner IPv4 billing changes, and more for
extra bandwidth use
web
Did a review of the 2021 web roadmap (from the wiki homepage), copied below:
- Donations page redesign - 10-50%
- Improves bridges.torproject.org - 80% done!
-
Remove outdated documentation from the header - the
“docs.tpo ticket”, considering using dev.tpo instead, focus on
launching dev.tpo next instead - Migrate blog.torproject.org from Drupal To Lektor: it needs a milestone and planning
- Support forum
- Developer portal AKA dev.tpo
- Get website build from Jenkins into to GitLabCI for the static mirror pool (before December)
-
Get up to speed on maintenance tasks:
- Bootstrap upgrade - uh god.
- browser documentation update - what is this?
- get translation stats available - what is this?
- rename ‘master’ branch as ‘main’
- fix wiki for documentation - what is this?
- get onion service tooling into TPO GitLab namespace - what is this?
Syadmin+web OKRs for 2022 Q1
We want to take more time to plan for the web team, in particular, and especially focused on this in the meeting.
web team
We did the following brainstorm. Anarcat will come up with a proposal for a better-formatted OKR set for next week, at which point we’ll prioritize this and the sysadmin OKRs for Q1.
- OKR: rewrite of the donate page (milestone 22)
- OKR: make it easier for translators to contribute
- help the translation team to switch to Weblate
- it is easier for translators to find their built copy of the website
- bring build time to 15 minutes to accelerate feedback to translators
- allow the web team to trigger manual builds for reviews
- OKR: documentation overhaul:
- launch dev.tpo
- “Remove outdated documentation from the header”, stop pointing to dead docs
- come with ideas on how to manage the wiki situation
- cleanup the queues and workflow
- OKR: resurrect bridge port scan
- do not scan private IP blocks
- make it pretty
Missed from the last meeting:
- sponsor 9 stuff: collected UX feedback for portals, which involves web to fix issues we found, need to prioritise
We also need to organise with the new people:
- onion SRE: new OTF project USAGM, starting in February
- new community person
Other discussions
Next meeting
We’re going to hold another meeting next week, same time, to review the web OKRs and prioritize Q1.
Metrics of the month
- hosts in Puppet: 89, LDAP: 91, Prometheus exporters: 139
- number of Apache servers monitored: 27, hits per second: 185
- number of Nginx servers: 0, hits per second: 0, hit ratio: 0.00
- number of self-hosted nameservers: 6, mail servers: 8
- pending upgrades: 7, reboots: 0
- average load: 0.35, memory available: 4.01 TiB/5.13 TiB, running processes: 643
- disk free/total: 84.95 TiB/39.99 TiB
- bytes sent: 325.45 MB/s, received: 190.66 MB/s
- planned bullseye upgrades completion date: 2024-09-07
-
GitLab tickets: 159 tickets including…
- open: 2
- icebox: 143
- backlog: 8
- next: 2
- doing: 2
- needs information: 2
- (closed: 2573)
Upgrade prediction graph now lives at:
… with someone accurate values, although the 2024 estimate above
should be taken with a grain of salt, as we haven’t really started the
upgrade at all.
Number of the month
- We just hit the 5TiB of deployed memory, kind of neat.
Another number of the month
- We have zero Nginx servers left, as we turned off our two Nginx
servers (ignoring the Nginx server in the GitLab instance, which is not
really monitored correctly), when we migrated the blog to a static
site. Those two servers were the caching server sitting in front of the
Drupal blog for cost savings. They served us well but are now retired
since they are not necessary for the static version.