Hello,
Below I share OONI’s status report for November 2022.
# OONI Monthly Report: November 2022
Throughout November 2022, the OONI team worked on the following sprints:
- Sprint 77 (1st-6th November 2022)
- Sprint 78 (7th-20th November 2022)
- Sprint 79 (21st-30th November 2022)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
## Technical multi-stakeholder research report on Internet shutdowns in Iran
In collaboration with IODA, M-Lab, Cloudflare, Kentik, Censored Planet, ISOC, and Article19, OONI published a new research report: “Technical multi-stakeholder report on Internet shutdowns: The case of Iran amid autumn 2022 protests”.
Read the report here: https://ooni.org/post/2022-iran-technical-multistakeholder-report/
This report combines multiple measurement datasets in an attempt to investigate the internet shutdown events in Iran over the past months amid protests. We consider this a particularly important research report, as it’s likely the first ever report that combines so many internet measurement datasets in investigating a case study.
Specifically, we provide data and analysis on the following in Iran:
- Mobile network outages
- Regional outages
- HTTP/3 and QUIC traffic drop
- IPv6 disruption on Irancell
- Increased blocking of encrypted DNS
- Impact on internet speed tests
- Blocking of WhatsApp, Instagram, Skype, Linkedin, and Viber
- Blocking of Google Play Store and Apple App Store
- Blocking of browser extension repositories
ISOC also shared their analysis on examining the economic impact of the shutdowns.
We provide a summary of the key findings here: https://ooni.org/post/2022-iran-technical-multistakeholder-report/#summary-of-key-findings
ISOC also published a blog post about the report: https://pulse.internetsociety.org/blog/technical-multistakeholder-report-on-internet-shutdowns-published
## OONI Team Meeting
Between 2nd-4th November 2022, we organized and hosted a 3-day OONI Team Meeting in Rome, Italy.
This was the first in-person OONI Team Meeting since the start of the COVID-19 pandemic. Unfortunately not everyone was able to join us in person in Rome, and so we facilitated a hybrid meeting, with some participants in person, and others joining us remotely online. In addition to OONI’s 9 team members, the meeting also included the 2 OTF Information Controls Research Fellows (Ain Ghazal and Gurshabad Grover) that we’re currently hosting.
During this 3-day meeting, we improved our roadmaps and discussed strategic priorities and implementation details for many new systems we’ll be building over the next few years.
Specifically, the 3-day OONI Team Meeting included the following sessions:
- Time for reflection
- Community needs and priorities
- Expanding OONI’s testing model to support richer testing input
- Incrementally integrating Websteps into Web Connectivity LTE
- Measuring the blocking of VPNs
- Research on Internet censorship in South Asia and analysis of “failed” measurements
- Creating a methodology for measuring throttling: Next steps
- Automating the detection and characterization of censorship
- Creating a Social Media Censorship Alert System
- Presenting thematic censorship findings on OONI Explorer
- Creating a Censorship Incident Reporting Platform
- Aligning the anomaly calculations between the pipeline, Explorer, and probe
- Handling of HTTP response bodies in measurements
- Triage and prioritization of issues
- Roadmapping sessions
- Fundraising priorities
During the OONI Team Meeting, we also had an OONI-themed cake to celebrate OONI’s 10th Anniversary together (https://ooni.org/post/ooni-in-2022/images/01.png). After the 3-day OONI Team Meeting, we had a “team bonding day” together in Rome.
## Organizing OONI’s 10th Anniversary
In preparation for OONI’s 10th Anniversary (on 5th December 2022), we worked towards organizing live-streamed events and creating multimedia content to mark the important milestone.
In November 2022, we published a blog post which invites the internet freedom community to attend 2 live-streamed events for OONI’s 10th Anniversary: https://ooni.org/post/10th-ooniversary-events/
Throughout November 2022, we interviewed 11 OONI community members and worked with a videographer on creating a video that highlights how OONI has been useful to the internet freedom community. We also wrote a script for an animation that provides a timeline of key OONI highlights from the past 10 years, and we worked with animators on producing an animation. Both the community video and animation were scheduled for publication around OONI’s 10th Anniversary in early December 2022.
## OONI Probe Mobile
In November 2022, we released OONI Probe Android 3.7.2 (https://github.com/ooni/probe-android/releases/tag/v3.7.2) with bug fixes and improvements. We worked towards ensuring that flutter implementations are updated properly on all platforms and we started making progress on unsupported platforms (https://github.com/ooni/probe-cli/pull/880). We also worked on triaging issues reported in Sentry.
## OONI Run
We edited the copy on the OONI Run platform: https://github.com/ooni/run/pull/118
## OONI Probe CLI
We released OONI Probe CLI 3.16.4 (https://github.com/ooni/probe-cli/releases/tag/v3.16.4) and v3.16.5 (https://github.com/ooni/probe-cli/releases/tag/v3.16.5) to fix issues with the Signal nettest caused by Signal changing the certification authority it uses to authenticate its endpoints.
Additionally, we started merging ooniprobe (the official command line client) and miniooni (the research command line client) by abstracting common functionality into shared Go packages to be used by both tools. This work is documented through the following pull requests: https://github.com/ooni/probe-cli/pull/979 and https://github.com/ooni/probe-cli/pull/982.
Notably, we added support to ooniprobe for a --proxy command line argument, thus facilitating the use of circumvention tools (such as tor): https://github.com/ooni/probe-cli/pull/981
We also made the code that communicates with the Web Connectivity Test Helper more reliable by trying all the available Test Helpers, rather than just using the first one returned by the backend API. This work is available through the following pull request: https://github.com/ooni/probe-cli/pull/980
## OONI Probe Desktop
We released OONI Probe Desktop 3.8.2 which includes Signal nettest fixes from OONI Probe CLI: https://github.com/ooni/probe-desktop/releases/tag/v3.8.2
## Published an OONI Probe Desktop screencast
To enable community members worldwide to use OONI Probe, we created an OONI Probe Desktop screencast which we published on our YouTube channel.
Our OONI Probe Desktop screencast is available here: https://www.youtube.com/watch?v=tLDVpyHFsW0
## Russian translation of the OONI Probe Mobile and Desktop User Guides
We translated the OONI Probe user guides to Russian, and published the translated versions:
- OONI Probe Mobile: https://ooni.org/ru/support/ooni-probe-mobile
- OONI Probe Desktop: https://ooni.org/ru/support/ooni-probe-desktop
## Expanding OONI’s testing model to support richer testing input
We started refactoring work to support providing richer input to OONI Probe tests (https://github.com/ooni/ooni.org/issues/1291). This work stems from the observation that OONI experiments require input that is more complex than a URL in several cases. For example, to correctly test HTTP/3 websites we would need to know whether a website supports HTTP/3. We started off by refactoring the signature of OONI experiment’s “Run” function to simplify the changing of arguments passed to each experiment (https://github.com/ooni/probe-cli/pull/983), without having to refactor ~30 Go packages each time.
## OONI Explorer
We continued to work on the user feedback reporting mechanism (https://github.com/ooni/explorer/pull/790), focusing on usability aspects with the help of a UI/UX designer. In order to improve the current OONI Explorer interface and plan new features, we started conducting UX research.
We added support for RTL languages and localized date formatting across OONI Explorer as part of localization work (https://github.com/ooni/explorer/pull/805). We also fixed the back navigation on pages that include filter params in the URL (https://github.com/ooni/explorer/pull/815) and we added support for filtering by anomalies (https://github.com/ooni/explorer/pull/816) on the Measurement Aggregation Toolkit (MAT).
## OONI backend
We started reviewing the security report from Cure53 (who conducted an independent security audit of our software) and we deployed different mitigations based on the feedback we received. We implemented Bearer Token in the API as an improvement over the previous security cookie (https://github.com/ooni/api/pull/313). We also implemented a mitigation to a minor security issue affecting parameter validation in the API documentation path “/apidocs”.
We deployed a fix to hide Clickhouse error traces from the API output for improved security. We also carried out some tests to compress ooniprobe binaries using UPX. We updated the fastpath pipeline (https://github.com/ooni/pipeline/pull/401) to support the updated version of the Signal test.
## OONI Test Lists Editor
Following the backend changes to authentication, the cookie based authentication was replaced by a Bearer Token one in the Test Lists Editor: https://github.com/ooni/test-lists-ui/pull/68
## Automating censorship detection and characterization based on OONI measurements
We created a new data analysis tool (https://github.com/ooni/data) for exposing anomaly details of website measurements and characterizing website blocking. Some of the building blocks for designing this tool were previously applied in practice as part of our investigation into emergent website blocks in Russia (https://ooni.org/post/2022-russia-blocks-amid-ru-ua-conflict/), where we were able to characterize the blocking methods across ISPs (https://ooni.org/post/2022-russia-blocks-amid-ru-ua-conflict/#blocked-website-about-captured-and-killed-russian-soldiers).
The high level architecture (https://github.com/ooni/data#architecture-overview) of this new data analysis tool is that of transforming the raw network measurements coming from OONI Probes into normalized and post-processed “Observations” (https://github.com/ooni/data#observation-generation). These Observations are “time stamped” statements about some network condition that was observed by a particular vantage point. These observations are then in turn post-processed to generate what we call an “Experiment result” (https://github.com/ooni/data#experimentresult-generation), which attributes to a set of observations from an individual test run a series of outcomes with an associated level of confidence.
In order to reach a design that would be flexible enough to be expanded, but also performant enough to run in a reasonable amount of time, we implemented several prototype iterations of this design (https://github.com/ooni/data/pull/15). Throughout the process we carried out benchmarks of key components to identify any performance bottlenecks, working to resolve them (https://github.com/ooni/data/issues/4). The last iteration (https://github.com/ooni/data/pull/17), based on some cursory benchmarks (https://github.com/ooni/data/pull/17#issue-1452100512), should be able to generate observations from raw OONI measurements in less than a week and re-generate all experiment results in about a day, running on a single machine.
Through this data analysis tool (which anyone can run on their own computer pointing it at the raw public OONI dataset: https://github.com/ooni/data#developer-setup), it’s possible to gain a much deeper understanding about the root cause of a blocking event, and to carry out more advanced research that might not be captured by the existing analysis.
The new design is an important shift in how we view results from OONI measurements and we are confident that it will allow us to characterize blocks in a much richer way. It also opens the door for empowering researchers to carry out more advanced censorship investigations using our datasets.
## Creating a Social Media Censorship Alert System
To enable human rights defenders to rapidly respond to social media blocks, we aim to create a Social Media Censorship Alert System which allows access to real-time OONI data on the blocking of major social media platforms around the world.
In November 2022, we started the initial research process for designing this new platform, which involved experimenting with implementations for social media blocking event detection. Specifically, we experimented with different designs to provide lightweight event detection without impacting the backend infrastructure.
As part of this initial experimentation, we applied analysis logic for issuing an “alert” when anomalies are present for selected social media targets (the initial experiment involved Twitter and Facebook). We set the thresholds for issuing “alerts” based on moving averages, as the volume and frequency of relevant measurements varies significantly from ASN to ASN, and country to country over time.
Based on our experimentation, we created Jupyter Notebook charts to investigate ways to classify groups of measurements as blocking or unblocking events.
## Social media blocked in Turkey
On 13th November 2022, following the bomb explosion in Istanbul, some ISPs in Turkey blocked access to Twitter, Instagram, Facebook, and YouTube.
We responded by sharing real-time OONI data collected from Turkey (along with a MAT chart), documenting the blocks: https://twitter.com/OpenObservatory/status/1591871076677701635
Our tweet was cited in articles published by:
- Global Voices: https://globalvoices.org/2022/11/13/turkey-throttles-internet-access-following-deadly-explosion-in-istanbul/
- Coda Story: https://www.codastory.com/newsletters/apple-beijing-turkey-istanbul-blast/#new_tab
## Test list updates
In collaboration with community members, we updated the test list for Kazakhstan (https://github.com/citizenlab/test-lists/pull/1182). We also reviewed and merged updates to the test list for Malaysia (https://github.com/citizenlab/test-lists/pull/1175) contributed by our partner, Sinar Project.
## Community use of OONI tools and data
### OONI censorship measurement campaign during Malaysia’s 2022 general elections
Leading up to and during Malaysia’s 2022 elections, our partner, Sinar Project, coordinated an OONI Probe measurement campaign (using OONI Run widgets for custom testing) to monitor potential censorship events.
Information about the censorship measurement campaign, along with OONI Run widgets for relevant testing (including election monitoring and political party websites, among others), was published on their website: https://sinarproject.org/digital-rights/measuring-and-detecting-network-interference/ge15
Sinar Project also shared relevant OONI Probe testing instructions through this Twitter thread: https://twitter.com/sinarproject/status/1592879196610920448
### OONI censorship measurement testing during Kazakhstan’s 2022 elections
Leading up to Kazakhstan’s 2022 elections, we collaborated with local groups on creating a custom list of websites that are most relevant to test in Kazakhstan for censorship.
To enable the testing of such sites during the elections, we embedded an OONI Run widget (https://github.com/ooni/ooni.org/pull/1304) into our website: https://ooni.org/get-involved/run (the “Kazakhstan (38 URLs)” widget)
We shared relevant OONI Probe testing instructions on Twitter: https://twitter.com/OpenObservatory/status/1593988313932828675
We also wrote documentation providing relevant OONI Probe testing instructions for community members in Kazakhstan.
### Global Voices blog post on the blocking of a website that digitizes books in Mexico
Global Voices published a blog post documenting the blocking of a website run by a Mexican collective that digitizes books as a political position. This post cites OONI data on the block.
Their blog post is available here: https://globalvoices.org/2022/11/30/mexican-collective-pirating-books-to-make-culture-accessible-is-blocked/
## Community activities
### OONI workshop for civil society in Kazakhstan
On 3rd November 2022, OONI’s Elizaveta facilitated an online OONI workshop for civil society groups in Kazakhstan.
Information about the workshop is available here: https://www.facebook.com/medianetkaz/posts/pfbid02nhnfrTotir9yaeZA3EJtG64oivV8rUrB7hs9yMDJoLFE152JkSh2ZDb4p2bX4puYl
### OONI presentations at IETF 115
On 8th November 2022, OONI’s Simone presented our censorship measurements in Iran as part of the Internet Architecture Board (IAB) open meeting during IETF 115 (https://www.ietf.org/live/ietf115-iab-open/). His IAB Open presentation can be viewed here: https://www.youtube.com/watch?v=7_G6XVjlun8&t=1730s
On 9th November 2022, OONI’s Simone presented our strategies for measuring the blocking and throttling of encrypted protocols (including HTTPS, DNS over HTTPS, and HTTP/3) as part of the Privacy Enhancements and Assessments Research Group (PEARG) meeting during IETF 115 (https://datatracker.ietf.org/meeting/115/session/pearg). His PEARG presentation can be viewed here: https://www.youtube.com/watch?v=NO3pw3_hees&t=3592s
### OONI workshop for journalists in Central Asia
On 10th November 2022, OONI’s Maria facilitated an online OONI workshop for journalists in Central Asia.
### OONI presentation at State of the Onion 2022
On 16th November 2022, OONI’s Maria presented OONI highlights from 2022, as well as upcoming OONI projects for 2023 as part of the Tor Project’s annual State of the Onion event (https://blog.torproject.org/state-of-the-onion-2022/).
The live-streamed event can be viewed here: https://www.youtube.com/watch?v=O-7k0PjnBbk
### OONI presentation at DataFest 2022
On 17th November 2022, OONI’s Elizaveta presented OONI at DataFest 2022 in Tbilisi (https://www.datafest.ge/schedule). As part of this presentation, Elizaveta explained how journalists can use OONI’s Measurement Aggregation Toolkit (MAT) to investigate internet censorship around the world.
### OONI datathon at DataFest 2022
On 19th November 2022, we facilitated an OONI datathon as part of DataFest 2022 in Tbilisi ((https://www.datafest.ge/datathons). During the OONI datathon, participants worked on analyzing OONI data collected from Russia, Kazakhstan and Uzbekistan to investigate local internet censorship.
### OONI presentation at Internet Without Borders conference
On 18th November 2022, OONI’s Arturo presented censorship measurement findings from Russia as part of the Internet Without Borders conference in Paris (https://web.archive.org/web/20221101023014/https://internetborders.net/conference-agenda/).
### OONI hackathon at Internet Without Borders conference
On 19th November 2022, we facilitated an OONI hackathon as part of the Internet Without Borders conference in Paris (https://internetborders.net/ooni-challenge/). During the hackathon, a participant wrote new SMTP(s), IMAP(s) (https://github.com/ooni/probe-cli/pull/989) and bittorrent experiments (https://github.com/ooni/probe-cli/pull/986) for OONI Probe!
### OONI workshops in Kyrgyzstan
Between 24th–26th November 2022, OONI’s Elizaveta traveled to Kyrgyzstan to facilitate 5 in-person OONI workshops for local journalists, students and civil society organizations. As part of these workshops, Elizaveta explained how they can use OONI Probe to collect internet measurements and how they can use OONI data to investigate internet censorship.
### OONI workshop in Kazakhstan by Access Now
On 18th November 2022, Access Now organized an offline event for elections observers in Kazakhstan. As part of this event, Anastasia Zhyrmont from Access Now facilitated an OONI workshop and explained how election observers can contribute to collecting internet measurements during the elections.
### OONI Community Meeting
On 29th November 2022, we hosted the monthly OONI Community Meeting on our Slack channel (https://slack.ooni.org/), during which we discussed the following topics:
-
Strategies for monitoring potential social media blocks during Indonesia’s 2024 elections. Discussion of how blocks are implemented in Indonesia and improving the data quality of OONI measurements.
-
Requesting feedback on the type of materials that community members would find most useful and would like to see in the upcoming OONI Outreach Kit.
## Userbase
In November 2022, 61,907,164 OONI Probe measurements were collected from 2,851 AS networks in 166 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ OONI team.