Hello,
Below I share OONI’s status report for December 2022.
# OONI Monthly Report: December 2022
Throughout December 2022, the OONI team worked on the following sprints:
- Sprint 79 (1st-4th December 2022)
- Sprint 80 (5th-18th December 2022)
- Sprint 81 (19th-31st December 2022)
Our work can be tracked through the various OONI GitHub repositories: https://github.com/ooni
Highlights are shared in this report below.
## Year in Review: OONI in 2022
We published our annual blog post, “Year in Review: OONI in 2022”, which shares:
- OONI highlights from 2022
- Highlights from the OONI community (including many of our partners)
- Our plans for 2023
You can read this post here: https://ooni.org/post/ooni-in-2022/
## OONI’s 10th Anniversary
On 5th December 2022, OONI celebrated its 10th anniversary! To mark this milestone, we organized and hosted 2 live-streamed events, published a timeline animation, community video, and blog post. Further details are shared below.
### Live-streamed events
In celebration of OONI’s 10th anniversary, we organized and hosted 2 live-streamed events (https://ooni.org/post/10th-ooniversary-events/) on our YouTube channel:
- 10th Ooniversary: OONI Highlights (5th December 2022)
- 10th Ooniversary: OONI Community (6th December 2022)
As part of the “OONI Highlights” live-streamed event, Arturo (OONI’s founder) shared OONI’s history and highlights from the past 10 years, as well as our plans for the future. This presentation can be viewed here: https://www.youtube.com/watch?v=f4D4gq9TBMg
As part of the “OONI Community” live-streamed event, we invited 5 community members who shared how they’ve used OONI tools and data as part of their research and advocacy efforts (https://ooni.org/post/10th-ooniversary-events/#10th-ooniversary-ooni-community). Their presentations can be viewed here: https://www.youtube.com/watch?v=Co_f8KPBIZM
Both events included presentations followed by discussion addressing questions shared by participants through the live chat and shared pads.
On 5th December 2022, we also hosted an online Distance Disco (https://distancedisco.nl/dd/ooni-s-10th-birthday-party) to dance with community members in celebration of OONI’s 10th anniversary.
### New animation of OONI highlights
On 5th December 2022, we published an animation that provides a timeline of OONI highlights from the past 10 years. This animation was produced in collaboration with Robotina (https://www.robotina.it/).
The animation is available here: https://www.youtube.com/watch?v=AjpfF2kQ9jA
### New OONI Community Video
In celebration of OONI’s 10th Anniversary, we published a new video with OONI community members.
You can watch this video here: https://www.youtube.com/watch?v=zrclQ2QZjVo
The video features 11 community members discussing how OONI has been useful to their work, and what they would like to see OONI do in the future.
You can learn about the featured community members through our blog post: https://ooni.org/post/ooni-community-video/
### Highlights: 10 Years of OONI
To mark OONI’s 10-year anniversary, we published a blog post where we document the key OONI highlights from the past 10 years, as well as some of our future plans.
This blog post is available here: https://ooni.org/post/highlights-10-years-of-ooni/
## OONI Probe Mobile
We released OONI Probe Android 3.7.3 (https://github.com/ooni/probe-android/releases/tag/v3.7.3) and OONI Probe iOS 3.73 (https://apps.apple.com/us/app/id1199566366) with bug fixes and improvements. We worked towards ensuring that OONI Probe Mobile uses the check-in API instead of the test list API (https://github.com/ooni/probe/issues/2390), we fixed broken Android tests (https://github.com/ooni/probe/issues/2386), and we worked towards ensuring that the testing of OONI Run links does not stop if a device is locked (https://github.com/ooni/probe/issues/2361).
## Published a new OONI Run user guide
To enable community members to use the OONI Run (https://run.ooni.io/) platform for custom website testing (and for coordinating censorship measurement campaigns around the world), we created and published a new user guide for OONI Run.
Our OONI Run user guide can be accessed here: https://ooni.org/support/ooni-run/
Through the OONI Run user guide, you can learn how to:
- Create OONI Run links for testing websites (of your choice) for censorship;
- Use OONI Run links with the OONI Probe mobile app;
- Share OONI Run links with other OONI Probe mobile app users;
- Generate OONI Run widget code.
Upon reading this guide, we hope you will feel empowered to share your OONI Run knowledge and skills with others! We also hope this guide helps with coordinating OONI Probe website testing around the world.
## OONI Probe CLI
We released OONI Probe CLI 3.16.6 (https://github.com/ooni/probe-cli/releases/tag/v3.16.6) and OONI Probe 3.16.7 (https://github.com/ooni/probe-cli/releases/tag/v3.16.7). These two releases include improvements to the experimental Web Connectivity v0.5 nettest, as well as the following important fixes:
-
In OONI Probe CLI 3.16.6, we included the reliability fix (developed in November) that makes OONI Probe try all the available test helpers, rather than just the first one in the list;
-
In OONI Probe CLI 3.16.7, we backported fixes for the WhatsApp and Telegram experiments that stop measuring unencrypted web endpoints to avoid possible false positives (an issue discussed in detail here: https://github.com/ooni/ooni.org/issues/1317).
We continued the process of merging ooniprobe (the official CLI client) and miniooni (the experimental CLI client) by introducing a common database abstraction that could be used by both in the following pull request: https://github.com/ooni/probe-cli/pull/988
## OONI Probe Desktop
We released OONI Probe Desktop 3.8.3 to include fixes from OONI Probe CLI 3.16.7 https://github.com/ooni/probe-desktop/releases/tag/v3.8.3
## Expanding OONI’s testing model to support richer testing input
To improve the quality of OONI measurements (and reduce the risk of false positives), we stopped measuring unencrypted web endpoints for WhatsApp and Telegram (https://github.com/ooni/ooni.org/issues/1317). We continued our refactoring work to support providing richer testing input (https://github.com/ooni/ooni.org/issues/1291, https://github.com/ooni/ooni.org/issues/1292, https://github.com/ooni/probe/issues/2362).
During December 2022, we focused on cleaning up “probeservices” (the Go package that communicates with the OONI API) to set the stage for implementing richer input. We removed support for deprecated APIs (https://github.com/ooni/probe/issues/2380), we started to sync up the data format with the backend (https://github.com/ooni/probe-cli/pull/996), and we added support in the backend and in ooniprobe for receiving compressed responses (https://github.com/ooni/probe/issues/2379). The latter change, in particular, is required because adding more testing options to each URL significantly increases the size of the response message. We then annotated code to communicate with the backend with the request and response message types (https://github.com/ooni/probe-cli/pull/1010, https://github.com/ooni/probe-cli/pull/1011). This change allows us to generate a swagger describing ooniprobe’s view of the API, which we can compare to the one generated by the server, to be sure the two implementations are (and stay) in sync.
Regarding richer input, we started sketching out a design document (https://github.com/ooni/probe/issues/2381) and a prototype (https://github.com/ooni/probe-cli/pull/1005) implementing this design document. The general idea is to give experiments freedom to call the specific backend API they require for getting their inputs, which removes the need to maintain middleware code that delivers inputs to the experiments. With more input diversity, such a middleware code becomes increasingly the bottleneck to innovation, so we want to experiment with removing it and adding additional functions that experiments can call to implement common functionality instead.
We also added support to the Web Connectivity Test Helper (TH) for measuring HTTP/3 hosts (https://github.com/ooni/probe-cli/pull/985). Support for HTTP/3 in the TH is crucial to start measuring websites using HTTP/3 in the Web Connectivity experiment.
Reflecting on our own experience writing experiments in the step-by-step style (https://github.com/ooni/probe-cli/blob/master/docs/design/dd-003-step-by-step.md), as well as by learning from users’ contributions (https://github.com/ooni/probe-cli/pull/989#discussion_r1032544878), we determined that the step-by-step API provides the basic building blocks for writing all the experiments we need, but there is also a need to have more user-friendly and intuitive wrappers on top of it. To this end, we started experimenting with a simple domain specific language for expressing and composing low-level operations that should simplify the writing of experiments (https://github.com/bassosimone/oonidsl/).
## OONI Explorer
We started working towards integrating relevant third-party data into OONI Explorer. Specifically, we worked on third-party data visualizations on country and network pages using the internet outage data from IODA and internet traffic data from Cloudflare. We also continued conducting UX research to improve the OONI Explorer measurement pages (https://6q8h6s64.optimalworkshop.com/optimalsort/avnrhnv5/sort).
We added support for choosing the time granularity on the Measurement Aggregation Toolkit (MAT), beyond day granularity (which has been the only option so far). Users can now also display the data per hour, week and month (https://github.com/ooni/explorer/pull/818).
## Published an OONI Explorer demo
To enable community members to use OONI Explorer (https://explorer.ooni.org/) for investigating internet censorship based on OONI data, we published an OONI Explorer demo on our YouTube channel.
The OONI Explorer demo is available here: https://www.youtube.com/watch?v=6Rce-xshLac
## OONI backend
In response to a change in the testing of WhatsApp that made our tests fail, we implemented a temporary workaround that restores the functionality of the test (https://github.com/ooni/pipeline/pull/403).
We added support for HTTPS to the deb.ooni.org Debian package repository after it was requested by the community to circumvent blocking of the HTTP version. We also moved the service to the new backend host.
We enabled gzip support in Nginx to compress responses from the check-in API and other JSON content and created a simple dashboard to monitor the improvement. This makes the responses from the Measurement Aggregation Toolkit (MAT) quicker for the browser to download.
## Automating censorship detection and characterization based on OONI measurements
As part of our ongoing effort to automate and improve our characterization of censorship events through the analysis of OONI measurements, we made significant progress on the ooni/data tool (https://github.com/ooni/data).
Following internal technical discussions, as well as conversations with members of the OONI community, we implemented a new iteration of the data model used for representing the results of experiments (https://github.com/ooni/data/pull/19).
Moreover, we started the process of evaluating the new analysis engine against the analysis that currently exists and is mostly performed as part of the measurement workflow (https://github.com/ooni/data/issues/21). Some very preliminary findings suggest that it’s quite an important improvement both in terms of accuracy and recall.
We also experimented with producing plots that breakdown the means through which blocking is implemented (https://github.com/ooni/data/issues/22).
Smaller incremental improvements include:
Handling the deduplication of already processed data to support the easier reprocessing of data (https://github.com/ooni/data/pull/19/commits/eea26f7138b14fc4595d55b432d287bb96f2fa6a);
Expanding cloud provider detection to reduce DNS false positives (https://github.com/ooni/data/pull/19/commits/01dc691d695dfc103d604846ad7568423d77375a);
Refactoring of the DNS anomaly analysis subsystem (https://github.com/ooni/data/pull/20).
We also started implementing the extraction of the blocking type in the fastpath pipeline.
## Creating a Social Media Censorship Alert System
We continued to work on the development of the Social Media Blocking Event detector. We started running a prototypal version, created a dashboard and a jupyter notebook. We ran the prototype for weeks to review the initial results and improve the algorithm over time.
## Published Test Lists Editor screencast
To enable community members to contribute websites for censorship testing, we created and published a new screencast for our Test Lists Editor (https://test-lists.ooni.org/).
The Test Lists Editor screencast can be viewed here: https://www.youtube.com/watch?v=6i2OVHUQEpE
Through our Test Lists Editor, you can review and contribute websites for censorship testing by OONI Probe users around the world.
## TikTok blocked in Jordan
On 16th December 2022, Jordan blocked access to TikTok amid fuel protests. We rapidly responded by sharing relevant OONI data, charts, and information about the block on Twitter: https://twitter.com/OpenObservatory/status/1603845292343889924
## Test list updates
Following the blocking of TikTok in Jordan, we added TikTok endpoints to the test list for Jordan (https://github.com/citizenlab/test-lists/pull/1201). Thanks to our URL prioritization system, these newly added URLs were immediately prioritized for testing in Jordan as soon as the pull request was merged (enabling the immediate collection of relevant measurements and supporting rapid response efforts).
Throughout December 2022, we also reviewed and merged many other updates to test lists contributed by community members: https://github.com/citizenlab/test-lists/pulls?q=is%3Apr+is%3Aclosed
## Creating a new OONI Outreach Kit
Based on review and feedback, we continued to make improvements to the materials and resources of the upcoming OONI Outreach Kit. Specifically, we made extensive edits and improvements to the set of OONI workshop slides included in the OONI Outreach Kit.
## Updated OONI Partner pages
On our website, we have an “OONI Partners” section which features our partners, their important work, and the specific projects and research reports that we’ve collaborated on (https://ooni.org/partners).
We updated many OONI Partner pages to include our latest collaborations: https://github.com/ooni/ooni.org/pull/1328
## Cloudflare blog post
Cloudflare published a blog post describing how civil society organizations use Cloudflare Radar (https://radar.cloudflare.com/) to track internet shutdowns around the world: https://blog.cloudflare.com/partnering-with-civil-society-to-track-shutdowns/
This blog post highlights OONI, where we share how Cloudflare Radar has been useful to our work.
## Community use of OONI data
### iMAP 2022: Research reports on internet censorship in 8 Asian countries
On 21st December 2022, our long-term partner, Sinar Project, published 8 new research reports on internet censorship in Southeast Asia and Hong Kong (China) in collaboration with their Internet Monitoring and Action Project (iMAP) partners (https://imap.sinarproject.org/). Based on the analysis of OONI data, these research reports investigate internet censorship in Malaysia, Thailand, Myanmar, Indonesia, Cambodia, Vietnam, Hong Kong, and the Philippines.
Their research reports can be found here: https://imap.sinarproject.org/reports/2022
Our iMAP partners launched and presented these 8 research reports through the following live-streamed presentation: https://www.youtube.com/watch?v=u9JsCqXebPI
To highlight and help disseminate their important work, we published a blog post which briefly discusses and links to their 8 research reports: https://ooni.org/post/2022-imap-8-research-reports-southeast-asia/
## Community activities
### Internet Governance Forum (IGF) 2022
On 1st December 2022, OONI’s Maria presented OONI on the panel “Help! The Kill switch is taking away my limited agency” at the Internet Governance Forum (IGF) 2022 (https://intgovforum.org/en/content/igf-2022-town-hall-54-help-the-kill-switch-is-taking-away-my-limited-agency).
### OONI workshop for journalists in Kyrgyzstan
On 20th December 2022, OONI’s Elizaveta facilitated an online OONI workshop for journalists in Kyrgyzstan as part of a training organized by Internews. During this workshop, Elizaveta explained how journalists can use OONI Probe and OONI data to investigate internet censorship.
## Userbase
In December 2022, 57,009,977 OONI Probe measurements were collected from 2,815 AS networks in 161 countries around the world.
This information can also be found through our measurement stats on OONI Explorer (see chart on “monthly coverage worldwide”): https://explorer.ooni.org/
~ OONI team.