backend: Create a social media censorship alert system

[hellais]

Major social media platforms (such as Facebook, WhatsApp, and Twitter) are frequently blocked around the world during political events (such as elections or large-scale protests). Over the years, OONI data has demonstrated the blocking of social media platforms in numerous countries (such as Uganda, Mali, Benin, Togo, Burundi, Tanzania and Zambia) during elections. OONI data has also shown the blocking of social media platforms during Cuba’s July 2021 anti-government protests, Zimbabwe’s 2019 fuel protests, and Iran’s 2017-2018 anti-government protests (among many other cases worldwide). Following the February 2021 military coup in Myanmar, ISPs in the country started blocking access to social media platforms. Following the February 2022 invasion of Ukraine, Russia started blocking access to Facebook and Twitter.

To enable human rights defenders to rapidly respond to social media blocks, we aim to create a Social Media Censorship Alert System which allows access to real-time OONI data on the blocking of major social media platforms around the world. This will be particularly useful to advocacy groups such as the #KeepItOn campaign, which makes use of OONI data to fight social media shutdowns worldwide.

As part of this proposed project, we aim to develop a system that the public can refer to in order to learn when and where access to social media platforms is being restricted around the world based on empirical OONI data. This will entail developing event detection capabilities in the OONI data processing pipeline (with a focus on reducing the ratio of false positives) and exposing this information to end users.

[FedericoCeratto]

In 2019 we developed the first event detector as an experiment to measure how effective it would be given the existing dataset and database.
Since then there has been very significant changes in the operating conditions including:

• Large increase in available measurements due to unattended probe runs
• Very large increase in measurements of certain URLs after implementing prioritization in the backend
• Database moved from OLTP to OLAP (PostgreSQL to Clickhouse)
• More powerful database/backend hardware
• Implemented blocking/anomaly type extraction

We are currently reusing the lessons learned from the first event detector to build a new one while reusing parts of the existing codebase where possible.
For an experimental/MVP version we want to:

• Move the codebase to Clickhouse
• Implement event detection on the database side
• Focus on selected Social Media websites (Facebook, Twitter, ...)
• Deploy on Test host
• Deploy on Prod host
• Update Grafana dashboards
• Monitor dashboards and outputs and fine-tune parameters
• Create event detection dashboard on Jupyter
• Publish Atom/RSS feeds on Nginx (not announced publicly yet)
• First iteration of code review

The next iterations will be described in dedicated issues.
Related to #206 and #520

[hellais]

I suspect this is a duplicate of: ooni/ooni.org#1300.

Closed the other one in favour of this one as it has more content in it.

[hellais]

We have a prototype of this available here: #651 and we have started writing up some documentation on how it works over here: https://docs.google.com/document/d/1WQ6_ybnPbO_W6Tq-xKuHQslG1dSPr4jUbZ3jQLaMqdw/edit

[jbonisteel]

Per roadmap discussions during team meeting - we will aim to collect feedback from researchers around the March timeframe

[jbonisteel]

I am going to close this epic, because future work is going to be tracked in other epics