Feat/spam digest job #905

BarbaraOliveira13 · 2025-10-10T10:05:20Z

✉️ Feature: Decidim-AI — Spam Summary Digest Event (Weekly / Daily)

🔧 This PR also fix a CI part by normalize nl local file.

MAIN SUBJECT :
This PR adds a Decidim digest system that aggregates spam reports detected by Decidim-AI and sends a daily or weekly summary email, to each organization’s administrators (selon leur choix de config 'frequency').

It integrates with Decidim EventsManager and EmailEvent notification system.

🧠 Technical Notes
resource: organization ensures Decidim’s BaseEvent behaves correctly
(the spam digest concerns all types of content: proposals, projects, comments, etc.)
If need more details, cf conclusion in the issue

🧭 Steps to Reproduce

Log in as an admin
Ensure Decidim-AI variables are configured in .env/.env.local:

DECIDIM_AI_BASIC_AUTH="decidimai:xxx"
DECIDIM_AI_ENABLED="true"
DECIDIM_AI_ENDPOINT=https://decidim-ai.k8s.osp.cat/spam/detection
DECIDIM_AI_GENERIC_SPAM_ANALYZER_ENABLED="true"
[email protected]
DECIDIM_AI_USER_SPAM_ANALYZER_ENABLED="true"

(Replace xxx with the password stored in Passbolt)
5. Restart the server: spring stop && bundle exec rails s
10. Create the reporting_user (Spam Bot): script for console:

user = Decidim::Organization.first.users.find_or_initialize_by(email: "[email protected]")
unless user.persisted?
  password = SecureRandom.hex(10)
  user.password = password
  user.password_confirmation = password
  user.deleted_at = Time.current
  user.tos_agreement = true
  user.name = "Spam Bot"
  user.skip_confirmation!
  user.save!
end

Start Sidekiq in a separate terminal: bundle exec sidekiq
Creates SPAM
Lot of possibilities, (exemple of SPAM: 'winamax.fr')
- Create a new user, clic on the link from the mail (http://localhost:3000/letter_opener) to log in
  - Go to his account, and add a SPAM in the 'about' part
- Add a SPAM comment
- Crate a proposal, process, assembly... with a SPAM name or description
Run the digest manually:

`Decidim::Ai::SpamDetection::SpamDigestGeneratorJob.perform_now(:weekly)` - => Before this commande: Update notifications settings to receive the right frequency (weekly with this command) `Decidim::Ai::SpamDetection::SpamDigestGeneratorJob.perform_now(:daily)` - - => Before this commande: Update notifications settings to receive the right frequency (daily with this command) - Open the generated email using LetterOpener

Expected output:

📌 Related Issues

https://github.com/orgs/OpenSourcePolitics/projects/26/views/1?filterQuery=assignee%3A%40me&pane=issue&itemId=129336496&issue=OpenSourcePolitics%7Cintern-tasks%7C114

Tasks

Add specs
In case of new dependencies or version bump, update related documentation

📷 Screenshots

Please add screenshots of the changes you're proposing if related to the UI

Extra information

For the DATA Team: ⚠️ The digest is not stored in any table — it’s generated dynamically when EventsManager.publish is called.

- Introduced SpamDigestEvent to notify admins through Decidim’s notification system - Added I18n translations for AI spam digest summaries

…mail

AyakorK

A lot of reviews, but we probably don't need to fix everything, some of them might be critical but other issues are reported just to keep a trace on it !

lib/tasks/ai_spam_digest_tasks.rake

app/events/decidim/ai/spam_detection/spam_digest_event.rb

AyakorK · 2025-11-24T08:52:33Z

app/events/decidim/ai/spam_detection/spam_digest_event.rb

+          Class.new do
+            def initialize(path, url)
+              @path = path
+              @url = url
+            end
+
+            def path(_params = nil)
+              @path
+            end
+
+            def url(_params = nil)
+              @url
+            end
+          end.new(
+            resource_path,
+            helpers.root_url(host:, protocol: "http")
+          )
+        end


I'm not a huge fan of anonymous classes, Struct might be a little more maintainable or even better an external class that would help making some specs on it (like a ResourceLocator)

app/events/decidim/ai/spam_detection/spam_digest_event.rb

AyakorK · 2025-11-24T08:58:10Z

app/jobs/decidim/ai/spam_detection/spam_digest_generator_job.rb

+            admins = organization.admins.where(notifications_sending_frequency: frequency)
+            next if admins.empty?
+
+            spam_count = count_spam(organization, frequency)


Hmm slight ⚠️ about memory usage there, Every spam report will be loaded into RAM which might cause an issue in the future

Maybe we should try to do the filter using SQL and not Ruby, but maybe try to see with other devs to check their POV

AyakorK · 2025-11-24T09:00:56Z

app/jobs/decidim/ai/spam_detection/spam_digest_generator_job.rb

+        # Returns all spam reports created since the given time
+        def spam_reports_since(since)
+          Decidim::Report
+            .joins(:moderation)
+            .where(reason: "spam")
+            .where("decidim_reports.created_at >= ?", since)
+        end
+
+        def spam_user_reports_since(since)
+          Decidim::UserReport
+            .joins(:user)
+            .where(reason: "spam")
+            .where("decidim_user_reports.created_at >= ?", since)
+        end


Great ! But we probably could refactor this as the only difference is the .joins of each method

AyakorK · 2025-11-24T09:05:29Z

app/jobs/decidim/ai/spam_detection/spam_digest_generator_job.rb

+      class SpamDigestGeneratorJob < ApplicationJob
+        queue_as :mailers
+
+        def perform(frequency)


Nice to have: We could validate the frequency parameter to avoid silent failures if an invalid value is passed.

Example but it might not be exactly what you wanted to do

FREQUENCIES = {
daily: "daily",
weekly: "weekly"
}.freeze

Then check with: raise ArgumentError unless FREQUENCIES.key?(frequency.to_sym)

AyakorK · 2025-11-24T09:12:02Z

config/initializers/instant_spam_mail_blocker.rb

+    end
+
+    def send_notification_to_admins!
+      return if spam_report? && !frequency_notifications_is_realtime?(@report.moderation.user.organization.admins)


@report.moderation.user.organization.admins
1/2 This probably loads all admins into memory just to check a boolean. We might consider using .exists? for better performance, once again it might not be a big problem as there is rarely more than 10 admins

AyakorK · 2025-11-24T09:13:15Z

config/initializers/instant_spam_mail_blocker.rb

+    end
+
+    def frequency_notifications_is_realtime?(admins)
+      admins.any? { |a| a.notifications_sending_frequency == "realtime" }


2/2

Maybe a refactor using something like this could work ? I'm not sure so please don't mind telling me if I'm mistaking

def has_realtime_admins?(organization) organization.admins.exists?(notifications_sending_frequency: "realtime") end

…don't have organization.resource_title. So I refacto using translated_attribute

…the hardcoded http protocol

…cted by mailer Decidim

… job

… spec: perform_now without (argument)

Stef-Rousset · 2025-12-10T14:01:17Z

Hello @BarbaraOliveira13 , thx for your PR !!!

I try to run the job Decidim::Ai::SpamDetection::SpamDigestGeneratorJob.perform_now(:daily) in console on local env, and I got an error

/Users/stephanierousset/.rvm/gems/ruby-3.2.2/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': ERROR:  column decidim_moderations.decidim_organization_id does not exist (PG::UndefinedColumn)
LINE 1: ...s.created_at >= '2025-12-09 13:51:44.487895') AND "decidim_m...

I think it may be related to this line https://github.com/OpenSourcePolitics/decidim-app/pull/905/files#diff-236c78d077b1c09f66a2df7f331aefd14b4c5c86155cd2a12f2f2fb42038ff26R53, because decidim_moderations table has no column decidim_organization_id

BarbaraOliveira13 · 2025-12-10T14:23:04Z

Hello @BarbaraOliveira13 , thx for your PR !!!

I try to run the job Decidim::Ai::SpamDetection::SpamDigestGeneratorJob.perform_now(:daily) in console on local env, and I got an error
/Users/stephanierousset/.rvm/gems/ruby-3.2.2/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': ERROR:  column decidim_moderations.decidim_organization_id does not exist (PG::UndefinedColumn)
LINE 1: ...s.created_at >= '2025-12-09 13:51:44.487895') AND "decidim_m...
I think it may be related to this line https://github.com/OpenSourcePolitics/decidim-app/pull/905/files#diff-236c78d077b1c09f66a2df7f331aefd14b4c5c86155cd2a12f2f2fb42038ff26R53, because decidim_moderations table has no column decidim_organization_id

Hello @Stef-Rousset Oh, I had the right chain before I think... I should changed too much time :s
I will watch and fix It!
Thanks a lot for your review ! 🙏

BarbaraOliveira13 · 2025-12-12T08:45:15Z

Fail CI is not relevant of my changes. It's a flaky CF Redesigned upload modal flaky test #10961
This PR is ready for review and more!

@Stef-Rousset you can re-run the job, I fix it ! 🙏

BarbaraOliveira13 · 2025-12-12T13:50:07Z

Next step:

Review tech Alain or Guillaume
Test directement après un déploiement sur une app de test

Stef-Rousset · 2025-12-15T08:45:24Z

Decidim::Ai::SpamDetection::SpamDigestGeneratorJob.perform_now(:daily)

hello @BarbaraOliveira13 , great job, I run the code locally and everything works !!!
Just a last quick refacto on the spam_user_reports_since method: I think you don't need .includes(:user) and reports.select { |r| r.user.decidim_organization_id == organization.id }, as the .where(decidim_users: { decidim_organization_id: organization.id }) already filters the user reports belonging to the organization.

BarbaraOliveira13 · 2026-01-05T10:59:37Z

Decidim::Ai::SpamDetection::SpamDigestGeneratorJob.perform_now(:daily)

hello @BarbaraOliveira13 , great job, I run the code locally and everything works !!! Just a last quick refacto on the spam_user_reports_since method: I think you don't need .includes(:user) and reports.select { |r| r.user.decidim_organization_id == organization.id }, as the .where(decidim_users: { decidim_organization_id: organization.id }) already filters the user reports belonging to the organization.

Thanks ! done 👍

BarbaraOliveira13 added 5 commits October 10, 2025 12:02

- Added SpamDigestGeneratorJob to aggregate spam reports (daily/weekly)

7e186fd

- Introduced SpamDigestEvent to notify admins through Decidim’s notification system - Added I18n translations for AI spam digest summaries

delete event_registration

f80fe0f

fix methode error .organization because resource = organization

d86bf4c

lint

1dbaefc

lint i18 trad key and rollback resource_locator method

e36b010

BarbaraOliveira13 changed the title ~~Spam Digest Job to aggregate spam reports (daily/weekly)~~ Feat/Spam Digest Job Oct 20, 2025

BarbaraOliveira13 changed the title ~~Feat/Spam Digest Job~~ Feat/spam digest job Oct 20, 2025

BarbaraOliveira13 and others added 16 commits October 20, 2025 14:11

run bundle exec i18n-tasks normalize --locales nl

2264031

test file for spam digest event

2e80e82

delete link from the view on decidim-core in the digest summary spam …

178f95a

…mail

patch to stop instant spam mail from messages spams

9028038

stop instant spam mail via decidim-ai also when is a spam user report

87aa452

lint spam_digest_event.rb

0886a57

delete unused trad key and clean mail text

590cc5c

add user_spam part logic who was missing

941c688

add link to moderations into email text

6dc2781

add trad key access of organization name for the text mail

4d61e46

don't stop report process when frequency notification is realtime

7bf6b05

Merge branch 'develop' into feat/decidim-ai-digest

752ae80

fix CI

e08c545

test file

9ef29b4

local from develop merged

c472dfc

normalise trad key

1d46b3a

BarbaraOliveira13 force-pushed the feat/decidim-ai-digest branch from d7719b0 to 1d46b3a Compare November 16, 2025 09:59

BarbaraOliveira13 requested a review from Stef-Rousset November 16, 2025 10:16

BarbaraOliveira13 marked this pull request as ready for review November 16, 2025 10:16

BarbaraOliveira13 requested review from AyakorK, luciegrau and moustachu November 16, 2025 10:17

Merge branch 'develop' into feat/decidim-ai-digest

03afaf2

AyakorK requested changes Nov 24, 2025

View reviewed changes

BarbaraOliveira13 added 10 commits November 26, 2025 13:25

change request: avoid js injection with sanitize

b53cd9b

change request: avoid js injection with sanitize

3a18ccd

change request: refacto with translated attribute

d960075

change request: refacto count_spam method

a9f80f1

change request: .exists? method instead of .any who load all admins...

891ac3d

Our resource here is Decidim::Organization but Decidim::Organization …

b53fce5

…don't have organization.resource_title. So I refacto using translated_attribute

refactored the anonymous class into a Struct for simplicity. Removed …

0d96783

…the hardcoded http protocol

fix CI: refacto Struct by a real class with route_name method as expe…

7c32feb

…cted by mailer Decidim

change request: add raise error if frequency is invalid when run the…

2fe3cec

… job

fix CI by adding a nil condition argument as expected by decidim-core…

465363f

… spec: perform_now without (argument)

fix the table path for spam report queries

dc52e5c

refacto spam_user_reports_since method

1753a33

BarbaraOliveira13 requested a review from AyakorK January 5, 2026 10:59

AyakorK approved these changes Jan 5, 2026

View reviewed changes

Feat/spam digest job #905

Are you sure you want to change the base?

Feat/spam digest job #905

Conversation

BarbaraOliveira13 commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📌 Related Issues

Tasks

📷 Screenshots

Extra information

Uh oh!

AyakorK left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AyakorK Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AyakorK Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

AyakorK Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

AyakorK Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

AyakorK Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

AyakorK Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Stef-Rousset commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BarbaraOliveira13 commented Dec 10, 2025

Uh oh!

BarbaraOliveira13 commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BarbaraOliveira13 commented Dec 12, 2025

Uh oh!

Stef-Rousset commented Dec 15, 2025

Uh oh!

BarbaraOliveira13 commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

BarbaraOliveira13 commented Oct 10, 2025 •

edited

Loading

Stef-Rousset commented Dec 10, 2025 •

edited

Loading

BarbaraOliveira13 commented Dec 12, 2025 •

edited

Loading