Alert Report #30343

fti-starawade · 2024-09-20T06:17:56Z

fti-starawade
Sep 20, 2024

@rohitpawar2811 I am trying to send report via SMTP using superset_config from #22099 (comment) but I am facing issues with docker-compose.yaml having superset worker celery beat and celery worker
I am providing the docker-compose.yaml

version: '3.7'

networks:
  sdk_network:
    external:
      name: sdk_bridge_network
  system_network:
    external:
      name: system_bridge_network

services:
  # Metabase service
  metabase:
    build:
      context: ./metabase
    container_name: metabase
    hostname: metabase
    volumes:
      - data.vol:/dev/metabase
    expose:
      - 3000
    ports:
      - "3000:3000"
    networks:
      - system_network
      - sdk_network
    restart: always
    tty: true

  # Superset app container
  superset:
    image: sdk_superset-x64_vaapi:1.0.0
    build:
      context: ./superset
      args:
        - USERNAME=$USERNAME
        - FIRSTNAME=$FIRSTNAME
        - LASTNAME=$LASTNAME
        - EMAIL=$EMAIL
        - PASSWORD=$PASSWORD
    environment:
      USERNAME: ${USERNAME}
      PASSWORD: ${PASSWORD}
      SUPERSET_WEBDRIVER_TYPE: "chrome"
    volumes:
      - data.vol:/data
      - /etc/localtime:/etc/localtime:ro
      - /tmp/.X11-unix:/tmp/.X11-unix
    expose:
      - "8088"
    ports:  
      - "9097:8088"
    networks:
      sdk_network:
        aliases:
          - superset
    restart: always
    tty: true

  # Celery worker for async tasks
  superset-worker:
    build:
      context: ./superset
    container_name: superset_worker
    command: ["celery", "--app=superset.tasks.celery_app:app", "worker", "--pool=prefork", "-O", "fair", "-c", "4"]
    env_file:
      - .env
    environment:
      CELERYD_CONCURRENCY: 2
    restart: unless-stopped
    depends_on:
      - superset
    volumes:
      - data.vol:/data
    extra_hosts:
      - "host.docker.internal:host-gateway"
    healthcheck:
      test: ["CMD-SHELL", "celery -A superset.tasks.celery_app:app inspect ping -d celery@$$HOSTNAME"]

  # Celery beat for scheduling tasks
  superset-worker-beat:
    build:
      context: ./superset
    container_name: superset_worker_beat
    command: ["celery", "--app=superset.tasks.celery_app:app", "beat"]
    env_file:
      - .env
    restart: unless-stopped
    depends_on:
      - superset
    volumes:
      - data.vol:/data
    healthcheck:
      disable: true

  # Redis message broker
  redis:
    image: redis:6.0-alpine
    container_name: superset_cache
    networks:
      - sdk_network
    restart: always

  # PostgreSQL database
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: superset
      POSTGRES_PASSWORD: superset
      POSTGRES_DB: superset
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - sdk_network
    restart: always

volumes:
  data.vol:
    external: true
  postgres_data:
    external: false
  celery_beat_data:
    external: false

And below is superset worker dockerfile.worker

FROM apache/superset:3.1.0

USER root

RUN apt-get update && \
    apt-get install -y wget zip libaio1

# Add Google Chrome repository and install Google Chrome
RUN wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | apt-key add - && \
    echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" | tee /etc/apt/sources.list.d/google-chrome.list && \
    apt-get update && apt-get install -y --no-install-recommends google-chrome-stable && \
    rm -rf /var/lib/apt/lists/*

# Install ChromeDriver
RUN CHROMEDRIVER_VERSION=$(wget -q -O - https://chromedriver.storage.googleapis.com/LATEST_RELEASE) && \
    wget -q "https://chromedriver.storage.googleapis.com/${CHROMEDRIVER_VERSION}/chromedriver_linux64.zip" && \
    unzip chromedriver_linux64.zip -d /usr/local/bin && \
    chmod +x /usr/local/bin/chromedriver && \
    rm chromedriver_linux64.zip

# Install necessary Python packages
RUN pip install --no-cache gevent psycopg2 redis

# Ensure the directory for Celery Beat exists and has the right permissions
RUN mkdir -p /usr/local/var/celerybeat && \
    chown -R superset:superset /usr/local/var/celerybeat

USER superset

I followed the https://superset.apache.org/docs/configuration/alerts-reports/#browse-to-your-report-from-the-worker

@fti-starawade · 2024-09-20T06:21:20Z

dosubot[bot]
bot Sep 20, 2024

Hey there, @fti-starawade! I'm here to help you with any bugs, questions, or contributions you have. Let's dive into this together!

I found some similar discussions and issues that might be relevant to your problem:

Enable Alerting in Superset - Unsolved ^[1].
Alerts & Reporting - Selenium timed out / DNS Resolution - Unsolved ^[2].
Alerting mechanism of apache superset does not work at all - Unsolved ^[3].
Unable to start superset with reporting via docker-compose - Closed ^[4].

Here is how you can configure your docker-compose.yaml and superset_config.py to send reports via SMTP using Apache Superset with Celery workers and Redis:

`docker-compose.yaml`

Ensure your docker-compose.yaml includes the necessary services: Redis, PostgreSQL, Celery worker, and Celery beat. Here is an example configuration:

version: '3.7'

networks:
  sdk_network:
    external:
      name: sdk_bridge_network
  system_network:
    external:
      name: system_bridge_network

services:
  superset:
    image: sdk_superset-x64_vaapi:1.0.0
    build:
      context: ./superset
      args:
        - USERNAME=$USERNAME
        - FIRSTNAME=$FIRSTNAME
        - LASTNAME=$LASTNAME
        - EMAIL=$EMAIL
        - PASSWORD=$PASSWORD
    environment:
      USERNAME: ${USERNAME}
      PASSWORD: ${PASSWORD}
      SUPERSET_WEBDRIVER_TYPE: "chrome"
    volumes:
      - data.vol:/data
      - /etc/localtime:/etc/localtime:ro
      - /tmp/.X11-unix:/tmp/.X11-unix
    expose:
      - "8088"
    ports:  
      - "9097:8088"
    networks:
      sdk_network:
        aliases:
          - superset
    restart: always
    tty: true

  superset-worker:
    build:
      context: ./superset
    container_name: superset_worker
    command: ["celery", "--app=superset.tasks.celery_app:app", "worker", "--pool=prefork", "-O", "fair", "-c", "4"]
    env_file:
      - .env
    environment:
      CELERYD_CONCURRENCY: 2
    restart: unless-stopped
    depends_on:
      - superset
    volumes:
      - data.vol:/data
    extra_hosts:
      - "host.docker.internal:host-gateway"
    healthcheck:
      test: ["CMD-SHELL", "celery -A superset.tasks.celery_app:app inspect ping -d celery@$$HOSTNAME"]

  superset-worker-beat:
    build:
      context: ./superset
    container_name: superset_worker_beat
    command: ["celery", "--app=superset.tasks.celery_app:app", "beat"]
    env_file:
      - .env
    restart: unless-stopped
    depends_on:
      - superset
    volumes:
      - data.vol:/data
    healthcheck:
      disable: true

  redis:
    image: redis:6.0-alpine
    container_name: superset_cache
    networks:
      - sdk_network
    restart: always

  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: superset
      POSTGRES_PASSWORD: superset
      POSTGRES_DB: superset
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - sdk_network
    restart: always

volumes:
  data.vol:
    external: true
  postgres_data:
    external: false
  celery_beat_data:
    external: false

`superset_config.py`

Add the following configurations to your superset_config.py to enable alerts and reports, configure Celery, and set up SMTP:

from celery.schedules import crontab

FEATURE_FLAGS = {
    "ALERT_REPORTS": True
}

REDIS_HOST = "superset_cache"
REDIS_PORT = "6379"

class CeleryConfig:
    broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    imports = (
        "superset.sql_lab",
        "superset.tasks.scheduler",
    )
    result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    worker_prefetch_multiplier = 10
    task_acks_late = True
    task_annotations = {
        "sql_lab.get_sql_results": {
            "rate_limit": "100/s",
        },
    }
    beat_schedule = {
        "reports.scheduler": {
            "task": "reports.scheduler",
            "schedule": crontab(minute="*", hour="*"),
        },
        "reports.prune_log": {
            "task": "reports.prune_log",
            "schedule": crontab(minute=0, hour=0),
        },
    }

CELERY_CONFIG = CeleryConfig

SCREENSHOT_LOCATE_WAIT = 100
SCREENSHOT_LOAD_WAIT = 600

# Email configuration
SMTP_HOST = "smtp.sendgrid.net"  # change to your host
SMTP_PORT = 2525  # your port, e.g. 587
SMTP_STARTTLS = True
SMTP_SSL_SERVER_AUTH = True  # If your using an SMTP server with a valid certificate
SMTP_SSL = False
SMTP_USER = "your_user"  # use the empty string "" if using an unauthenticated SMTP server
SMTP_PASSWORD = "your_password"  # use the empty string "" if using an unauthenticated SMTP server
SMTP_MAIL_FROM = "[email protected]"
EMAIL_REPORTS_SUBJECT_PREFIX = "[Superset] "  # optional - overwrites default value in config.py of "[Report] "

# WebDriver configuration
WEBDRIVER_TYPE = "chrome"
WEBDRIVER_OPTION_ARGS = [
    "--force-device-scale-factor=2.0",
    "--high-dpi-support=2.0",
    "--headless",
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--disable-extensions",
]
WEBDRIVER_BASEURL = "http://superset:8088"  # When running using docker compose use "http://superset_app:8088"
WEBDRIVER_BASEURL_USER_FRIENDLY = "http://localhost:8088"

# User for rendering dashboards
from superset.tasks.types import ExecutorType

THUMBNAIL_SELENIUM_USER = 'admin'
ALERT_REPORTS_EXECUTE_AS = [ExecutorType.SELENIUM]

Dockerfile for Celery Worker

Ensure the Dockerfile for the Celery worker installs the necessary dependencies:

FROM apache/superset:3.1.0

USER root

RUN apt-get update && \
    apt-get install -y wget zip libaio1

# Add Google Chrome repository and install Google Chrome
RUN wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | apt-key add - && \
    echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" | tee /etc/apt/sources.list.d/google-chrome.list && \
    apt-get update && apt-get install -y --no-install-recommends google-chrome-stable && \
    rm -rf /var/lib/apt/lists/*

# Install ChromeDriver
RUN CHROMEDRIVER_VERSION=$(wget -q -O - https://chromedriver.storage.googleapis.com/LATEST_RELEASE) && \
    wget -q "https://chromedriver.storage.googleapis.com/${CHROMEDRIVER_VERSION}/chromedriver_linux64.zip" && \
    unzip chromedriver_linux64.zip -d /usr/local/bin && \
    chmod +x /usr/local/bin/chromedriver && \
    rm chromedriver_linux64.zip

# Install necessary Python packages
RUN pip install --no-cache gevent psycopg2 redis

# Ensure the directory for Celery Beat exists and has the right permissions
RUN mkdir -p /usr/local/var/celerybeat && \
    chown -R superset:superset /usr/local/var/celerybeat

USER superset

Make sure the ALERT_REPORTS_NOTIFICATION_DRY_RUN setting is set to False in your superset_config.py to enable notifications:

ALERT_REPORTS_NOTIFICATION_DRY_RUN = False

This configuration should help you set up your environment to send reports via SMTP using Apache Superset with Celery workers and Redis ^[5]^[4].

To continue talking to Dosu, mention @dosu.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alert Report #30343

{{title}}

Replies: 1 comment

{{title}}

Select a reply

Alert Report #30343

fti-starawade Sep 20, 2024

Replies: 1 comment

dosubot[bot] bot Sep 20, 2024

docker-compose.yaml

superset_config.py

Dockerfile for Celery Worker

fti-starawade
Sep 20, 2024

dosubot[bot]
bot Sep 20, 2024

`docker-compose.yaml`

`superset_config.py`