Telegram-SpottedDMI-Bot

Telegram-SpottedDMI-Bot is the platform that powers @Spotted_DMI_Bot, a Telegram bot that let students send an anonymous message to the channel community.

🔴 Using the live version

The bot is live on Telegram with the username @Spotted_DMI_Bot. To see the posts, once published, check Spotted DMI Send '/start' to start it, '/help' to see a list of commands.

Please note that the commands and their answers are in Italian.

🤖 Telegram bot setup

If you want to deploy your own version of this bot, you will need to have a telegram bot available. You should read this guide for more details, but in short:

• Send a message to @Botfather
• Follow the guided procedure
• You will receive a token. Whoever knows that token has complete control over your bot, so handle it with care
• You will need that token later, for it is a needed value in the settings

📂 Project structure

.
├── .devcontainer # DevContainer configuration for VsCode
├── .github/workflows # CI/CD workflows
├── docs # Documentation
├── scripts # Utility script for setting up the project
├── src
│   ├── spotted # Main package
│   │   ├── __init__.py # Init file for the package
│   │   ├── __main__.py # Entry point for the package
│   │   ├── config # Configuration files
│   │   ├── data # Data related function
│   │   ├── debug # Debug related functions
│   │   ├── handlers # Collection of handlers for the bot
│   │   └── utils # Utility functions
├── tests # Tests
│   ├── e2e # End to end tests
│   ├── integration # Integration tests
│   └── unit # Unit tests
├── Dockerfile # Dockerfile used to build the image
├── pyproject.toml # Package configuration file
└── README.md # This file

💻 Setting up a local instance for contributing (from git)

System requirements

• Python 3 (3.10)
• pip3

Dependencies

All the requirements are listed in the pyproject.toml file. The most important ones are:

• python-telegram-bot
• requests
• PyYAML

Install with pip3

# [Optional] Create a virtual environment
python3 -m venv .venv

# [Optional] Activate the virtual environment
source .venv/bin/activate # Linux
.venv\Scripts\activate.bat # Windows

# Using the -e flag will install the package in editable mode
# If you change the code, you won't need to reinstall the package
# If you want to install it in a non-editable mode, just remove the -e flag
pip3 install -e .

Steps

• Clone this repository
• Create a "settings.yaml" file and edit the desired parameters. It must contain at least a valid 'token' and 'post.admin_group_id' values.
  • You could also skip the files and use environment variables instead.
• Make sure the bot is in present both in the admin group and in the spot channel. It may need to have admin privileges. If comments are enabled, the bot has to be in the comment group too as an admin.
• Run python3 -m spotted to start the bot

💻 Setting up a local instance for running (from pip)

System requirements

• Python 3 (3.10)
• pip3

Install with pip3

Install the package:

pip3 install telegram-spotted-dmi-bot

Steps

• Create a "settings.yaml" file and edit the desired parameters. It must contain at least a valid 'token' and 'post.admin_group_id' values.
  • You could also skip the files and use environment variables instead.
• Run python3 -m spotted to start the bot

🐳 Setting up a Docker container

System requirements

• Docker

Steps

• Clone this repository
• Make sure the bot is in present both in the admin group and in the spot channel. It may need to have admin privileges. If comments are enabled, the bot has to be in the comment group too as an admin.
• Run docker build --tag spotted-image .
• When starting the container, use environment variables to configure the bot.
• Run docker run -d --name spotted -e TOKEN=<token_arg> [other env vars] spotted-image

Tip

When it is built, the container will copy the "settings.yaml" and the "autoreplies.yaml" files, in the root directory, if present. While it is possible to provide the configuration this way, it is recommended to use the environment variables instead, as they will override the values from the files.

Warning

The database file created inside the container will be lost when the container is removed. If you want to keep the database, you should mount a volume to the container. You can do so by adding the -v <host_path>:<container_path> flag to the docker run command. See the examples for more details.

Examples

First run

docker build --tag spotted-image .

Then something like

docker run -d --name spotted-container -e TOKEN=<token_arg> -e POST_CHANNEL_ID=-4 -e POST_GROUP_ID=-5 -e POST_CHANNEL_GROUP_ID=-6 spotted-image

If you want to check the logs in real time, you can run

# -f flag to follow the logs, otherwise it will just print the last ones
docker logs -f spotted-container

If you want to access the container's shell, you can run

docker exec -it spotted-container /bin/bash

Mounting a volume

If you want to keep the database file, you can mount a volume to the container. This will assume the default database path, which is "./spotted.sqlite3".

# The file will survive the container removal, stored in the volume "spotted-db"
docker run -d --name spotted-container -v spotted-db:/app -e TOKEN=<token_arg> -e POST_CHANNEL_ID=-4 -e POST_GROUP_ID=-5 -e POST_CHANNEL_GROUP_ID=-6 spotted-image

If you want to be able to access the database file from the host, you can use a bind mount instead.

# The file will be available in the host under the path "./spotted.sqlite3"
docker run -d --name spotted-container -v .:/app -e TOKEN=<token_arg> -e POST_CHANNEL_ID=-4 -e POST_GROUP_ID=-5 -e POST_CHANNEL_GROUP_ID=-6 spotted-image

Package version

If you want to change the version displayed by the package inside the container, you can do when building the image:

docker build --tag spotted-image --build-arg VERSION=<version> .
# Example
docker build --tag spotted-image --build-arg VERSION=3.0.0 .

Keep in mind that this will only change the version displayed by python3 -m spotted --version, not the version of the package itself.

To stop/remove the container:

• Run docker stop spotted-container to stop the container
• Run docker rm -f spotted-container to remove the container

📦 DevContainer

The VsCode Remote - Containers extension lets you use a Docker container as a full-featured development environment. It allows you to open any folder inside (or mounted into) a container and take advantage of VsCode's full feature set.

System requirements

• VsCode
• Docker

Steps

• Start VsCode, run the Remote-Containers: Reopen in container command from the Command Palette (F1)

Running the bot

After installation, the bot can be started with the command:

python3 -m spotted

There are a few command-line arguments you can use to customize the bot behaviour:

• --settings or -s: path to the "settings.yaml" file. Default: "./settings.yaml" (relative to the pwd)
• --auto-replies or -a: path to the "auto_replies.yaml" file. Default: "./auto_replies.yaml" (relative to the pwd)

Also, keep in mind that the bot will generate an sqlite database file in the path indicated by the settings.yaml file named "spotted.sqlite3" by default. The file will be created if missing, but the path must be valid. Lastly, if logs are enabled, the bot will log under the path specified in settings.yaml. By default, it would be "logs/spotted.log" and "logs/spotted_error.log". The path will be created if it does not exist.

Settings

When it is initialized, the bot reads both the "config/yaml/autoreplies.yaml" and the "config/settings.yaml" files inside the package, which contain the default values for the settings. Then the configuration gets overwritten using the "settings.yaml" and the "auto_replies.yaml" files you provide. The bot expects to find both files in the pwd directory, meaning the directory from which you run the bot. You can change this behaviour by specifying the path to the files with the --config and --auto-replies flags.

python3 -m spotted -c /path/to/settings.yaml -a /path/to/autoreplies.yaml

Feel free to customize the settings file,but make sure to add a valid token and post.admin_group_id values, since they are mandatory.

# config/settings.yaml
debug:
  local_log: false # save each and every message in a log file. Make sure the path "logs/messages.log" is valid when enabled
  reset_on_load: false # whether or not the database should reset every time the bot launches. USE CAREFULLY
  log_file: "logs/spotted.log" # path to the log file, if local_log is enabled. Relative to the pwd
  log_error_file: "logs/spotted_error.log" # path to the error log file. Relative to the pwd
  db_file: "spotted.sqlite3" # path to the database file. Relative to the pwd
  # key used to encrypt the database backup sent to the admin periodically.
  # Must be 32 bytes long and base64 encoded.
  # If not specified (default), the backup will be sent in clear text
  crypto_key: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

post:
  # id of the group associated with the channel. Telegram will send automatically forward the posts there. Required if comments are enabled
  community_group_id: -100
  channel_id: -200 # id of the channel to which the bot will send the approved posts
  channel_tag: "@channel" # tag of the channel to which the bot will send the approved posts
  comments: true # whether or not the channel the bot will send the posts to has comments enabled
  admin_group_id: -300 # id of the admin group the bot will use
  n_votes: 2 # votes needed to approve/reject a pending post
  remove_after_h: 12 # pending posts older than this number of hours will be removed by /clean_pending
  report_wait_mins: 30 # number of minutes the user has to wait before being able to report another user again
  report:
    true # whether to add a report button as an inline keyboard after each post
    # whether the bot should delete any anonym comment coming from a channel.
    # The bots must have delete permission in the group and comments must be enabled
  delete_anonymous_comments:
    true
    # whether the bot should replace any anonymous comment with a message by itself.
    # WARNING: delete_anonymous_comments must be true for this option to make sense.
    # Otherwise the comment would be doubled.
    # The bots must have delete permission in the group and comments must be enabled
  replace_anonymous_comments: false
  blacklist_messages: []
    # example: ["spam_word_1", "spam_word_2"]
    # the bot will delete any comment in the community chat that includes a word of the blacklist

token: xxxxxxxxxxxx # token of the telegram bot
bot_tag: "@bot" # tag of the telegram bot

Settings override

The settings may also be set through environment variables.
All the env vars with the same name (case insensitive) will override the ones in the settings file. To update the post settings, prefix the env var name with POST_. The same is true for the debug settings, to be prefixed with DEBUG_.

# Environment values
TOKEN=xxxxxx        # will override the *token* value found in settings.yaml
POST_N_VOTES=4      # will override the *post.n_votes* value found in settings.yaml
DEBUG_LOCAL_LOG=4   # will override the *debug.local_log* value found in settings.yaml

The complete order of precedence for the application of configuration settings is

env var > (user provided) settings.yaml > (package provided) settings.yaml

Note

The bot provides a /reload command that will reload the settings without restarting the bot. Keep in mind that the order of precedence will be the same, meaning that the env vars will always override the settings.yaml files.

Since every setting has a default value specified in the default settings.yaml except for token and the chat_ids of the groups and the channel, those are the only necessary setting to add when setting up the bot for the first time. All other values will be assigned a valid default value if not specified.

Settings typing

Typings are provided by default for eny value specified through the .yaml files. On the other hand, if you use the environment variables, everything will be treated as a string.
To prevent this, you can use the settings.yaml.types specifying a type for each setting. This way it will be casted in the specified type.

Supported types:

• bool
• int
• float
• list (of str)

Any unknown type won't be casted. This means that env var will remain strings.

# settings.yaml.types
debug:
  local_log: bool
  reset_on_load: bool
  log_file: str
  log_error_file: str
  db_file: str
  crypto_key: str
post:
  community_group_id: int
  channel_id: int
  channel_tag: str
  comments: bool
  admin_group_id: int
  n_votes: int
  remove_after_h: int
  report: bool
  report_wait_mins: int
  replace_anonymous_comments: bool
  delete_anonymous_comments: bool
token: str
bot_tag: str

🧪 [Optional] Setting up testing and linting

If you plan to contribute to this project, you may want to run the tests and the linters locally.

Note

When creating a pull request, all the tests and linters will be run automatically using the github actions. If the tests fail, the pull request won't be merged until all the errors are fixed. Hence, it is recommended to run the tests and the linters locally before pushing your changes.

Dependencies

All the dev requirements are listed in the pyproject.toml file under [project.optional-dependencies].

• pytest
• pytest-cov
• pytest-mock
• pytest-asyncio
• pylint
• black
• isort

Install with pip3

After having cloned the repository and having installed the main package with pip3 install -e ., you can install the dev dependencies with:

pip3 install -e .[test]
pip3 install -e .[lint]

Testing:

• Run pytest tests/unit to run the unit tests
• Run pytest tests/integration to run the integration tests

Linting:

• Run pylint src tests to lint the code
• Run black --check src tests to make sure the code is formatted correctly. If it is not, you can run black src to format it automatically
• Run isort --check-only src tests to make sure the imports are sorted correctly. If they are not, you can run isort src to sort them automatically

Scripts

The script folder contains some utility scripts such as script/local-ci.sh that can be used to simulate the whole CI pipeline locally. Make sure to have the dev dependencies installed before running them.

Furthermore, the package provides some utility scripts:

• run_sql script that can be used to run an arbitrary sql script on the indicated sqlite3 database.
• f_crypto script that can be used to encrypt/decrypt files with a key or generate a new key.

run_sql <path_to_sql_script> <path_to_db_file>
# Example
run_sql ./script/create_db.sql ./spotted.sqlite3

f_crypto <encrypt|decrypt> <input_file> <output_file> --key <key>
# Example
f_crypto encrypt ./spotted.sqlite3 ./spotted.sqlite3.enc --key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Generate a new key
f_crypto generate_key

Important

Make sure to backup your database before running any scripts on it, since all alteration will happen in place, and there is no way to undo them.

📚 Documentation

Link to the documentation