Switch to asyncio (python-telegram-bot#2731)

Co-authored-by: tsnoam <[email protected]>
Co-authored-by: Harshil <[email protected]>
Co-authored-by: Poolitzer <[email protected]>

Author: 4 people

.deepsource.toml

@@ -5,7 +5,6 @@ test_patterns = ["tests/**"]
 exclude_patterns = [
     "tests/**",
     "docs/**",
-    "telegram/vendor/**",
     "setup.py",
     "setup-raw.py"
 ]

.github/CONTRIBUTING.rst

@@ -153,12 +153,6 @@ Here's how to make a one-off code change.
         $ git commit -a
         $ git push origin your-branch-name
 
-   - If after merging you see local modified files in ``telegram/vendor/`` directory, that you didn't actually touch, that means you need to update submodules with this command:
-
-     .. code-block:: bash
-
-        $ git submodule update --init --recursive
-
    - At the end, the reviewer will merge the pull request.
 
 6. **Tidy up!** Delete the feature branch from both your local clone and the GitHub repository:
@@ -260,11 +254,12 @@ break the API classes. For example:
 
     # GOOD
     def __init__(self, id, name, last_name=None, **kwargs):
-       self.last_name = last_name
+        self.last_name = last_name
+
 
     # BAD
     def __init__(self, id, name, last_name=None):
-       self.last_name = last_name
+        self.last_name = last_name
 
 
 .. _`Code of Conduct`: https://www.python.org/psf/codeofconduct/

.github/pull_request_template.md

@@ -8,7 +8,7 @@ Hey! You're PRing? Cool! Please have a look at the below checklist. It's here to
 - [ ] Created new or adapted existing unit tests
 - [ ] Documented code changes according to the [CSI standard](https://standards.mousepawmedia.com/en/stable/csi.html)
 - [ ] Added myself alphabetically to `AUTHORS.rst` (optional)
-- [ ] Added new classes & modules to the docs
+- [ ] Added new classes & modules to the docs and all suitable `__all__` s
 
 
 ### If the PR contains API changes (otherwise, you can delete this passage)

.github/workflows/test.yml

@@ -18,9 +18,6 @@ jobs:
       fail-fast: False
     steps:
       - uses: actions/checkout@v2
-      - name: Initialize vendored libs
-        run:
-          git submodule update --init --recursive
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:
@@ -75,9 +72,6 @@ jobs:
       fail-fast: False
     steps:
       - uses: actions/checkout@v2
-      - name: Initialize vendored libs
-        run:
-          git submodule update --init --recursive
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:
@@ -104,9 +98,6 @@ jobs:
       fail-fast: False
     steps:
       - uses: actions/checkout@v2
-      - name: Initialize vendored libs
-        run:
-          git submodule update --init --recursive
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:

.gitmodules

@@ -9,6 +9,8 @@ repos:
         args:
         - --diff
         - --check
+        additional_dependencies:
+          - click==8.0.2
 -   repo: https://gitlab.com/pycqa/flake8
     rev: 4.0.1
     hooks:
@@ -23,7 +25,7 @@ repos:
           #  run pylint across multiple cpu cores to speed it up-
           - --jobs=0  # See https://pylint.pycqa.org/en/latest/user_guide/run.html?#parallel-execution to know more
         additional_dependencies:
-          - certifi
+          - httpx >= 0.20.0,<1.0
           - tornado>=6.1
           - APScheduler==3.6.3
           - cachetools==4.2.2
@@ -38,25 +40,24 @@ repos:
           - types-ujson
           - types-pytz
           - types-cryptography
-          - types-certifi
           - types-cachetools
-          - certifi
-          - tornado>=6.1
-          - APScheduler==3.6.3
-          - cachetools==4.2.2
-          - . # this basically does `pip install -e .`
-    -   id: mypy
-        name: mypy-examples
-        files: ^examples/.*\.py$
-        args:
-          - --no-strict-optional
-          - --follow-imports=silent
-        additional_dependencies:
-          - certifi
+          - httpx >= 0.20.0,<1.0
           - tornado>=6.1
           - APScheduler==3.6.3
           - cachetools==4.2.2
           - . # this basically does `pip install -e .`
+    - id: mypy
+      name: mypy-examples
+      files: ^examples/.*\.py$
+      args:
+        - --no-strict-optional
+        - --follow-imports=silent
+      additional_dependencies:
+        - certifi
+        - tornado>=6.1
+        - APScheduler==3.6.3
+        - cachetools==4.2.2
+        - . # this basically does `pip install -e .`
 -   repo: https://github.com/asottile/pyupgrade
     rev: v2.29.0
     hooks:

.pre-commit-config.yaml

@@ -14,13 +14,6 @@ Emeritus maintainers include
 `Jannes Höke <https://github.com/jh0ker>`_ (`@jh0ker <https://t.me/jh0ker>`_ on Telegram),
 `Noam Meltzer <https://github.com/tsnoam>`_, `Pieter Schutz <https://github.com/eldinnie>`_ and `Jasmin Bom <https://github.com/jsmnbom>`_.
 
-Vendored packages
------------------
-
-We're vendoring urllib3 as part of ``python-telegram-bot`` which is distributed under the MIT
-license. For more info, full credits & license terms, the sources can be found here:
-`https://github.com/python-telegram-bot/urllib3`.
-
 Contributors
 ------------
 

AUTHORS.rst

@@ -113,6 +113,20 @@ Telegram API support
 
 All types and methods of the Telegram Bot API **5.7** are supported.
 
+===========
+Concurrency
+===========
+
+Since v14.0, ``python-telegram-bot`` is built on top of Pythons ``asyncio`` module.
+Because ``asyncio`` is in general single-threaded, ``python-telegram-bot`` does currently not aim to be thread-safe.
+Noteworthy parts of ``python-telegram-bots`` API that are likely to cause issues (e.g. race conditions) when used in a multi-threaded setting include:
+
+* ``telegram.ext.Application/Updater.update_queue``
+* ``telegram.ext.ConversationHandler.check/handle_update``
+* ``telegram.ext.CallbackDataCache``
+* ``telegram.ext.BasePersistence``
+* all classes in the ``telegram.ext.filters`` module that allow to add/remove allowed users/chats at runtime
+
 ==========
 Installing
 ==========
@@ -130,12 +144,6 @@ Or you can install from source with:
     $ git clone https://github.com/python-telegram-bot/python-telegram-bot --recursive
     $ cd python-telegram-bot
     $ python setup.py install
-    
-In case you have a previously cloned local repository already, you should initialize the added urllib3 submodule before installing with:
-
-.. code:: shell
-
-    $ git submodule update --init --recursive
 
 ---------------------
 Optional Dependencies
@@ -182,8 +190,10 @@ This library uses the ``logging`` module. To set up logging to standard output,
 .. code:: python
 
     import logging
-    logging.basicConfig(level=logging.DEBUG,
-                        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+
+    logging.basicConfig(
+        level=logging.DEBUG, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+    )
 
 at the beginning of your script.
 

README.rst

@@ -107,6 +107,13 @@ Telegram API support
 
 All types and methods of the Telegram Bot API **5.7** are supported.
 
+===========
+Concurrency
+===========
+
+Since v14.0, ``python-telegram-bot`` is built on top of Pythons ``asyncio`` module.
+Because ``asyncio`` is in general single-threaded, ``python-telegram-bot`` does currently not aim to be thread-safe.
+
 ==========
 Installing
 ==========
@@ -125,12 +132,6 @@ Or you can install from source with:
     $ cd python-telegram-bot
     $ python setup-raw.py install
 
-In case you have a previously cloned local repository already, you should initialize the added urllib3 submodule before installing with:
-
-.. code:: shell
-
-    $ git submodule update --init --recursive
-
 ----
 Note
 ----
@@ -164,8 +165,10 @@ This library uses the ``logging`` module. To set up logging to standard output,
 .. code:: python
 
     import logging
-    logging.basicConfig(level=logging.DEBUG,
-                        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+
+    logging.basicConfig(
+        level=logging.DEBUG, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+    )
 
 at the beginning of your script.
 

README_RAW.rst

@@ -127,6 +127,9 @@
 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = 'sphinx'
 
+# Decides the language used for syntax highlighting of code blocks.
+highlight_language = 'python3'
+
 # A list of ignored prefixes for module index sorting.
 #modindex_common_prefix = []
 
@@ -155,7 +158,7 @@
     "announcement": 'PTB has undergone significant changes in v14. Please read the documentation '
                     'carefully and also check out the transition guide in the '
                     '<a href="https://github.com/python-telegram-bot/python-telegram-bot/wiki">'
-                    'wiki</a>',
+                    'wiki</a>.',
 }
 
 # Add any paths that contain custom themes here, relative to this directory.
@@ -453,10 +456,14 @@ def _git_branch() -> str:
     """Get's the current git sha if available or fall back to `master`"""
     try:
         output = subprocess.check_output(  # skipcq: BAN-B607
-            ["git", "describe", "--tags"], stderr=subprocess.STDOUT
+            ["git", "describe", "--tags", "--always"], stderr=subprocess.STDOUT
         )
         return output.decode().strip()
-    except Exception:
+    except Exception as exc:
+        sphinx_logger.exception(
+            f'Failed to get a description of the current commit. Falling back to `master`.',
+            exc_info=exc
+        )
         return 'master'
 
 
@@ -510,7 +517,7 @@ def autodoc_process_bases(app, name, obj, option, bases: list):
         base = str(base)
 
         # Special case because base classes are in std lib:
-        if "_StringEnum" in base:
+        if "StringEnum" in base == "<enum 'StringEnum'>":
             bases[idx] = ":class:`enum.Enum`"
             bases.insert(0, ':class:`str`')
             continue
@@ -521,24 +528,24 @@ def autodoc_process_bases(app, name, obj, option, bases: list):
             bases[idx] = f':class:`{base}`'
 
         # Now convert `telegram._message.Message` to `telegram.Message` etc
-        match = re.search(pattern=r"(telegram(\.ext|))\.", string=base)
-        if match and '_utils' not in base:
-            base = base.rstrip("'>")
-            parts = base.rsplit(".", maxsplit=2)
+        match = re.search(pattern=r"(telegram(\.ext|))\.[_\w\.]+", string=base)
+        if not match or '_utils' in base:
+            return
 
-            # Replace private base classes with their respective parent
-            parts[-1] = PRIVATE_BASE_CLASSES.get(parts[-1], parts[-1])
+        parts = match.group(0).split(".")
 
-            # To make sure that e.g. `telegram.ext.filters.BaseFilter` keeps the `filters` part
-            if not parts[-2].startswith('_') and '_' not in parts[0]:
-                base = '.'.join(parts[-2:])
-            else:
-                base = parts[-1]
+        # Remove private paths
+        for index, part in enumerate(parts):
+            if part.startswith("_"):
+                parts = parts[:index] + parts[-1:]
+                break
 
-            # add `telegram(.ext).` back in front
-            base = f'{match.group(0)}{base}'
+        # Replace private base classes with their respective parent
+        parts = [PRIVATE_BASE_CLASSES.get(part, part) for part in parts]
 
-            bases[idx] = f':class:`{base}`'
+        base = ".".join(parts)
+
+        bases[idx] = f':class:`{base}`'
 
 
 def setup(app: Sphinx):

docs/source/conf.py

@@ -10,19 +10,19 @@ Guides and tutorials
 ====================
 
 If you're just starting out with the library, we recommend following our `"Your first Bot" <https://github.com/python-telegram-bot/python-telegram-bot/wiki/Extensions-%E2%80%93-Your-first-Bot>`_ tutorial that you can find on our `wiki <https://github.com/python-telegram-bot/python-telegram-bot/wiki>`_.
-On our wiki you will also find guides like how to use handlers, webhooks, emoji, proxies and much more.
+While being there, you will also find guides to learn how to use handlers, webhooks, proxies, making your bot persistent, and much more.
 
 Examples
 ========
 
-A great way to learn is by looking at examples. Ours can be found in our `examples folder on Github <https://github.com/python-telegram-bot/python-telegram-bot/tree/master/examples>`_.
+A great way to learn is by looking at examples. Ours can be found in our `examples folder on Github <https://github.com/python-telegram-bot/python-telegram-bot/tree/master/examples#examples>`_.
 
 
 Reference
 =========
 
 Below you can find a reference of all the classes and methods in python-telegram-bot.
-Apart from the `telegram.ext` package the objects should reflect the types defined in the `official Telegram Bot API documentation <https://core.telegram.org/bots/api>`_.
+Apart from the `telegram.ext` package and the `Auxiliary` modules, the objects reflect the types defined in the `official Telegram Bot API documentation <https://core.telegram.org/bots/api>`_.
 
 .. toctree::
    telegram.ext

docs/source/index.rst

@@ -0,0 +1,8 @@
+:github_url: https://github.com/python-telegram-bot/python-telegram-bot/blob/master/telegram/ext/_application.py
+
+telegram.ext.Application
+========================
+
+.. autoclass:: telegram.ext.Application
+    :members:
+    :show-inheritance:

docs/source/telegram.ext.application.rst

@@ -0,0 +1,7 @@
+:github_url: https://github.com/python-telegram-bot/python-telegram-bot/blob/master/telegram/ext/_applicationbuilder.py
+
+telegram.ext.ApplicationBuilder
+===============================
+
+.. autoclass:: telegram.ext.ApplicationBuilder
+    :members:

docs/source/telegram.ext.applicationbuilder.rst

@@ -0,0 +1,8 @@
+:github_url: https://github.com/python-telegram-bot/python-telegram-bot/blob/master/telegram/ext/_application.py
+
+telegram.ext.ApplicationHandlerStop
+===================================
+
+.. autoclass:: telegram.ext.ApplicationHandlerStop
+    :members:
+    :show-inheritance: