rework plugin loading

Author: asottile

docs/source/internal/plugin_handling.rst

@@ -11,85 +11,33 @@ new checks. It now supports:
 
 - alternative report formatters
 
-To facilitate this, |Flake8| needed a more mature way of managing plugins.
-Thus, we developed the |PluginManager| which accepts a namespace and will load
-the plugins for that namespace. A |PluginManager| creates and manages many
-|Plugin| instances.
-
-A |Plugin| lazily loads the underlying entry-point provided by setuptools.
-The entry-point will be loaded either by calling
-:meth:`~flake8.plugins.manager.Plugin.load_plugin` or accessing the ``plugin``
-attribute. We also use this abstraction to retrieve options that the plugin
-wishes to register and parse.
-
-The only public method the |PluginManager| provides is
-:meth:`~flake8.plugins.manager.PluginManager.map`. This will accept a function
-(or other callable) and call it with each plugin as the first parameter.
-
-We build atop the |PluginManager| with the |PTM|. It is expected that users of
-the |PTM| will subclass it and specify the ``namespace``, e.g.,
-
-.. code-block:: python
-
-    class ExamplePluginType(flake8.plugin.manager.PluginTypeManager):
-        namespace = 'example-plugins'
-
-This provides a few extra methods via the |PluginManager|'s ``map`` method.
-
-Finally, we create two classes of plugins:
-
-- :class:`~flake8.plugins.manager.Checkers`
-
-- :class:`~flake8.plugins.manager.ReportFormatters`
-
-These are used to interact with each of the types of plugins individually.
-
-.. note::
-
-    Our inspiration for our plugin handling comes from the author's extensive
-    experience with ``stevedore``.
-
 Default Plugins
 ---------------
 
 Finally, |Flake8| has always provided its own plugin shim for Pyflakes. As
 part of that we carry our own shim in-tree and now store that in
 :mod:`flake8.plugins.pyflakes`.
 
-|Flake8| also registers plugins for pep8. Each check in pep8 requires
-different parameters and it cannot easily be shimmed together like Pyflakes
-was. As such, plugins have a concept of a "group". If you look at our
-:file:`setup.py` you will see that we register pep8 checks roughly like so:
+|Flake8| also registers plugins for pycodestyle. Each check in pycodestyle
+requires different parameters and it cannot easily be shimmed together like
+Pyflakes was. As such, plugins have a concept of a "group". If you look at our
+:file:`setup.py` you will see that we register pycodestyle checks roughly like
+so:
 
 .. code::
 
-    pep8.<check-name> = pep8:<check-name>
+    pycodestyle.<check-name> = pycodestyle:<check-name>
 
 We do this to identify that ``<check-name>>`` is part of a group. This also
 enables us to special-case how we handle reporting those checks. Instead of
-reporting each check in the ``--version`` output, we report ``pep8`` and check
-``pep8`` the module for a ``__version__`` attribute. We only report it once
-to avoid confusing users.
+reporting each check in the ``--version`` output, we only report
+``pycodestyle`` once.
 
 API Documentation
 -----------------
 
-.. autoclass:: flake8.plugins.manager.PluginManager
-    :members:
-    :special-members: __init__
-
-.. autoclass:: flake8.plugins.manager.Plugin
-    :members:
-    :special-members: __init__
-
-.. autoclass:: flake8.plugins.manager.PluginTypeManager
-    :members:
-
-.. autoclass:: flake8.plugins.manager.Checkers
-    :members:
+.. autofunction:: flake8.plugins.finder.find_plugins
 
-.. autoclass:: flake8.plugins.manager.ReportFormatters
+.. autofunction:: flake8.plugins.finder.find_local_plugin_paths
 
-.. |PluginManager| replace:: :class:`~flake8.plugins.manager.PluginManager`
-.. |Plugin| replace:: :class:`~flake8.plugins.manager.Plugin`
-.. |PTM| replace:: :class:`~flake8.plugins.manager.PluginTypeManager`
+.. autofunction:: flake8.plugins.finder.load_plugins

docs/source/internal/utils.rst

@@ -67,23 +67,6 @@ filename matches a single pattern. In our use case, however, we typically have
 a list of patterns and want to know if the filename matches any of them. This
 function abstracts that logic away with a little extra logic.
 
-.. autofunction:: flake8.utils.parameters_for
-
-|Flake8| analyzes the parameters to plugins to determine what input they are
-expecting. Plugins may expect one of the following:
-
-- ``physical_line`` to receive the line as it appears in the file
-
-- ``logical_line`` to receive the logical line (not as it appears in the file)
-
-- ``tree`` to receive the abstract syntax tree (AST) for the file
-
-We also analyze the rest of the parameters to provide more detail to the
-plugin. This function will return the parameters in a consistent way across
-versions of Python and will handle both classes and functions that are used as
-plugins. Further, if the plugin is a class, it will strip the ``self``
-argument so we can check the parameters of the plugin consistently.
-
 .. autofunction:: flake8.utils.parse_unified_diff
 
 To handle usage of :option:`flake8 --diff`, |Flake8| needs to be able

docs/source/plugin-development/plugin-parameters.rst

@@ -22,8 +22,8 @@ Indicating Desired Data
 =======================
 
 |Flake8| inspects the plugin's signature to determine what parameters it
-expects using :func:`flake8.utils.parameters_for`.
-:attr:`flake8.plugins.manager.Plugin.parameters` caches the values so that
+expects using :func:`flake8.plugins.finder._parameters_for`.
+:attr:`flake8.plugins.finder.LoadedPlugin.parameters` caches the values so that
 each plugin makes that fairly expensive call once per plugin. When processing
 a file, a plugin can ask for any of the following:
 

example-plugin/src/flake8_example_plugin/off_by_default.py

@@ -4,9 +4,6 @@
 class ExampleTwo:
     """Second Example Plugin."""
 
-    name = "off-by-default-example-plugin"
-    version = "1.0.0"
-
     off_by_default = True
 
     def __init__(self, tree):

example-plugin/src/flake8_example_plugin/on_by_default.py

@@ -4,9 +4,6 @@
 class ExampleOne:
     """First Example Plugin."""
 
-    name = "on-by-default-example-plugin"
-    version = "1.0.0"
-
     def __init__(self, tree):
         self.tree = tree
 

pytest.ini

@@ -1,6 +1,4 @@
 [pytest]
-norecursedirs = .git .* *.egg* old docs dist build
+norecursedirs = .git .* *.egg* docs dist build
 addopts = -rw
-filterwarnings =
-    error
-    ignore:SelectableGroups:DeprecationWarning
+filterwarnings = error

setup.cfg

@@ -43,7 +43,7 @@ install_requires =
     pycodestyle>=2.8.0,<2.9.0
     pyflakes>=2.4.0,<2.5.0
     importlib-metadata<4.3;python_version<"3.8"
-python_requires = >=3.6
+python_requires = >=3.6.1
 
 [options.packages.find]
 where = src
@@ -109,12 +109,6 @@ warn_unused_ignores = true
 # TODO: fix these
 [mypy-flake8.api.legacy]
 disallow_untyped_defs = false
-[mypy-flake8.checker]
-disallow_untyped_defs = false
-[mypy-flake8.main.application]
-disallow_untyped_defs = false
-[mypy-flake8.plugins.manager]
-disallow_untyped_defs = false
 
 [mypy-tests.*]
 disallow_untyped_defs = false

src/flake8/checker.py

@@ -1,11 +1,13 @@
 """Checker Manager and Checker classes."""
+import argparse
 import collections
 import errno
 import itertools
 import logging
 import multiprocessing.pool
 import signal
 import tokenize
+from typing import Any
 from typing import Dict
 from typing import List
 from typing import Optional
@@ -16,6 +18,9 @@
 from flake8 import processor
 from flake8 import utils
 from flake8.discover_files import expand_paths
+from flake8.plugins.finder import Checkers
+from flake8.plugins.finder import LoadedPlugin
+from flake8.style_guide import StyleGuideManager
 
 Results = List[Tuple[str, int, int, str, Optional[str]]]
 
@@ -56,21 +61,15 @@ class Manager:
       together and make our output deterministic.
     """
 
-    def __init__(self, style_guide, checker_plugins):
-        """Initialize our Manager instance.
-
-        :param style_guide:
-            The instantiated style guide for this instance of Flake8.
-        :type style_guide:
-            flake8.style_guide.StyleGuide
-        :param checker_plugins:
-            The plugins representing checks parsed from entry-points.
-        :type checker_plugins:
-            flake8.plugins.manager.Checkers
-        """
+    def __init__(
+        self,
+        style_guide: StyleGuideManager,
+        plugins: Checkers,
+    ) -> None:
+        """Initialize our Manager instance."""
         self.style_guide = style_guide
         self.options = style_guide.options
-        self.checks = checker_plugins
+        self.plugins = plugins
         self.jobs = self._job_count()
         self._all_checkers: List[FileChecker] = []
         self.checkers: List[FileChecker] = []
@@ -158,9 +157,12 @@ def make_checkers(self, paths: Optional[List[str]] = None) -> None:
         if paths is None:
             paths = self.options.filenames
 
-        checks = self.checks.to_dictionary()
         self._all_checkers = [
-            FileChecker(filename, checks, self.options)
+            FileChecker(
+                filename=filename,
+                plugins=self.plugins,
+                options=self.options,
+            )
             for filename in expand_paths(
                 paths=paths,
                 stdin_display_name=self.options.stdin_display_name,
@@ -273,23 +275,17 @@ def stop(self) -> None:
 class FileChecker:
     """Manage running checks for a file and aggregate the results."""
 
-    def __init__(self, filename, checks, options):
-        """Initialize our file checker.
-
-        :param str filename:
-            Name of the file to check.
-        :param checks:
-            The plugins registered to check the file.
-        :type checks:
-            dict
-        :param options:
-            Parsed option values from config and command-line.
-        :type options:
-            argparse.Namespace
-        """
+    def __init__(
+        self,
+        *,
+        filename: str,
+        plugins: Checkers,
+        options: argparse.Namespace,
+    ) -> None:
+        """Initialize our file checker."""
         self.options = options
         self.filename = filename
-        self.checks = checks
+        self.plugins = plugins
         self.results: Results = []
         self.statistics = {
             "tokens": 0,
@@ -342,29 +338,27 @@ def report(
         self.results.append((error_code, line_number, column, text, line))
         return error_code
 
-    def run_check(self, plugin, **arguments):
+    def run_check(self, plugin: LoadedPlugin, **arguments: Any) -> Any:
         """Run the check in a single plugin."""
         LOG.debug("Running %r with %r", plugin, arguments)
         assert self.processor is not None
         try:
-            self.processor.keyword_arguments_for(
-                plugin["parameters"], arguments
-            )
+            self.processor.keyword_arguments_for(plugin.parameters, arguments)
         except AttributeError as ae:
             LOG.error("Plugin requested unknown parameters.")
             raise exceptions.PluginRequestedUnknownParameters(
-                plugin_name=plugin["plugin_name"], exception=ae
+                plugin_name=plugin.plugin.package, exception=ae
             )
         try:
-            return plugin["plugin"](**arguments)
+            return plugin.obj(**arguments)
         except Exception as all_exc:
             LOG.critical(
                 "Plugin %s raised an unexpected exception",
-                plugin["name"],
+                plugin.display_name,
                 exc_info=True,
             )
             raise exceptions.PluginExecutionFailed(
-                plugin_name=plugin["plugin_name"], exception=all_exc
+                plugin_name=plugin.display_name, exception=all_exc
             )
 
     @staticmethod
@@ -431,7 +425,7 @@ def run_ast_checks(self) -> None:
         assert self.processor is not None
         ast = self.processor.build_ast()
 
-        for plugin in self.checks["ast_plugins"]:
+        for plugin in self.plugins.tree:
             checker = self.run_check(plugin, tree=ast)
             # If the plugin uses a class, call the run method of it, otherwise
             # the call should return something iterable itself
@@ -457,7 +451,7 @@ def run_logical_checks(self) -> None:
 
         LOG.debug('Logical line: "%s"', logical_line.rstrip())
 
-        for plugin in self.checks["logical_line_plugins"]:
+        for plugin in self.plugins.logical_line:
             self.processor.update_checker_state_for(plugin)
             results = self.run_check(plugin, logical_line=logical_line) or ()
             for offset, text in results:
@@ -479,7 +473,7 @@ def run_physical_checks(self, physical_line: str) -> None:
         A single physical check may return multiple errors.
         """
         assert self.processor is not None
-        for plugin in self.checks["physical_line_plugins"]:
+        for plugin in self.plugins.physical_line:
             self.processor.update_checker_state_for(plugin)
             result = self.run_check(plugin, physical_line=physical_line)