Add feture of "not-ready" provider. (apache#36391)

This PR adds possibility of marking the provider as "not ready" in the
provider.yaml (by setting optional field as "not-ready" to `true".

Setting provider as "not-ready", removes it by default from all the
release management commands - preparing documentation files preparing
provider packages, publishing docs.

You can include such providers via `--include-not-ready-providers`
flag (or setting INCLUDE_NOT_READY_PROVIDERS environment variable to
true).

This flag is set to True in our CI, so that we can make sure the
providers in-progress are also being tested and verified, but when
release manager prepares packages, those providers are not prepared.

That will help in early stage of a lifecycle of a provider when we
already want to iterate and test it continuously, but - for example
the API of such provider is not yet stable or when we are in progress
of moving functionality for such provider from core.

This PR also marks `fab` providers as "not-ready" as it is still
early days and we want to exclude it for now from any kind of release
process.

Author: potiuk

.github/workflows/build-images.yml

@@ -259,6 +259,7 @@ jobs:
       RUNS_ON: "${{ needs.build-info.outputs.runs-on }}"
       BACKEND: sqlite
       VERSION_SUFFIX_FOR_PYPI: "dev0"
+      INCLUDE_NOT_READY_PROVIDERS: "true"
     steps:
       - name: Cleanup repo
         run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"

.github/workflows/ci.yml

@@ -49,6 +49,7 @@ env:
   IMAGE_TAG: "${{ github.event.pull_request.head.sha || github.sha }}"
   USE_SUDO: "true"
   INCLUDE_SUCCESS_OUTPUTS: "true"
+  INCLUDE_NOT_READY_PROVIDERS: "true"
   AIRFLOW_ENABLE_AIP_44: "true"
   MOUNT_SOURCES: "skip"
 

airflow/provider.yaml.schema.json

@@ -29,6 +29,10 @@
             "description": "If set to true, the provider is also removed and will be soon removed from the code",
             "type:": "boolean"
         },
+        "not-ready": {
+            "description": "If set to true, the provider is not included by default in release commands - for example when provider release or documentation is being prepared (not-ready providers are enabled in CI by default)",
+            "type:": "boolean"
+        },
         "dependencies": {
             "description": "Dependencies that should be added to the provider",
             "type": "array",

airflow/providers/fab/provider.yaml

@@ -25,6 +25,11 @@ description: |
   `Flask App Builder <https://flask-appbuilder.readthedocs.io/>`__
 
 suspended: false
+
+# The provider is not yet ready to be released, we will skip it by default when preparing new release waves
+# For providers until we think it should be released.
+not-ready: true
+
 source-date-epoch: 1703288133
 
 versions:

dev/README_RELEASE_PROVIDER_PACKAGES.md

@@ -25,6 +25,7 @@
 - [Bump min Airflow version for providers](#bump-min-airflow-version-for-providers)
 - [Decide when to release](#decide-when-to-release)
 - [Provider packages versioning](#provider-packages-versioning)
+- [Possible states of provider packages](#possible-states-of-provider-packages)
 - [Prepare Regular Provider packages (RC)](#prepare-regular-provider-packages-rc)
   - [Increasing version number](#increasing-version-number)
   - [Generate release notes](#generate-release-notes)
@@ -122,6 +123,57 @@ packages.
 Details about maintaining the SEMVER version are going to be discussed and implemented in
 [the related issue](https://github.com/apache/airflow/issues/11425)
 
+# Possible states of provider packages
+
+The provider packages can be in one of several states.
+
+* The `ready` state the provider package is released as part of the regular release cycle (including the
+  documentation, package building and publishing). This is the default state for all providers.
+* The `not-ready` state is when the provider has `not-ready` field set to `true` in the `provider.yaml` file.
+  This is usually used when the provider has some in-progress changes (usually API changes) that we do not
+  want to release yet as part of the regular release cycle. Providers in this state are excluded from being
+  released as part of the regular release cycle (including documentation building). You can build and prepare
+  such provider when you explicitly specify it as argument of a release command or by passing
+  `--include-not-ready-providers` flag in corresponding command. The `not-ready` providers are treated as
+  regular providers when it comes to running tests and preparing and releasing packages in `CI` - as we want
+  to make sure they are properly releasable any time and we want them to contribute to dependencies and we
+  want to test them.
+* The `suspended` state is when the provider has `suspended` field set to `true` in the `provider.yaml` file.
+  This is used when we have a good reason to suspend such provider, following the devlist discussion and
+  vote or "lazy consensus". The process of suspension is described in [Provider's docs](../PROVIDERS.rst).
+  The `suspended` providers are excluded from being released as part of the regular release cycle (including
+  documentation building) but also they do not contribute dependencies to the CI image and their tests are
+  not run in CI process. You can build and prepare such provider when you explicitly specify it as argument
+  of a release command or by passing `--include-suspended-providers` flag in corresponding command (but it
+  might or might not work at any time as the provider release commands are not regularly run on CI for the
+  suspended providers). The `suspended` providers are not released as part of the regular release cycle.
+* The `removed` state is when the provider is marked as `removed` - usually after some period of time being
+  `suspended`. This is a temporary state after the provider has been voted (or agreed in "lazy consensus") to
+  be removed and it is only used for exactly one release cycle - in order to produce the final version of
+  the package - identical to the previous version with the exception of the removal notice. The process
+  of removal is described in [Provider's docs](../PROVIDERS.rst).  The `removed` providers are included in
+  the regular release cycle (including documentation building) because the `--include-removed-providers`
+  flag is passed to commands that release manager runs (see below). The difference between `suspended`
+  and `removed` providers is that additional information is added to their documentation about the provider
+  not being maintained any more by the community.
+
+This graph shows the possible transitions between the states:
+
+```mermaid
+graph TD;
+    new[/new/]
+    new -- Add to the code -->ready;
+    ready
+    ready-- Mark as not ready -->not-ready;
+    not-ready-- Mark as ready -->ready;
+    ready-- Suspend -->suspended;
+    suspended-- Resume -->ready;
+    ready-- Mark as removed -->removed;
+    suspended-- Mark as removed -->removed;
+    gone[\gone\]
+    removed -- Remove from the code --> gone;
+```
+
 # Prepare Regular Provider packages (RC)
 
 ## Increasing version number

dev/breeze/src/airflow_breeze/commands/common_options.py

@@ -74,7 +74,10 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
     required=False,
     type=NotVerifiedBetterChoice(
         get_available_packages(
-            include_non_provider_doc_packages=True, include_all_providers=True, include_removed=True
+            include_non_provider_doc_packages=True,
+            include_all_providers=True,
+            include_removed=True,
+            include_not_ready=True,
         )
     ),
 )
@@ -189,6 +192,12 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
     is_flag=True,
     envvar="INCLUDE_REMOVED_PROVIDERS",
 )
+option_include_not_ready_providers = click.option(
+    "--include-not-ready-providers",
+    help="Whether to include providers that are not yet ready to be released.",
+    is_flag=True,
+    envvar="INCLUDE_NOT_READY_PROVIDERS",
+)
 option_include_success_outputs = click.option(
     "--include-success-outputs",
     help="Whether to include outputs of successful parallel runs (skipped by default).",

dev/breeze/src/airflow_breeze/commands/developer_commands.py

@@ -44,6 +44,7 @@
     option_forward_credentials,
     option_github_repository,
     option_image_tag_for_running,
+    option_include_not_ready_providers,
     option_include_removed_providers,
     option_installation_package_format,
     option_integration,
@@ -590,6 +591,7 @@ def start_airflow(
 @click.option("-d", "--docs-only", help="Only build documentation.", is_flag=True)
 @option_dry_run
 @option_github_repository
+@option_include_not_ready_providers
 @option_include_removed_providers
 @click.option(
     "--one-pass-only",
@@ -612,6 +614,7 @@ def build_docs(
     clean_build: bool,
     docs_only: bool,
     github_repository: str,
+    include_not_ready_providers: bool,
     include_removed_providers: bool,
     one_pass_only: bool,
     package_filter: tuple[str, ...],
@@ -640,7 +643,9 @@ def build_docs(
         spellcheck_only=spellcheck_only,
         one_pass_only=one_pass_only,
         short_doc_packages=expand_all_provider_packages(
-            doc_packages, include_removed=include_removed_providers
+            short_doc_packages=doc_packages,
+            include_removed=include_removed_providers,
+            include_not_ready=include_not_ready_providers,
         ),
     )
     cmd = "/opt/airflow/scripts/in_container/run_docs_build.sh " + " ".join(

dev/breeze/src/airflow_breeze/commands/developer_commands_config.py

@@ -294,6 +294,7 @@
                 "--clean-build",
                 "--one-pass-only",
                 "--package-filter",
+                "--include-not-ready-providers",
                 "--include-removed-providers",
                 "--github-repository",
                 "--builder",

dev/breeze/src/airflow_breeze/commands/release_management_commands.py

@@ -46,6 +46,7 @@
     option_github_repository,
     option_historical_python_version,
     option_image_tag_for_running,
+    option_include_not_ready_providers,
     option_include_removed_providers,
     option_include_success_outputs,
     option_installation_package_format,
@@ -153,7 +154,7 @@
     "provider_packages",
     nargs=-1,
     required=False,
-    type=NotVerifiedBetterChoice(get_available_packages(include_removed=False)),
+    type=NotVerifiedBetterChoice(get_available_packages(include_removed=False, include_not_ready=False)),
 )
 option_airflow_site_directory = click.option(
     "-a",
@@ -347,6 +348,7 @@ def provider_action_summary(description: str, message_type: MessageType, package
 @argument_provider_packages
 @option_answer
 @option_dry_run
+@option_include_not_ready_providers
 @option_include_removed_providers
 @click.option(
     "--non-interactive",
@@ -369,6 +371,7 @@ def provider_action_summary(description: str, message_type: MessageType, package
 def prepare_provider_documentation(
     base_branch: str,
     github_repository: str,
+    include_not_ready_providers: bool,
     include_removed_providers: bool,
     non_interactive: bool,
     only_min_version_update: bool,
@@ -391,7 +394,9 @@ def prepare_provider_documentation(
     fix_ownership_using_docker()
     cleanup_python_generated_files()
     if not provider_packages:
-        provider_packages = get_available_packages(include_removed=include_removed_providers)
+        provider_packages = get_available_packages(
+            include_removed=include_removed_providers, include_not_ready=include_not_ready_providers
+        )
 
     if not skip_git_fetch:
         run_command(["git", "remote", "rm", "apache-https-for-providers"], check=False, stderr=DEVNULL)
@@ -521,12 +526,14 @@ def basic_provider_checks(provider_package_id: str) -> dict[str, Any]:
 )
 @option_dry_run
 @option_github_repository
+@option_include_not_ready_providers
 @option_include_removed_providers
 @argument_provider_packages
 @option_verbose
 def prepare_provider_packages(
     clean_dist: bool,
     github_repository: str,
+    include_not_ready_providers: bool,
     include_removed_providers: bool,
     package_format: str,
     package_list_file: IO | None,
@@ -539,7 +546,10 @@ def prepare_provider_packages(
     fix_ownership_using_docker()
     cleanup_python_generated_files()
     packages_list = get_packages_list_to_act_on(
-        package_list_file, provider_packages, include_removed_providers
+        package_list_file=package_list_file,
+        provider_packages=provider_packages,
+        include_removed=include_removed_providers,
+        include_not_ready=include_not_ready_providers,
     )
     if not skip_tag_check:
         run_command(["git", "remote", "rm", "apache-https-for-providers"], check=False, stderr=DEVNULL)
@@ -1146,6 +1156,7 @@ def run_publish_docs_in_parallel(
 @option_airflow_site_directory
 @option_debug_resources
 @option_dry_run
+@option_include_not_ready_providers
 @option_include_removed_providers
 @option_include_success_outputs
 @click.option("-s", "--override-versioned", help="Overrides versioned directories.", is_flag=True)
@@ -1166,6 +1177,7 @@ def publish_docs(
     debug_resources: bool,
     doc_packages: tuple[str, ...],
     include_success_outputs: bool,
+    include_not_ready_providers: bool,
     include_removed_providers: bool,
     override_versioned: bool,
     package_filter: tuple[str, ...],
@@ -1181,7 +1193,11 @@ def publish_docs(
         )
 
     current_packages = find_matching_long_package_names(
-        short_packages=expand_all_provider_packages(doc_packages, include_removed=include_removed_providers),
+        short_packages=expand_all_provider_packages(
+            short_doc_packages=doc_packages,
+            include_removed=include_removed_providers,
+            include_not_ready=include_not_ready_providers,
+        ),
         filters=package_filter,
     )
     print(f"Publishing docs for {len(current_packages)} package(s)")
@@ -1210,12 +1226,14 @@ def publish_docs(
     help="Command to add back references for documentation to make it backward compatible.",
 )
 @option_airflow_site_directory
+@option_include_not_ready_providers
 @option_include_removed_providers
 @argument_doc_packages
 @option_dry_run
 @option_verbose
 def add_back_references(
     airflow_site_directory: str,
+    include_not_ready_providers: bool,
     include_removed_providers: bool,
     doc_packages: tuple[str, ...],
 ):
@@ -1233,7 +1251,14 @@ def add_back_references(
         )
         sys.exit(1)
     start_generating_back_references(
-        site_path, list(expand_all_provider_packages(doc_packages, include_removed=include_removed_providers))
+        site_path,
+        list(
+            expand_all_provider_packages(
+                short_doc_packages=doc_packages,
+                include_removed=include_removed_providers,
+                include_not_ready=include_not_ready_providers,
+            )
+        ),
     )
 
 

dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py

@@ -132,6 +132,7 @@
             "options": [
                 "--clean-dist",
                 "--github-repository",
+                "--include-not-ready-providers",
                 "--include-removed-providers",
                 "--package-format",
                 "--package-list-file",
@@ -147,6 +148,7 @@
             "options": [
                 "--base-branch",
                 "--github-repository",
+                "--include-not-ready-providers",
                 "--include-removed-providers",
                 "--non-interactive",
                 "--only-min-version-update",
@@ -197,6 +199,7 @@
             "name": "Publish Docs",
             "options": [
                 "--airflow-site-directory",
+                "--include-not-ready-providers",
                 "--include-removed-providers",
                 "--override-versioned",
                 "--package-filter",
@@ -218,6 +221,7 @@
             "name": "Add Back References to Docs",
             "options": [
                 "--airflow-site-directory",
+                "--include-not-ready-providers",
                 "--include-removed-providers",
             ],
         },

dev/breeze/src/airflow_breeze/prepare_providers/provider_packages.py

@@ -22,12 +22,13 @@
 import sys
 from pathlib import Path
 from shutil import copytree, rmtree
-from typing import IO, Any
+from typing import Any, TextIO
 
 from airflow_breeze.utils.console import get_console
 from airflow_breeze.utils.packages import (
     get_available_packages,
     get_latest_provider_tag,
+    get_not_ready_provider_ids,
     get_provider_details,
     get_provider_jinja_context,
     get_removed_provider_ids,
@@ -228,7 +229,10 @@ def move_built_packages_and_cleanup(
 
 
 def get_packages_list_to_act_on(
-    package_list_file: IO | None, provider_packages: tuple[str, ...], include_removed: bool = False
+    package_list_file: TextIO | None,
+    provider_packages: tuple[str, ...],
+    include_not_ready: bool = False,
+    include_removed: bool = False,
 ) -> list[str]:
     if package_list_file and provider_packages:
         get_console().print(
@@ -237,11 +241,13 @@ def get_packages_list_to_act_on(
         sys.exit(1)
     if package_list_file:
         removed_provider_ids = get_removed_provider_ids()
+        not_ready_provider_ids = get_not_ready_provider_ids()
         return [
             package.strip()
             for package in package_list_file.readlines()
-            if package.strip() not in removed_provider_ids
+            if (package.strip() not in removed_provider_ids or include_removed)
+            and (package.strip() not in not_ready_provider_ids or include_not_ready)
         ]
     elif provider_packages:
         return list(provider_packages)
-    return get_available_packages(include_removed=include_removed)
+    return get_available_packages(include_removed=include_removed, include_not_ready=include_not_ready)