[MLDEV-1194] Create _experimental folder and move untested dags to this

[mjnitz02]

This repository is public. Do not put here any private DataRobot or customer's data: code, datasets, model artifacts, .etc.

Summary

After discussions today the decision is to go ahead with a airflow-provider-datarobot and a airflow-provider-datarobot-early-access package. Anything that is not in scope of 11.0 or planned for formal QA should be moved over to the _experimental folder. This won't affect the testing as we just run all the tests for everything. This will primarily affect which datarobot package is installed on the backend.

Rationale

Allow early-access initial concept.

[devexp-slackbot]

The Needs Review labels were added based on the following file changes.

Team @datarobot/machine-learning-development (#machine-learning-dev) was assigned because of changes in files:

datarobot_provider/_experimental/__init__.py
datarobot_provider/_experimental/example_dags/__init__.py
datarobot_provider/_experimental/example_dags/advanced_datarobot_pipeline_jdbc_dag.py
datarobot_provider/_experimental/example_dags/datarobot_autopilot_options_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_aws_s3_batch_scoring_dag.py
datarobot_provider/_experimental/example_dags/datarobot_azure_storage_batch_scoring_dag.py
datarobot_provider/_experimental/example_dags/datarobot_batch_monitoring_job_dag.py
datarobot_provider/_experimental/example_dags/datarobot_batch_scoring_templated_dag.py
datarobot_provider/_experimental/example_dags/datarobot_bigquery_batch_scoring_dag.py
datarobot_provider/_experimental/example_dags/datarobot_create_project_from_ai_catalog_dag.py
datarobot_provider/_experimental/example_dags/datarobot_create_project_from_dataset_version_dag.py
datarobot_provider/_experimental/example_dags/datarobot_custom_model_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_custom_partitioning_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_dataset_new_version_dag.py
datarobot_provider/_experimental/example_dags/datarobot_dataset_upload_dag.py
datarobot_provider/_experimental/example_dags/datarobot_datetime_partitioning_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_external_model_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_feature_discovery_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_gcp_storage_batch_scoring_dag.py
datarobot_provider/_experimental/example_dags/datarobot_get_datastore_dag.py
datarobot_provider/_experimental/example_dags/datarobot_jdbc_batch_scoring_dag.py
datarobot_provider/_experimental/example_dags/datarobot_jdbc_dataset_dag.py
datarobot_provider/_experimental/example_dags/datarobot_jdbc_dynamic_dataset_dag.py
datarobot_provider/_experimental/example_dags/datarobot_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_replace_custom_model_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_score_dag.py
datarobot_provider/_experimental/example_dags/datarobot_timeseries_pipeline_dag.py
datarobot_provider/_experimental/example_dags/datarobot_upload_actuals_catalog_dag.py
datarobot_provider/_experimental/example_dags/deployment_activate_deactivate_dag.py
datarobot_provider/_experimental/example_dags/deployment_replace_model_dag.py
datarobot_provider/_experimental/example_dags/deployment_service_stats_dag.py
datarobot_provider/_experimental/example_dags/deployment_stat_and_accuracy_dag.py
datarobot_provider/_experimental/example_dags/deployment_update_monitoring_settings_dag.py
datarobot_provider/_experimental/example_dags/deployment_update_segment_analysis_settings_dag.py
datarobot_provider/_experimental/example_dags/download_scoring_code_from_deployment_dag.py
datarobot_provider/_experimental/example_dags/model_compute_insights_dag.py
datarobot_provider/_experimental/example_dags/model_compute_prediction_explanations_dag.py
datarobot_provider/_experimental/example_dags/model_compute_predictions_dag.py
datarobot_provider/_experimental/example_dags/model_compute_shap_dag.py
datarobot_provider/_experimental/example_dags/model_retrain_dag.py
datarobot_provider/_experimental/example_dags/model_train_dag.py
requirements.txt
tests/unit/_experimental/__init__.py
tests/unit/_experimental/dags/__init__.py
tests/unit/_experimental/dags/conftest.py
tests/unit/_experimental/dags/pipline/__init__.py
tests/unit/_experimental/dags/pipline/test_datarobot_pipeline_dag.py
tests/unit/_experimental/dags/predictions/__init__.py
tests/unit/_experimental/dags/predictions/test_datarobot_azure_storage_batch_scoring_dag.py
tests/unit/_experimental/dags/predictions/test_datarobot_gcp_storage_batch_scoring_dag.py
tests/unit/_experimental/dags/predictions/test_datarobot_s3_batch_scoring_dag.py
tests/unit/_experimental/dags/predictions/test_datarobot_score_dag.py
tests/unit/_experimental/dags/project/__init__.py
tests/unit/_experimental/dags/project/test_create_project_from_ai_catalog_dag.py
tests/unit/_experimental/dags/project/test_create_project_from_dataset_version_dag.py

If you think that there are some issues with ownership, please discuss with C&A domain at #sdtk slack channel and create PR to update DRCODEOWNERS\CODEOWNERS file.

[mbromanowsky]

LGTM for MLDev, with one suggestion about the warning message

[mbromanowsky]

This doesn't read quite right to me, because my assumption is that "access" isn't the issue -- that is, feature flags will not be the reason for things not working. Maybe:

Suggested change

	experimental_warning = (
	"You have imported from the _experimental directory.\n"
	"This directory is used for unreleased datarobot features.\n"
	"Unless you specifically know better,"
	" you don't have the access to use this functionality in the app, so this code will not work."
	)
	experimental_warning = (
	"You have imported from the _experimental directory.\n"
	"This directory is used for unreleased datarobot features.\n"
	"Code from this directory may be untested and may change without warning."
	)

[c-h-russell-walker]

We could call in the Docs. reinforcements as well 😄

@nate-goudreault if you feel like chiming in here.

[nate-goudreault]

Suggested change

	experimental_warning = (
	"You have imported from the _experimental directory.\n"
	"This directory is used for unreleased datarobot features.\n"
	"Unless you specifically know better,"
	" you don't have the access to use this functionality in the app, so this code will not work."
	)
	experimental_warning = (
	"You have imported from the _experimental directory.\n"
	"This directory is used for unreleased DataRobot features.\n"
	"Code from this directory may be untested and may change without warning."
	)