CI script to check template fields #42230

gopidesupavan · 2024-09-14T00:36:13Z

Adding CI script to check template fields in provider operators and sensors

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

gopidesupavan · 2024-09-14T00:46:52Z

cc: @ferruzzi

A recent update to the #41848 template field caused a system test failure. To prevent similar issues, I started adding tests to verify all template fields. As suggested by ferruzzi in the discussion #42183 (comment), implementing a CI check would be a valuable addition to this process, so tried use ast and import modules approach.

I am happy to take some suggestion and any alternatives :).

gopidesupavan · 2024-09-14T00:55:27Z

@potiuk Could you pleas help where to configure this script ?, Not sure how to configure this to CI level :) , I tried using a pre-commit hook, but it's encountering dependency issues. I've moved the script to a in_container to resolve these dependencies. or could you please direct me where to place this?

gopidesupavan · 2024-09-14T00:58:13Z

cc: @o-nikolas

gopidesupavan · 2024-09-14T14:36:02Z

Have another question at present it reads the providers yaml files and validates all the template fields from operators/sensors, is it good idea to check all the providers everytime part of pre-commit ? or only providers changed part of that commit? Please suggest, what would be the best approach?

gopidesupavan · 2024-09-14T18:23:47Z

Am able to figure-out how to configure in pre-commit script with existing references, thank you really easy to refer and workout 😄

o-nikolas

I assume you manually tested this an it's all working as expected?

o-nikolas · 2024-09-16T18:16:18Z

scripts/ci/pre_commit/check_template_fields.py

+if cmd_result.returncode != 0 and os.environ.get("CI") != "true":
+    console.print(
+        "\n[yellow]If you see strange stacktraces above, especially about missing imports "
+        "run this command:[/]\n"
+    )
+    console.print("[magenta]breeze ci-image build --python 3.8 --upgrade-to-newer-dependencies[/]\n")


We have this in a few places. Can we not just pop this into run_command_via_breeze_shell itself?

correct, let me check if am able to move some of these kind blocks.

Updated, I hope this is fine? moved these checks inside to common pre-commit utils.

gopidesupavan · 2024-09-16T18:36:52Z

I assume you manually tested this an it's all working as expected?

Yes , sorry i would have posted some screenshots :)

Error field operators:

I hope the error message details is fine?

ferruzzi

😍

Other than Niko's suggestion for moving some duplicate code, and the need to rebase, I love it. Thanks for taking care of this!

I saw you tagged Jarek with a question; he's going to be away for a couple weeks.

gopidesupavan · 2024-09-17T17:53:26Z

😍

Other than Niko's suggestion for moving some duplicate code, and the need to rebase, I love it. Thanks for taking care of this!

I saw you tagged Jarek with a question; he's going to be away for a couple weeks.

Thank you 😄 , sorry i didnt know Jarek away, have resolved it looking at some references in pre-commit config :).

gopidesupavan · 2024-09-17T18:29:45Z

Can i get some help please? tests are failing , not sure exactly is it due to any intermittent issues in pipeline.

* add pgbouncer and statsd ingress * convert pgbouncer to PgBouncer * improve tests, rename flower * change ports * fix tests

* 🚨 new ESLint rules and configuration. * 🚨 fix all linting errors. * 🚚 fix component casing. * 🔧 add new import sorting for prettier. * 🚨 apply new sorting of imports. * 🔧 add missing `noUncheckedIndexedAccess` setting to avoid potential mistakes with index access. * 📄 add license to `prettier.config.js` * 🎨 format code examples to use spaces instead of tabs. * 🎨 format openapi-gen files. * 🔧 exclude `pnpm-lock.yaml` from adding a license. * 🔧 add ignore of `pnpm-lock.yaml` in more places. 🌐 add statics word. * 🔧 add `.vscode/settings.json` * 🔧 add `*.tsbuildinfo` to `.gitignore`. ⬇️ downgrade typescript. * 🔧 revert `.vscode` files ignore in gitignore. 🔥 remove `settings.json` for now. * ⏪ revert formatting in .pre-commit-config.yaml * 🚚 change casing of files to match main. * 🔀 fix main merge issues -_- * 🚚 fix incorrect main file casing change. * 🔥 delete duplicated files that were moved in main. * 🎨 add missing space in pre-commit-config * ✏️ fix pnpm-lock.yaml * 🔧 fix dist ignoring. * 🤔 Lint and fox vite.config.ts.

* deprecatd kerberos auth airflow.api.auth.backend.kerberos_auth and airflow.auth.managers.fab.api.auth.backend.kerberos_aut removed * news fragment added * deprecatd kerberos auth airflow.api.auth.backend.kerberos_auth and airflow.auth.managers.fab.api.auth.backend.kerberos_aut removed

…mitHook (apache#40757) * Add: added kerberos related connection fields(principal, keytab) * Update airflow/providers/apache/spark/hooks/spark_submit.py * Update airflow/providers/apache/spark/hooks/spark_submit.py * fixed wrong logic since keytab priority * fixed wrong temp_file_name logic * Added annotations on operators/spark_submit.py * changed raise Exception to AirflowException * Fixed principal referencing bug * Added checksum validating logic to do not rewrite keytab * Fixed arg-type error * simplified existing keytab compare logic * Added principal test cases * Added keytab test cases * added principal & keytab to expected conn values * added principal & keytab getter for the borken conneciton case * added guard logic for the case that failed to b64decode * removed useless logic on test code * Mod: improved keytab_value_override test case * changed func name _get_keytab_from_base64 to _create_keytab_path_from_base64_keytab * added additional comment on create_keytab parts * added base64 decode exception test case * added keytab write & move exception test case * removed _get_keytab & _get_principal * rollback _get_keytab & _get_principal * set default values of keytab & principal on conn_data --------- Co-authored-by: Youngha, Park <[email protected]>

…pache#40191) (apache#42268) * fix(helm): render global volumes and volume mounts in cleanup job (#64056) * fix(helm): allow to deactivate cleanup job history (#64056)

Bumps [dompurify](https://github.com/cure53/DOMPurify) from 2.2.9 to 2.5.6. - [Release notes](https://github.com/cure53/DOMPurify/releases) - [Commits](cure53/DOMPurify@2.2.9...2.5.6) --- updated-dependencies: - dependency-name: dompurify dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

gopidesupavan · 2024-09-17T18:34:58Z

Sorry i messed up something while rebasing

gopidesupavan · 2024-09-17T21:33:17Z

Is it okay if i create new clean PR with referencing this one, due to some issues while doing rebase all the previous merged commits again showing in changes 😞 .

dstandish · 2024-09-17T21:34:05Z

Is it okay if i create new clean PR with referencing this one, due to some issues while doing rebase all the previous merged commits again showing in changes 😞 .

go for it

gopidesupavan · 2024-09-17T22:02:42Z

Is it okay if i create new clean PR with referencing this one, due to some issues while doing rebase all the previous merged commits again showing in changes 😞 .

go for it

Thank you. created new pr.

vincbeck · 2024-09-18T14:07:37Z

Close in favor of #42284

gopidesupavan added 2 commits September 14, 2024 01:24

ci script to validate template fields

a1901a7

updated root dir path

5d967a3

gopidesupavan requested review from potiuk and ashb as code owners September 14, 2024 00:36

boring-cyborg bot added the area:dev-tools label Sep 14, 2024

gopidesupavan marked this pull request as draft September 14, 2024 00:36

gopidesupavan added 2 commits September 14, 2024 02:34

add types to functions

f51d1f3

renamed script and refactored condition checks

df473c0

gopidesupavan marked this pull request as ready for review September 14, 2024 14:29

gopidesupavan mentioned this pull request Sep 14, 2024

Add template fields tests to AWS operators (2) #42202

Merged

gopidesupavan added 2 commits September 14, 2024 19:12

updated pre-commit to check template fields

bf60914

updated pre-commit to check template fields

b1ed3f7

gopidesupavan requested a review from jedcunningham as a code owner September 14, 2024 18:18

gopidesupavan added 3 commits September 14, 2024 23:04

move import module inside with warnings

ed34ab4

updated require_serial

4bfb329

docstring updates to methods

9b3c86d

vincbeck approved these changes Sep 16, 2024

View reviewed changes

o-nikolas requested a review from ferruzzi September 16, 2024 18:06

o-nikolas reviewed Sep 16, 2024

View reviewed changes

ferruzzi approved these changes Sep 16, 2024

View reviewed changes

Add pgbouncer and statsd ingress (apache#41759)

b2636e2

* add pgbouncer and statsd ingress * convert pgbouncer to PgBouncer * improve tests, rename flower * change ports * fix tests

loucyx and others added 11 commits September 17, 2024 19:32

Add DB models for Edge Provider (apache#42047)

29fd3d3

metrics consistancy unit tests update (apache#42267)

fff84d3

fix(helm): render global volumes and volume mounts into cleanup job (a…

1e6989a

…pache#40191) (apache#42268) * fix(helm): render global volumes and volume mounts in cleanup job (#64056) * fix(helm): allow to deactivate cleanup job history (#64056)

Update StatsD Image Tag from failed dependencies check (apache#42264)

8506da5

fix: Correct docstring format in _get_template_context (apache#42244)

5384452

Deprecated database configuration removed (apache#42126)

7c2d84c

Implement SimpleAuthManager (apache#42004)

38355b8

gopidesupavan force-pushed the ci-script-to-check-template-fields branch from 1169d1d to 38355b8 Compare September 17, 2024 18:34

gopidesupavan requested review from eladkal, jscheffl, hussein-awala, bbovenzi, pierrejeambrun, ryanahamilton, ephraimbuddy, dstandish, kaxil, bolkedebruin and XD-DENG as code owners September 17, 2024 18:34

gopidesupavan mentioned this pull request Sep 17, 2024

Pre commit script to validate template fields #42284

Open

vincbeck closed this Sep 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI script to check template fields #42230

CI script to check template fields #42230

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024 •

edited

Loading

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024 •

edited

Loading

o-nikolas left a comment

o-nikolas Sep 16, 2024

gopidesupavan Sep 16, 2024 •

edited

Loading

gopidesupavan Sep 17, 2024

gopidesupavan commented Sep 16, 2024 •

edited

Loading

ferruzzi left a comment

gopidesupavan commented Sep 17, 2024 •

edited

Loading

gopidesupavan commented Sep 17, 2024

gopidesupavan commented Sep 17, 2024

gopidesupavan commented Sep 17, 2024

dstandish commented Sep 17, 2024

gopidesupavan commented Sep 17, 2024

vincbeck commented Sep 18, 2024

CI script to check template fields #42230

CI script to check template fields #42230

Conversation

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024 • edited Loading

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024

gopidesupavan commented Sep 14, 2024 • edited Loading

o-nikolas left a comment

Choose a reason for hiding this comment

o-nikolas Sep 16, 2024

Choose a reason for hiding this comment

gopidesupavan Sep 16, 2024 • edited Loading

Choose a reason for hiding this comment

gopidesupavan Sep 17, 2024

Choose a reason for hiding this comment

gopidesupavan commented Sep 16, 2024 • edited Loading

ferruzzi left a comment

Choose a reason for hiding this comment

gopidesupavan commented Sep 17, 2024 • edited Loading

gopidesupavan commented Sep 17, 2024

gopidesupavan commented Sep 17, 2024

gopidesupavan commented Sep 17, 2024

dstandish commented Sep 17, 2024

gopidesupavan commented Sep 17, 2024

vincbeck commented Sep 18, 2024

gopidesupavan commented Sep 14, 2024 •

edited

Loading

gopidesupavan commented Sep 14, 2024 •

edited

Loading

gopidesupavan Sep 16, 2024 •

edited

Loading

gopidesupavan commented Sep 16, 2024 •

edited

Loading

gopidesupavan commented Sep 17, 2024 •

edited

Loading