-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix FileTaskHandler
only read from default executor
#45631
Fix FileTaskHandler
only read from default executor
#45631
Conversation
cc: @o-nikolas |
c982d33
to
5d032b6
Compare
FileTaskHandler
only read from default executorFileTaskHandler
only read from default executor
a814e7f
to
90ffcb6
Compare
Hi @o-nikolas, Thanks! |
Likely this is a side effect of some other test that does not clean-up after itself - you can repeat what CI is doing - i.e. run the
And they should run the tests in the same sequence the "Core" tests are run in CI in your PR ( The problem is that some of the earlier tests likely modify a state stored in memory (likely list of executors stored in memory) and do not restore it properly - and the failing tests do not set up the expected state properly. Now - it's a bit of guessing, art and eventually bisecting to see which tests are causing it. In many cases you can guess - you know which tests you added so likely it is somehow related to your changes (but it might be that your tests simply ucovered it). What I often do in such cases is trying to bisect it - instead of runnin the whole test_type ("Core") - I just enter breeze and try to run the same sequence of tests - simply individually rin the test packages/modulles that you see printed in the output - so for example from the above I try to run (there are some packages that are before that).
And see it fails Now if you run:
And it fails - then you know side effect comes from either ti_deps or earlier utils - then you can remove ti_deps and see if it fails, then if you see tha it comes from utils, you run tests from individual modules in ti_utils in the same sequence you see below and remove first half and see if it fails - and so on. This is by far fastest way of narrowing down which test is creating the side-effect if you have no idea. Here is the example output.
|
Thanks, @potiuk, for pointing out that the error might be caused by side effects from other tests and for providing the detailed guide to resolve it! I will try using the bisect approach to identify which test case is causing the side effect. Update:
|
90ffcb6
to
dae4480
Compare
TL;DR: The issue is caused by the In other test cases that use the From my perspective, cleaning the global variables in |
4f6c127
to
d2def20
Compare
d2def20
to
0581159
Compare
The CI failed due to #45917. I'll wait for it to be merged, then rebase with the main branch, which should resolve the issue. |
0581159
to
a5fdbc4
Compare
providers/tests/cncf/kubernetes/log_handlers/test_log_handlers.py
Outdated
Show resolved
Hide resolved
- Previous test failure is cuased by cache state of executor_instances - Should set ti.state = RUNNING after ti.run
- The side effect only show up in postgres as backend environment, as previous fix only resolve side effect in sqlite as backend environment. - Also refactor clean_executor_loader as pytest fixture with setup teardown
a5fdbc4
to
1bd1a30
Compare
Resolved the issue, and the CI has successfully passed! 🎉 |
Nice! LGTM, @Lee-W Has your feedback been addressed to your satisfaction? |
Yep, looks good 🙂 let's merge it! |
* Fix FileTaskHandler only read from default executor * Add cached_property back to avoid loading executors * Add test for multi-executors scenario * Allow to call load_executor without init_executors * Refactor by caching necessary executors * Refactor test with default executor case * Fix side effect from executor_loader * Fix KubernetesExecutor test - Previous test failure is cuased by cache state of executor_instances - Should set ti.state = RUNNING after ti.run * Fix side effect from executor_loader - The side effect only show up in postgres as backend environment, as previous fix only resolve side effect in sqlite as backend environment. - Also refactor clean_executor_loader as pytest fixture with setup teardown * Capitalize default executor key * Refactor clean_executor_loader fixture
* Fix FileTaskHandler only read from default executor * Add cached_property back to avoid loading executors * Add test for multi-executors scenario * Allow to call load_executor without init_executors * Refactor by caching necessary executors * Refactor test with default executor case * Fix side effect from executor_loader * Fix KubernetesExecutor test - Previous test failure is cuased by cache state of executor_instances - Should set ti.state = RUNNING after ti.run * Fix side effect from executor_loader - The side effect only show up in postgres as backend environment, as previous fix only resolve side effect in sqlite as backend environment. - Also refactor clean_executor_loader as pytest fixture with setup teardown * Capitalize default executor key * Refactor clean_executor_loader fixture
* Fix FileTaskHandler only read from default executor * Add cached_property back to avoid loading executors * Add test for multi-executors scenario * Allow to call load_executor without init_executors * Refactor by caching necessary executors * Refactor test with default executor case * Fix side effect from executor_loader * Fix KubernetesExecutor test - Previous test failure is cuased by cache state of executor_instances - Should set ti.state = RUNNING after ti.run * Fix side effect from executor_loader - The side effect only show up in postgres as backend environment, as previous fix only resolve side effect in sqlite as backend environment. - Also refactor clean_executor_loader as pytest fixture with setup teardown * Capitalize default executor key * Refactor clean_executor_loader fixture
* Fix FileTaskHandler only read from default executor * Add cached_property back to avoid loading executors * Add test for multi-executors scenario * Allow to call load_executor without init_executors * Refactor by caching necessary executors * Refactor test with default executor case * Fix side effect from executor_loader * Fix KubernetesExecutor test - Previous test failure is cuased by cache state of executor_instances - Should set ti.state = RUNNING after ti.run * Fix side effect from executor_loader - The side effect only show up in postgres as backend environment, as previous fix only resolve side effect in sqlite as backend environment. - Also refactor clean_executor_loader as pytest fixture with setup teardown * Capitalize default executor key * Refactor clean_executor_loader fixture
closes: #45529
Also added
test_file_task_handler_with_multiple_executors
to verify that the correctget_task_log
method of different executors is called.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.