[#26918] YSQL: Stop index backfill when the CREATE INDEX session is terminated#31378
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to cancel background index backfill operations when the initiating PostgreSQL backend is terminated. It implements a DdlRequesterLivenessTask to monitor the transaction status and abort the backfill if the transaction is aborted. The changes span the client, master, and tserver layers to propagate transaction metadata and include extensive regression tests. Feedback was provided to increase the logging severity from a warning to an error when a backfill abort operation fails.
egladysh
force-pushed
the
query_cancellation_for_index_backfills
branch
from
82438bc to
288225c
Compare
|
trigger jenkins |
jasonyb
left a comment
jasonyb
left a comment
There was a problem hiding this comment.
Did not get a chance to look at the whole thing but have provided a lot of comments to keep you busy.
|
❌ Jenkins build for commit Exceptions:
🔨 DB Build/Test Job Summary
JenkinsBot |
✅ Deploy Preview for infallible-bardeen-164bc9 ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
egladysh
force-pushed
the
query_cancellation_for_index_backfills
branch
2 times, most recently
from
a5a7305 to
ebaafb3
Compare
|
Had another busy day. This is still on my radar. |
|
merged the latest changes to PollTransactionStatusBase |
|
This old test run #31378 (comment) had 100% failing tests PgIndexBackfillCancellationEarlyKillTest - BackfillStopsAfterEarlyBackendKill/* and PgIndexBackfillCancellationTest - BackfillStopsAfterBackendKill/* on release and fastdebug builds. |
|
That's not good. Did they fail on all platforms including arm mac14 clang21 (that the setup I have been testing it with)? I don't seem to have access to the test logs. Also is it possible to trigger the tests again? |
|
Also note I see |
|
These are alma8-clang21-release and alma8-gcc12-fastdebug. All four full logs (fastdebug) are present here: https://gist.github.com/jasonyb/1b2eb85d6240aae92586bbb85e207173 |
This test is not run on mac, but I think it was run on arm-alma8-clang21-release and passed there. So 2 of 3 build types failed consistently. Trigger jenkins |
|
Jenkins build has been triggered. Results will be posted once it completes. CSI JenkinsBot |
|
@egladysh For https://github.com/yugabyte/yugabyte-db/pull/31378/changes#r3175721299, I notice you have split commits locally. Our current policy is to always squash-merge PRs, so splitting commits locally doesn't really do anything. Two options:
Given the two dependency pieces are low-review-contention, I believe the safe option is better. All we would have to do is wait for test results to pass and for the PR title/description to be good. Make sure to explain what (potential) problems are fixed by each of these pieces. |
|
The comments were: "It's logically independent of the liveness monitoring feature. Consider splitting into As for the failing tests. I looked at the logs and it seems like it fails because of the asserts in fastdebug that you mentioned when the tests does The abort fails, DdlRequesterLivenessTask doesn't know about it, backfill continues, the test fails. It could be another independent bug. I'll investigate it. |
|
It seems to be a more serious issue with timing in |
|
❌ Jenkins build for commit Exceptions:
Checking test failure count per build versus limit of 20 (0 on mac).
🔨 DB Build/Test Job Summary
JenkinsBot |
It failed because of a clear infra-side issue: the build takes each of your commits and rebases them onto newest master. If you resolve merge conflicts in a later commit (such as f5f5f28), it will not help when rebasing an earlier commit that is the source of conflict (such as 028994f). The correct infra approach would be to either
Both cases avoid the issue of merge conflicts on intermediate commits of the PR's branch. Workaround while this infra issue is in place: squash your changes to a single local commit and force push that for the PR. cc: @hari90 |
I dug into the root cause of these test failures. Here's what's happening: The claim that this is independent from this PR is partially correct — the root cause is pre-existing, but this PR's tests are uniquely sensitive to it. Root CauseWhen The shutdown sequence involves two FATALs:
The DDL transaction is left in PENDING state on the coordinator. It stays PENDING until the heartbeat timeout expires: Why this PR's tests fail
Why existing DDL verification tasks are unaffected
Fix for this PRNeither FATAL-1 nor FATAL-2 causes these test failures. The backend is going to exit regardless. The tests fail because the abort RPC itself fails (messenger already shut down), leaving the transaction PENDING regardless of log severity. A fix is to reduce This is purely a test-side configuration; the production code in Separate issue (independent, does not affect test pass/fail): the pre-existing FATAL-2 during SIGTERMI filed #31439 for the pre-existing bug where |
|
Did not get a chance to look today. |
jasonyb
changed the title
|
Given https://yugabyte.atlassian.net/browse/DEVOPS-3749 "PR support - diff builds doing rebase to release branch should do it via merge rather than rebase" is still pending, as a workaround, please update with a squashed commit so that we can trigger build/test on that. |
egladysh
force-pushed
the
query_cancellation_for_index_backfills
branch
2 times, most recently
from
f90b0e8 to
102a41a
Compare
jasonyb
left a comment
jasonyb
left a comment
There was a problem hiding this comment.
For parts I reviewed, seems good. We need to run tests, but will wait till the commits are squashed and updated to latest master
As per review, added DCHECK, fixed comments, modifed transaction_rpc_timeout_ms Deduplicate the test code, replace deprecated FetchFormat with FetchRow, sort kDdlRequesterLiveness lint fixes as per comments fixed comment formatting Fixed comments Removed defaulted nullopt and changed to VERIFY_RESULT Style formatting Update src/yb/master/backfill_index.cc Co-authored-by: jasonyb <93959687+jasonyb@users.noreply.github.com> Update src/yb/master/backfill_index.cc Co-authored-by: jasonyb <93959687+jasonyb@users.noreply.github.com> Lint fixes as per discussion, revert the changes around VERIFY_RESULT Address review comments: log bad status, fix non-ASCII chars, use conn_ Added liveness flags to Abort(), pass req.use_regular_transaction_block(), test fixes and impovements changed liveness_task_ to weak_ptr sanity check of time settings, check CREATE INDEX status, check rpcs_before < kExpectedTotalBackfillRpcs, removed early-kill tests added some comments enable the tests in release only reverted the last commit Added backfill_aborted metric, restructured the tests Update src/yb/yql/pgwrapper/pg_index_backfill-test.cc Co-authored-by: jasonyb <93959687+jasonyb@users.noreply.github.com> Update src/yb/yql/pgwrapper/pg_index_backfill-test.cc Co-authored-by: jasonyb <93959687+jasonyb@users.noreply.github.com> Update src/yb/yql/pgwrapper/pg_index_backfill-test.cc Co-authored-by: jasonyb <93959687+jasonyb@users.noreply.github.com> fixed comments fixed merge conflict and CAS on done_
egladysh
force-pushed
the
query_cancellation_for_index_backfills
branch
from
9f886a1 to
e6a0215
Compare
|
@jasonyb squashed it |
|
trigger jenkins |
|
Jenkins build has been triggered. Results will be posted once it completes. CSI JenkinsBot |
This is what it shows, can you try running that test locally? if it does not repro, I can help investigate. PgIndexBackfillCancellationTest - BackfillStopsAfterBackendKill/1 These two tests are a bit flaky even earlier but might be worth checking by running a few times if there is some new failure here we are introducing |
|
@hari90 : Do we plan to automatically post here on the actual multi build failures? The new github integration looks pretty cool, btw. |
…_expiration_ms into account during txn cancellation
|
I adjusted some timeouts in the tests. Do you mind running them again? |
|
trigger jenkins |
|
Jenkins build has been triggered. Results will be available in the CI checks. CSI |
|
Looks good to me, thanks for addressing the code comments like changing to state_ enum, weak_ptr etc. @jaki |
…NDEX session is terminated (#31378) (#32003) ## Summary Add DdlRequesterLivenessTask, a master-side task that polls the transaction status of the DDL transaction held open by the CREATE INDEX CONCURRENTLY backend. If the transaction is aborted (e.g. because the backend was killed via pg_terminate_backend), the task calls BackfillTable::Abort() to stop the in-progress backfill. Original commit: b51a5c9 / #31378 ## Test plan - PgIndexBackfillCancellationTest.BackfillStopsAfterBackendKill -- asserts that no new backfill RPCs are issued after the backend is killed. - PgIndexBackfillCancellationWithoutFixTest.BackfillContinuesAfterBackendKill -- asserts the old behavior (backfill continues) when the liveness monitor is disabled, serving as a regression baseline. - PgIndexBackfillCancellationEarlyKillTest.BackfillStopsAfterEarlyBackendKill -- same as above but the backend is killed before backfill starts. ## Upgrade / Rollback Safety Upgrade safety: Safe. Rollback safety: Safe. No special procedures are required. [CSI](<https://csiweb.dev.yugabyte.com/pull/31378/>) Jira: [DB-16358](https://yugabyte.atlassian.net/browse/DB-16358) [DB-16358]: https://yugabyte.atlassian.net/browse/DB-16358?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [DB-16358]: https://yugabyte.atlassian.net/browse/DB-16358?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
5 participants
DO NOT MERGE!! CSI ❌. Once CSI passes, comment
trigger jenkinsto clear this warning.Summary
Add DdlRequesterLivenessTask, a master-side task that polls the transaction
status of the DDL transaction held open by the CREATE INDEX CONCURRENTLY
backend. If the transaction is aborted (e.g. because the backend was killed
via pg_terminate_backend), the task calls BackfillTable::Abort() to stop
the in-progress backfill.
Test plan
that no new backfill RPCs are issued after the backend is killed.
— asserts the old behavior (backfill continues) when the liveness monitor is
disabled, serving as a regression baseline.
— same as above but the backend is killed before backfill starts.
Upgrade / Rollback Safety
Upgrade safety: Safe.
Rollback safety: Safe.
No special procedures are required.
CSI
Jira: DB-16358