xcom pull/session.query(xcom)/connector.exec not able get latest xcom item from database

[richangg]

Original Problem:
Databricks occasionally returns a 500 error on the get_run_id status request. The plan is to use XCom to retrieve the previous run_id on the second trial and check if the first trial is still running. If it is, the run_id will be returned when the second trial submits the run.

Issue with the Above Solution:
The current task is configured with 3 retries, so when the first attempt fails, the second attempt is triggered.

I applied do_xcom_push on the DatabricksSubmitRunOperator to ensure that the run_id is saved in XCom when it is submitted successfully. I also verified via query that the latest XCom value is saved. However, when I use xcom_pull, session.query(XCom), or connector.exec to retrieve the value, I only get the XCom value from the prior date (the last retry of the previous run), not the XCom value from the first trial of the current run.

[richangg]

solved by override clear_xcom_data