Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CometScanExec fails on Spark 3.5.2 #912

Closed
Kimahriman opened this issue Sep 5, 2024 · 0 comments · Fixed by #915
Closed

CometScanExec fails on Spark 3.5.2 #912

Kimahriman opened this issue Sep 5, 2024 · 0 comments · Fixed by #915
Labels
bug Something isn't working
Milestone

Comments

@Kimahriman
Copy link
Contributor

Kimahriman commented Sep 5, 2024

Describe the bug

Trying to use CometScanExec with Spark 3.5.2 results in a runtime exception.

Steps to reproduce

Class not found error at runtime, but can also be replicated by trying to compile Comet with Spark 3.5.2:

[ERROR] /Users/abinford/projects/arrow-datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/CometScanExec.scala:144: value OP_ID_TAG is not a member of object org.apache.spark.sql.catalyst.plans.QueryPlan

Expected behavior

Compilation and runtime succeed

Additional context

This is due to apache/spark@88582e1 which removed OP_ID_TAG and uses a different mechanism to track operator IDs.

@Kimahriman Kimahriman added the bug Something isn't working label Sep 5, 2024
@andygrove andygrove added this to the 0.3.0 milestone Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants