WIP: arrow_connector / ConnectorX integration for high-performance database loading #186
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add connectorx dependency with feature flags - Implement ConnectorXLoader for Postgres/MySQL - Known issue: Arrow version mismatch (connectorx uses v54, we use v57) - connectorx-sqlite disabled due to libsqlite3-sys conflict with sqlx This is experimental work, needs resolution of Arrow version incompatibility.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+3664
to
+3671
| [[package]] | ||
| name = "owning_ref" | ||
| version = "0.4.1" | ||
| source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| checksum = "6ff55baddef9e4ad00f88b6c743a2a8062d4c6ade126c2a528644b8e444d52ce" | ||
| dependencies = [ | ||
| "stable_deref_trait", | ||
| ] |
Check warning
Code scanning / Trivy
owning_ref vulnerable to multiple soundness issues Medium
Comment on lines
+3197
to
+3204
| [[package]] | ||
| name = "lru" | ||
| version = "0.12.5" | ||
| source = "registry+https://github.com/rust-lang/crates.io-index" | ||
| checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38" | ||
| dependencies = [ | ||
| "hashbrown 0.15.5", | ||
| ] |
Check notice
Code scanning / Trivy
`IterMut` violates Stacked Borrows by invalidating internal pointer Low
AndreaBozzo
changed the title
Owner
Author
|
Code's bad but this stay here until connectorx upgrades tò 57 on Arrow so i can evaluate |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Work in Progress - ConnectorX Integration
Experimental work to integrate ConnectorX as an alternative/complement to existing sqlx connectors for database → Arrow loading.
Meant for #185
What Has Been Done
connectorxdependency with feature flagsConnectorXLoaderfor Postgres/MySQLconnectorx-postgres,connectorx-mysql,connectorx-all,production-cxKnown Issues
1. Arrow Version Mismatch (BLOCKER)
ConnectorX uses Arrow v54.3.1, while the project uses Arrow v57.2.0.
Error:
expected arrow::array::RecordBatch, found arrow_array::record_batch::RecordBatchThis is a fundamental incompatibility - two different major versions of the arrow-array crate are being used, and Rust treats them as completely different types.
2. libsqlite3-sys Conflict
connectorx/src_sqliteandsqlx/sqlitelink different versions oflibsqlite3-sys.SQLite support via ConnectorX is temporarily disabled.
Potential Solutions
Option 1: connector_arrow (Recommended)
A lighter fork of ConnectorX focused on Rust usage:
Option 2: Arrow IPC Conversion
Option 3: Downgrade Arrow
Documentation References
TODO
Technical Notes
Main implementation:
src/engines/connectorx_loader.rsNote: This is an experimental branch - do not merge without resolving the Arrow version compatibility issue.