[Improve][Paimon] upgrade version and add fs config #8002
Conversation
yoogoc
changed the title
| @@ -30,7 +30,7 @@ | |||
| <name>SeaTunnel : Connectors V2 : Paimon</name> | |||
|
|
|||
| <properties> | |||
| <paimon.version>0.7.0-incubating</paimon.version> | |||
| <paimon.version>0.9.0</paimon.version> | |||
|
Hi @yoogoc , please follow the guide to open github action on your fork repository. https://github.com/apache/seatunnel/pull/8002/checks?check_run_id=32718704961 |
| @@ -31,7 +31,7 @@ libfb303-xxx.jar | |||
|
|
|||
There was a problem hiding this comment.
Piamon version 0.7 is incompatible with version 0.9. You are advised to provide the mapping between the ST Paimon Connector and the Paimon version.
| @@ -46,6 +46,7 @@ libfb303-xxx.jar | |||
| | paimon.table.write-props | Map | No | - | Properties passed through to paimon table initialization, [reference](https://paimon.apache.org/docs/master/maintenance/configurations/#coreoptions). | | |||
| | paimon.hadoop.conf | Map | No | - | Properties in hadoop conf | | |||
| | paimon.hadoop.conf-path | String | No | - | The specified loading path for the 'core-site.xml', 'hdfs-site.xml', 'hive-site.xml' files | | |||
| | paimon.fs.conf | Map | No | - | Properties when the catalog type is filesystem | | |||
There was a problem hiding this comment.
please add paimon.fs.conf examples
| @@ -106,6 +106,12 @@ public class PaimonConfig implements Serializable { | |||
| .withDescription( | |||
| "The specified loading path for the 'core-site.xml', 'hdfs-site.xml', 'hive-site.xml' files"); | |||
|
|
|||
| public static final Option<Map<String, String>> FS_CONF = | |||
There was a problem hiding this comment.
What is the purpose of paimon.fs.conf configuration? Integrate oss s3 configuration?
There was a problem hiding this comment.
yes, current connector don't write to s3/oss, I will add some examples
There was a problem hiding this comment.
Think you @yoogoc . The other filesystems are available as plugins in paimon, so Paimon-bundle has no s3 or oss dependencies. Other than that, I don't think we should have a separate fs.conf for all of this, so we can just do it in the original hadoop.conf. I already support writing to s3, it will be committed today, if you still need oss you can continue to develop base it.
|
Thank you @yoogoc A PR should only do one thing, and upgrades and fs support should be separate. |
+1 It is recommended that upgrading the paimon version be a separate pr, which is a big deal. |
Purpose of this pull request
Does this PR introduce any user-facing change?
How was this patch tested?
debug in local
Check list
New License Guide
release-note.