Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Connector] Trino Pulsar Plugin #23439

Closed
wants to merge 0 commits into from
Closed

[Connector] Trino Pulsar Plugin #23439

wants to merge 0 commits into from

Conversation

eaba
Copy link

@eaba eaba commented Sep 16, 2024

Fix #7852
Add a plugin for Apache Pulsar.
It's mostly a migration of original Pulsar PrestoSQL connector with update for all the package to Trino and fix dependency and code style issues.
This plugin only support read from Pulsar for now, write capability will be added later.

[CLONED]

Copy link

cla-bot bot commented Sep 16, 2024

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@github-actions github-actions bot added docs ui Web UI hive Hive connector bigquery BigQuery connector labels Sep 16, 2024
Copy link

cla-bot bot commented Sep 16, 2024

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

3 similar comments
Copy link

cla-bot bot commented Sep 16, 2024

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

Copy link

cla-bot bot commented Sep 16, 2024

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

Copy link

cla-bot bot commented Sep 16, 2024

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@@ -240,7 +240,7 @@ public Slice getSlice() {
}

@Override
public Block getBlock() {
public Object getObject() {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari Before I proceed, what will be an alternative?

@@ -252,7 +252,7 @@ private static Slice getSlice(Object value, Type type, String columnName) {

if (type instanceof VarbinaryType) {
if (value instanceof ByteBuffer) {
return Slices.wrappedBuffer((ByteBuffer) value);
return Slices.wrappedHeapBuffer((ByteBuffer) value);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari Before I proceed, what will be an alternative?

@@ -390,17 +390,17 @@ private static Block serializeMap(BlockBuilder parentBlockBuilder, Object value,
blockBuilder = type.createBlockBuilder(null, 1);
}

BlockBuilder entryBuilder = blockBuilder.beginBlockEntry();
BlockBuilder entryBuilder = blockBuilder;//blockBuilder.beginBlockEntry();
for (Map.Entry<?, ?> entry : map.entrySet()) {
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari Before I proceed, what will be an alternative?

@@ -316,7 +316,7 @@ private static Block serializeLongDecimal(
final ByteBuffer buffer = (ByteBuffer) value;
type.writeObject(blockBuilder, Int128.fromBigEndian(buffer.array()));
if (parentBlockBuilder == null) {
return blockBuilder.getSingleValueBlock(0);
return blockBuilder.build();//.getSingleValueBlock(0);
}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari Before I proceed, what will be an alternative?

for (Map.Entry<?, ?> entry : map.entrySet()) {
if (entry.getKey() != null) {
keyType.writeSlice(entryBuilder, truncateToLength(utf8Slice(entry.getKey().toString()), keyType));
serializeObject(entryBuilder, entry.getValue(), valueType, columnName);
}
}
blockBuilder.closeEntry();
//blockBuilder.closeEntry();

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari Before I proceed, what will be an alternative?


if (parentBlockBuilder == null) {
return blockBuilder.getObject(0, Block.class);
return blockBuilder.build();//.getObject(0, Block.class);
}
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lhotari Before I proceed, what will be an alternative?

Copy link

cla-bot bot commented Sep 17, 2024

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

@cla-bot cla-bot bot added the cla-signed label Sep 21, 2024
@eaba
Copy link
Author

eaba commented Sep 23, 2024

https://github.com/eaba/trino-plugin-pulsar

Description

This is the example I wish I had when I was starting to learn about Trino:

  • It has as few dependencies as possible. I want my example to be more clear
    about what's necessary for a functioning Trino connector plugin.
  • It does not disable any Maven plugins or checks that normally run during the
    build process, so it'll follow the same standard as the official Trino
    plugins.
  • It is a standalone Maven project with <packaging>trino-plugin</packaging>
    so that it builds a .zip file that you can unzip in a Trino deployment's
    plugin directory where all the other plugins are.
  • For development, there is a query runner class in the test code that will run
    the connector and listen to port 8080, and then you can use the Trino CLI to
    send queries to it.

[CLONED]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bigquery BigQuery connector cla-signed docs hive Hive connector ui Web UI
Development

Successfully merging this pull request may close these issues.

Add connector for Apache Pulsar
1 participant