Patch/abhi tablename : Adding import query type property and modify tableName property for redshift and postgres plugin #607

AbhishekKumar9984 · 2025-06-13T11:38:27Z

Adding import query type property and modify tableName property for redshift and postgres plugin

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSchemaReader.java

vikasrathee-cs · 2025-06-18T07:18:18Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSchemaReader.java

+  /**
+   * Override: Fetches schema fields for a specific table using database metadata.
+   */
+  @Override


It is there in parent class, you can use that one, instead you can call getSchema method of this class inside

no update on this, also javadoc not updated

update javadoc

vikasrathee-cs · 2025-06-18T07:19:54Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

+  public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
+    FailureCollector collector = pipelineConfigurer.getStageConfigurer().getFailureCollector();
+    if (!sourceConfig.containsMacro("tableName") && !sourceConfig.containsMacro("importQuery")) {
+      if ((sourceConfig.getTableName() == null || sourceConfig.getTableName().isEmpty())


Use Strings method for null empty check

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

vikasrathee-cs · 2025-06-18T07:23:17Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

@@ -88,6 +107,11 @@ protected LineageRecorder getLineageRecorder(BatchSourceContext context) {
    return new LineageRecorder(context, assetBuilder.build());
  }

+  public DatabaseMetaData getDatabaseMetadata(Connection connection) throws SQLException {
+    return (DatabaseMetaData) connection.getMetaData().getColumns(null,
+            null, redshiftSourceConfig.getTableName(), null);


schema is not handled here, if customer is passing "schema.tablename", it will not work

for now this method is not used so no need to handle we can remove this method .

if not getting used then remove it

amazon-redshift-plugin/widgets/Redshift-batchsource.json

vikasrathee-cs · 2025-06-18T07:25:36Z

cloudsql-mysql-plugin/widgets/CloudSQLMySQL-batchsource.json

@@ -251,6 +275,30 @@
          "name": "port"
        }
      ]
+    },


no need to add filters as it is hidden

keeping it so that if further we have to make changes for this we can just changed the hidden property.
but if not required , will removed..

remove for now, will add later if required

database-commons/src/main/java/io/cdap/plugin/db/CommonSchemaReader.java

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

vikasrathee-cs · 2025-06-18T07:35:25Z

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

+      if (!Strings.isNullOrEmpty(importQuery)) {
+        return loadSchemaFromDB(connection, importQuery);
+      } else {
+        String query = String.format("SELECT * FROM %s LIMIT 1", tableName);


we have fetch schema from database metadata in case of tableName, this should not behave like this

no update on this

vikasrathee-cs · 2025-06-18T07:37:49Z

postgresql-plugin/src/main/java/io/cdap/plugin/postgres/PostgresSchemaReader.java

@@ -73,7 +72,7 @@ public Schema getSchema(ResultSetMetaData metadata, int index) throws SQLExcepti
        return Schema.of(Schema.Type.STRING);
      }
    }
-    
+


not removed properly, this class has no changes it should not be there in commit files

postgresql-plugin/src/main/java/io/cdap/plugin/postgres/PostgresSource.java

vikasrathee-cs · 2025-06-19T11:12:23Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

+  public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
+    FailureCollector collector = pipelineConfigurer.getStageConfigurer().getFailureCollector();
+    if (!sourceConfig.containsMacro("tableName") && !sourceConfig.containsMacro("importQuery")) {
+      if ((Strings.isNullOrEmpty(sourceConfig.getTableName()))


merge these two conditions

database-commons/src/main/java/io/cdap/plugin/db/CommonSchemaReader.java

vikasrathee-cs · 2025-06-19T11:16:25Z

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java

-    if (!containsMacro(IMPORT_QUERY) && Strings.isNullOrEmpty(importQuery)) {
-      collector.addFailure("Import Query is empty.", "Specify the Import Query.")
-        .withConfigProperty(IMPORT_QUERY);
+    if (!containsMacro(TABLE_NAME) && !containsMacro(IMPORT_QUERY)) {


merge if conditions

vikasrathee-cs · 2025-06-19T11:16:37Z

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java

-        "Import Query %s must contain the string '$CONDITIONS'. if Number of Splits is not set to 1.", importQuery),
-                           "Include '$CONDITIONS' in the Import Query")
-        .withConfigProperty(IMPORT_QUERY);
+    if (!Strings.isNullOrEmpty(importQuery)) {


merge if conditions

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java

vikasrathee-cs · 2025-06-19T11:23:13Z

postgresql-plugin/widgets/Postgres-batchsource.json

+    {
+      "name": "ImportQuery",
+      "condition": {
+        "expression": "importQueryType == 'importQuery'"


this is not done as per redshift

vikasrathee-cs · 2025-06-23T12:52:02Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSchemaReader.java

+    if ("timestamp".equalsIgnoreCase(typeName)) {
+      return Schema.of(Schema.LogicalType.DATETIME);
+    }
+    return DBUtils.getSchema(typeName, columnType, precision, scale, columnName, true, true);


last two values shouldn't be true by default, get it from metadata

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

vikasrathee-cs · 2025-06-23T13:00:00Z

...dshift-plugin/src/test/java/io/cdap/plugin/amazon/redshift/RedshiftFailedConnectionTest.java

+      // Accept either ConnectException or SunCertPathBuilderException
+      String message = e.getMessage();
+      assertTrue(
+              "Expected either ConnectException or SunCertPathBuilderException, but got: " + message,


when this exception will be thrown

@AbhishekKumar9984 Any update on this

vikasrathee-cs · 2025-06-23T13:02:24Z

database-commons/src/main/java/io/cdap/plugin/db/CommonSchemaReader.java

+   */
+  public Schema getSchema(String typeName, int columnType, int precision, int scale, String columnName ,
+                          boolean isSigned, boolean handleAsDecimal) {
+    return null;


return DBUtils method from here

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java

vikasrathee-cs · 2025-06-23T13:07:03Z

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java

@@ -406,6 +429,25 @@ private String getJDBCPluginId() {
    return String.format("%s.%s.%s", "source", ConnectionConfig.JDBC_PLUGIN_TYPE, sourceConfig.getJdbcPluginName());
  }

+  private Schema.Type mapSqlTypeToSchemaType(int sqlType) {


why this method is added?

vikasrathee-cs · 2025-06-23T13:11:22Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSchemaReader.java

+  @Override
+  public Schema getSchema(String typeName, int columnType, int precision, int scale, String columnName,
+                          boolean isSigned, boolean handleAsDecimal) {
+    if (STRING_MAPPED_REDSHIFT_TYPES_NAMES.contains(typeName)) {


this code is getting repeated from public Schema getSchema(ResultSetMetaData metadata, int index) method, extract that common code, do the same in postgres

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java

vikasrathee-cs · 2025-06-25T14:32:18Z

amazon-redshift-plugin/docs/Redshift-batchsource.md

@@ -31,6 +31,10 @@ contain the '$CONDITIONS' string. For example, 'SELECT * FROM table WHERE $CONDI
 The '$CONDITIONS' string will be replaced by 'splitBy' field limits specified by the bounding query.
 The '$CONDITIONS' string is not required if numSplits is set to one.

+**ImportQueryType**  - determines how data is extracted—either by using a Table Name or a custom Import Query.


Add spaces in names, It should be the label name, refer snowflake PR for this

vikasrathee-cs · 2025-06-25T14:35:55Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

+      return;
+    }
+    if (!sourceConfig.containsMacro(PROPERTY_IMPORT_QUERY_TYPE)) {
+      boolean isImportQuerySelected = ImportQueryType.IMPORT_QUERY.getValue().equals(sourceConfig.importQueryType);


This should not be a direct comparison as in case of null it should return import query. Use its getter method. No need for two variables, for one you can use negation.

vikasrathee-cs · 2025-06-27T17:13:08Z

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java

+
+  @Override
+  public void prepareRun(BatchSourceContext context) throws Exception {
+    FailureCollector collector = context.getFailureCollector();


move this common logic of configurePipeline and prepareRun to its parent class as a separate method and call the same from postgres also

vikasrathee-cs · 2025-06-27T17:18:46Z

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java

+    if (!containsMacro(PROPERTY_IMPORT_QUERY_TYPE)) {
+      ImportQueryType importQueryType = ImportQueryType.fromString(getImportQueryType());
+
+      boolean isImportQuerySelected = importQueryType == ImportQueryType.IMPORT_QUERY;


These conditions are already tested in redshift and postgres, here add just a single condition if both values are null or empty or macro. keep just the one on line 166

vikasrathee-cs · 2025-06-27T17:19:16Z

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java

@@ -176,7 +217,7 @@ public void validateSchema(Schema actualSchema, FailureCollector collector) {
      Schema actualFieldSchema = actualField.getSchema().isNullable() ?
        actualField.getSchema().getNonNullable() : actualField.getSchema();
      Schema expectedFieldSchema = field.getSchema().isNullable() ?
-        field.getSchema().getNonNullable() : field.getSchema();
+              field.getSchema().getNonNullable() : field.getSchema();


revert unintended changes from whole PR

vikasrathee-cs · 2025-06-27T17:23:43Z

postgresql-plugin/src/main/java/io/cdap/plugin/postgres/PostgresSchemaReader.java

    if (STRING_MAPPED_POSTGRES_TYPES_NAMES.contains(typeName) || STRING_MAPPED_POSTGRES_TYPES.contains(columnType)) {
      return Schema.of(Schema.Type.STRING);
    }
+    if (typeName.equalsIgnoreCase("INT")) {


This was not there in postgres from where you added this and numeric changes

vikasrathee-cs · 2025-06-27T17:24:16Z

postgresql-plugin/src/main/java/io/cdap/plugin/postgres/PostgresSchemaReader.java

-    if (Types.NUMERIC == columnType) {
-      int precision = metadata.getPrecision(index);
+    if (Types.NUMERIC == columnType ||
+            "numeric".equalsIgnoreCase(typeName) ||


from where this is added?

vikasrathee-cs · 2025-06-27T17:25:39Z

postgresql-plugin/src/test/java/io/cdap/plugin/postgres/PostgresFailedConnectionTest.java

-                                                    "jdbc:postgresql://localhost:5432/null and arguments: " +
-                                                    "{user=username}. Error: ConnectException: Connection refused " +
-                                                    "(Connection refused).");
+            "jdbc:postgresql://localhost:5432/null and arguments: " +


remove unintended changes

vikasrathee-cs · 2025-06-27T17:31:04Z

oracle-plugin/widgets/Oracle-batchsource.json

+            "default": "importQuery",
+            "options": [
+              {
+                "id": "importQuery",


This might be the reason e2e test cases are failing. Change name of id to nativeQuery and namedTable in all widget jsons and change name in enum also.
Text field with name tableName can be there as it is.

…t in redshift and Postgres plugin.

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSchemaReader.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

amazon-redshift-plugin/widgets/Redshift-batchsource.json Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/CommonSchemaReader.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/source/AbstractDBSource.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 18, 2025

View reviewed changes

postgresql-plugin/src/main/java/io/cdap/plugin/postgres/PostgresSource.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 19, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/CommonSchemaReader.java Show resolved Hide resolved

vikasrathee-cs reviewed Jun 19, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/CommonSchemaReader.java Show resolved Hide resolved

vikasrathee-cs reviewed Jun 19, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 19, 2025

View reviewed changes

vikasrathee-cs added build and removed build labels Jun 19, 2025

AbhishekKumar9984 force-pushed the patch/abhi-tablename branch 2 times, most recently from c91b373 to bf6a780 Compare June 23, 2025 07:11

vikasrathee-cs reviewed Jun 23, 2025

View reviewed changes

amazon-redshift-plugin/src/main/java/io/cdap/plugin/amazon/redshift/RedshiftSource.java Outdated Show resolved Hide resolved

vikasrathee-cs reviewed Jun 23, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java Show resolved Hide resolved

vikasrathee-cs reviewed Jun 23, 2025

View reviewed changes

database-commons/src/main/java/io/cdap/plugin/db/config/AbstractDBSpecificSourceConfig.java Show resolved Hide resolved

AbhishekKumar9984 force-pushed the patch/abhi-tablename branch 2 times, most recently from e627cb7 to 467a8cd Compare June 25, 2025 09:24

vikasrathee-cs reviewed Jun 25, 2025

View reviewed changes

vikasrathee-cs added the build label Jun 25, 2025

AbhishekKumar9984 force-pushed the patch/abhi-tablename branch 3 times, most recently from 7675916 to 27bbd79 Compare June 27, 2025 09:45

vikasrathee-cs reviewed Jun 27, 2025

View reviewed changes

AbhishekKumar9984 force-pushed the patch/abhi-tablename branch 3 times, most recently from b83def1 to 5719a06 Compare July 2, 2025 10:46

psainics and others added 3 commits July 2, 2025 16:19

adding import query type property to db plugin.which will only reflec…

5828606

…t in redshift and Postgres plugin.

e2e fixes

f1bf6d5

fixing ups.

e2a9c67

AbhishekKumar9984 force-pushed the patch/abhi-tablename branch from 5719a06 to e2a9c67 Compare July 2, 2025 10:50

Patch/abhi tablename : Adding import query type property and modify tableName property for redshift and postgres plugin #607

Are you sure you want to change the base?

Patch/abhi tablename : Adding import query type property and modify tableName property for redshift and postgres plugin #607

Uh oh!

Conversation

AbhishekKumar9984 commented Jun 13, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vikasrathee-cs Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AbhishekKumar9984 Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vikasrathee-cs Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vikasrathee-cs Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vikasrathee-cs Jun 19, 2025 •

edited

Loading

AbhishekKumar9984 Jun 20, 2025 •

edited

Loading

vikasrathee-cs Jun 19, 2025 •

edited

Loading

vikasrathee-cs Jun 23, 2025 •

edited

Loading

vikasrathee-cs Jun 25, 2025 •

edited

Loading

vikasrathee-cs Jun 25, 2025 •

edited

Loading