查询default库表的字段时获取到重复的列名

[user-GitHub-user]

kyuubi:1.7.0
spark:3.2.1

使用 connection.getMetaData().getColumns 方法, 除了default库外，其他库均正常无重复列

@Test
    public void testKyuubiJDBC() throws SQLException {
        try (Connection connection = DriverManager.getConnection("jdbc:hive2://dev1:2181,dev2:2181,dev3:2181/;"
            + "serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi#kyuubi.engine.share.level=SERVER")) {
            ResultSet resultSet = connection.getMetaData().getColumns("spark_catalog", "default", "table",
                null);
            int length = resultSet.getMetaData().getColumnCount();
            for (int i = 1; i <= length; i++) {
                System.out.println(resultSet.getMetaData().getColumnName(i));
            }
            while (resultSet.next()) {
                for (int i = 1; i <= length; i++) {
                    System.out.print(resultSet.getString(i));
                    System.out.print("  ");
                }
                System.out.println();
            }
        }
    }

[cxzl25]

Using the provided version, failed to reproduce, can you provide a little more detail?

[user-GitHub-user]

hi, i use spark for hive catalog, my database and table info:

and my spark catalog config(i also use iceberg):

spark.sql.catalog.spark_catalog.type = hive
spark.hive.metastore.uris=thrift://dev2:9083,thrift://dev3:9083,thrift://dev1:9083
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.warehouse=hdfs://namespace-HA-1/user/hive/warehouse

[cxzl25]

Because iceberg's default namespace is default, kyuubi does not really deduplicate the namespace deduplication logic, resulting in repeated fields being returned.

This should be a bug, are you interested in submitting a PR to fix it?

kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/shim/CatalogShim_v3_0.scala

Line 87 in 3108c8e

catalog.defaultNamespace() +: catalog.listNamespaces(Array())

kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/shim/CatalogShim_v3_0.scala

Lines 106 to 113 in 3108c8e

	private def listNamespacesWithPattern(
	catalog: CatalogPlugin,
	schemaPattern: String): Array[Array[String]] = {
	val p = schemaPattern.r.pattern
	listAllNamespaces(catalog).filter { ns =>
	val quoted = ns.map(quoteIfNeeded).mkString(".")
	p.matcher(quoted).matches()
	}.distinct

[cxzl25]

#5105 Solved this duplicate column issue.