Fail to use memory catalog-backend and s3 as warehouse, warehouse is hardcoded as /tmp

[jovezhong]

I am using the standalone rest server 0.8.0. It seems the warehouse is hardcoded somewhere for /tmp, so I cannot insert data

conf file

gravitino.iceberg-rest.catalog-backend = memory
gravitino.iceberg-rest.io-impl= org.apache.iceberg.aws.s3.S3FileIO
gravitino.iceberg-rest.warehouse = s3://tp-internal2/jove/gravitino1
gravitino.iceberg-rest.s3-endpoint = https://tp-internal2.s3.us-west-2.amazonaws.com
gravitino.iceberg-rest.s3-region = us-west-2
gravitino.iceberg-rest.credential-providers = s3-secret-key
gravitino.iceberg-rest.s3-access-key-id = AKIA2..
gravitino.iceberg-rest.s3-secret-access-key =

I made sure gravitino.iceberg-rest.warehouse = /tmp is not enabled.

However, when I use SparkSQL to create a database then create a table by following the doc

CREATE DATABASE IF NOT EXISTS dml;
DESCRIBE DATABASE EXTENDED dml;
CREATE TABLE dml.test (id bigint COMMENT 'unique id') using iceberg;
DESCRIBE TABLE EXTENDED dml.test;

I got /tmp/dml/test in the location

spark-sql (default)> DESCRIBE TABLE EXTENDED dml.test;
DESCRIBE TABLE EXTENDED dml.test
25/03/09 09:08:57 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
id                  	bigint              	unique id           
                    	                    	                    
# Metadata Columns  	                    	                    
_spec_id            	int                 	                    
_partition          	struct<>            	                    
_file               	string              	                    
_pos                	bigint              	                    
_deleted            	boolean             	                    
                    	                    	                    
# Detailed Table Information	                    	                    
Name                	spark_catalog.dml.test	                    
Type                	MANAGED             	                    
Location            	/tmp/dml/test       	                    
Provider            	iceberg             	                    
Owner               	jove                	                    
Table Properties    	[current-snapshot-id=none,format=iceberg/parquet,format-version=2,write.parquet.compression-codec=zstd]	

I even tried REST call directly for http://127.0.0.1:9001/iceberg/v1/namespaces/dml/tables/test and got

{"metadata-location":"/tmp/dml/test/metadata/00000-1c4006d0-807f-45ed-9923-dfc3285796b2.metadata.json","metadata":{"format-version":2,"table-uuid":"897ef33a-6e99-4a36-bb5a-982b8990315d","location":"/tmp/dml/test","last-sequence-number":0,"last-updated-ms":1741536661460,"last-column-id":1,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"long","doc":"unique id"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"owner":"jove","write.parquet.compression-codec":"zstd"},"current-snapshot-id":-1,"refs":{},"snapshots":[],"statistics":[],"partition-statistics":[],"snapshot-log":[],"metadata-log":[{"timestamp-ms":1741536531035,"metadata-file":"/tmp/dml/test/metadata/00000-1c4006d0-807f-45ed-9923-dfc3285796b2.metadata.json"}]},"config":{}}

So data cannot be added

INSERT INTO dml.test VALUES (1), (2)
25/03/09 09:21:34 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
org.apache.iceberg.exceptions.ValidationException: Invalid S3 URI, cannot determine scheme: /tmp/dml/test/data/00001-1-1daa1ad0-d698-4e8c-a764-6b5d77dcd1fd-0-00001.parquet
	at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:49)
	at org.apache.iceberg.aws.s3.S3URI.<init>(S3URI.java:75)
	at org.apache.iceberg.aws.s3.S3OutputFile.fromLocation(S3OutputFile.java:42)

[FANNG1]

memory catalog are used to do simple testing, suggest to try the docker image with sqllite as the default jdbc catalog backend in https://gravitino.apache.org/docs/0.8.0-incubating/iceberg-rest-service#docker-instructions.

[jovezhong]

Thanks for the reply. I can try JDBC backend, but in memory catalog is convenient to test without clean up resources. Can you check whether this is a bug somewhere in the memory catalog?

[FANNG1]

Yes, this seems a bug, memory catalog hard codes the warehouse to /tmp in Gravitino in IcebergCatalogUtil,

  private static InMemoryCatalog loadMemoryCatalog(IcebergConfig icebergConfig) {
    String icebergCatalogName = icebergConfig.getCatalogBackendName();
    InMemoryCatalog memoryCatalog = new InMemoryCatalog();
    Map<String, String> resultProperties = icebergConfig.getIcebergCatalogProperties();
    resultProperties.put(CatalogProperties.WAREHOUSE_LOCATION, "/tmp");
    memoryCatalog.initialize(icebergCatalogName, resultProperties);
    return memoryCatalog;
  }

[jovezhong]

Thanks for confirming this. It'll be great to fix this, otherwise it will confuse other users. I personally spent a few hours on this and slowly realized maybe memory backend has this /tmp hardcoded.

Now I moved to JDBC backend using a PG in the cloud, the iceberg_namespace_properties and iceberg_tables tables are created in PG, I can create a database via SparkSQL but cannot create tables (I also found 2 bugs not related to this: #6646 #6647)

spark-sql (default)> DESCRIBE DATABASE EXTENDED dml;
DESCRIBE DATABASE EXTENDED dml
Catalog Name	spark_catalog
Namespace Name	dml
Location	s3://tp-internal2/jove/gravitino1/dml
Owner	jove
Properties	
Time taken: 0.307 seconds, Fetched 5 row(s)
spark-sql (default)> CREATE TABLE dml.test (id bigint COMMENT 'unique id') using iceberg;
CREATE TABLE dml.test (id bigint COMMENT 'unique id') using iceberg
25/03/09 19:19:58 WARN ErrorHandlers: Unable to parse error response
java.lang.IllegalArgumentException: Cannot parse missing field: error
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
	at org.apache.iceberg.rest.responses.ErrorResponseParser.fromJson(ErrorResponseParser.java:79)
	at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:104)
	at org.apache.iceberg.rest.responses.ErrorResponseParser.fromJson(ErrorResponseParser.java:71)
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.parseResponse(ErrorHandlers.java:194)
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:194)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:308)
	at org.apache.iceberg.rest.BaseHTTPClient.post(BaseHTTPClient.java:88)
	at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:113)
	at org.apache.iceberg.rest.RESTSessionCatalog$Builder.create(RESTSessionCatalog.java:848)

The /gravitino-iceberg-rest-server-0.8.0-incubating-bin/logs/gravitino-iceberg-rest-server.out log file is not useful, I am expecting stacktrack in the log