Questions about "No FileSystem for scheme "s3""

[senlizishi]

I encountered a similar situation to #21372. When executing CREATE SCHEMA iceberg.trino_s3_parquet_schema WITH (location = 's3://xxx/parquets/'); I encountered Got exception: org.apache.hadoop.fs .UnsupportedFileSystemException No FileSystem for scheme "s3"

I'm using the trinodb/trino:latest image. I first tried to download hadoop-aws-3.4.0.jar and placed it in my HMS /lib directory and restarted HMS and Trino, but the same error still occurred.

Next, I modified the docker startup method, mounted hadoop-aws-3.4.0.jar to the /usr/lib/trino/plugin/iceberg/ directory and restarted the service, but the result still did not work.

docker run --name trino -d -p 8080:8080 \
  --volume /usr/local/src/volume/catalog/iceberg.properties:/etc/trino/catalog/iceberg.properties \
  --volume /usr/local/src/volume/hadoop-aws-3.4.0.jar:/usr/lib/trino/plugin/iceberg/hadoop-aws-3.4.0.jar \
  trinodb/trino

My iceberg. properties content is as follows:

connector.name=iceberg
hive.metastore.uri=thrift://xxxxxx:9083
iceberg.catalog.type=hive_metastore
iceberg.file-format=PARQUET
fs.hadoop.enabled=false
fs.native-s3.enabled=true
s3.region=xxxxx
s3.aws-access-key=xxxxxx
s3.aws-secret-key=xxxxxx
s3.path-style-access=true  

Please tell me where is the configuration problem? Thx ;)

[senlizishi]

I have solved the above problem. This is mainly related to the deployment configuration of HMS.

Solutions

• Check whether the downloaded Hadoop version is consistent with the version of /usr/lib/trino/plugin/iceberg/hdfs/hadoop-apache-x.jar in the Trino container. Then check whether the Hive version is compatible with the downloaded Hadoop version. For example, I currently use Trino-451, hadoop-3.3.5, hive-3.1.3.
• Download the aws-java-sdk-core-x.jar, aws-java-sdk-s3-x.jar, and hadoop-aws-x.jar from https://mvnrepository.com/ and place them in ${HADOOP_HOME}/share/hadoop/tools/lib/ directory. The versions of aws-java-sdk-core and aws-java-sdk-s3 Jar are consistent with the jars of the same name in the /usr/lib/trino/plugin/iceberg directory of the Trino container. The hadoop-aws Jar version is consistent with Hadoop.
• Modify the hive /conf/hive-env.sh file, save and restart the HMS service. The content of the file is as follows

export HADOOP_HOME=/usr/local/src/hadoop-3.3.5
export HIVE_AUX_JARS_PATH=${HADOOP_HOME}/share/hadoop/tools/lib/aws-java-sdk-core-1.12.749.jar:${HADOOP_HOME}/share/hadoop/tools/lib/aws-java-sdk-s3-1.12.749.jar:${HADOOP_HOME}/share/hadoop/tools/lib/hadoop-aws-3.3.5.jar
export AWS_ACCESS_KEY_ID=xxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxx

• Last modified s3:// to s3a://

CREATE SCHEMA iceberg.trino_s3_parquet_schema
WITH (location = 's3a://<bucketname>/parquets/');