Connect to localstack using AWS_ENDPOINT_URL environment variable

[zabil]

Hi,

I am trying to run AWS glue locally using the docker image amazon/aws-glue-libs:glue_libs_3.0.0_image_01
I am trying to make this connect to s3 bucket in local stack. However I can't seem to override the AWS_ENDPOINT_URL

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: ****; S3 Extended Request ID: ****; 
Proxy: null), S3 Extended Request ID: ****
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)

Is there a way for me to set an environment variable and make libraries that depend on this sdk like aws glue job point to a local stack?

[debora-ito]

Hey @zabil we are not the owners of aws-glue-libs, you can reach out to the team who owns it in their github repo: https://github.com/awslabs/aws-glue-libs

[zabil]

@debora-ito thanks for the quick response!

Apologies for not explaining this better.

I am using aws glue libs, however internally it uses aws-sdk-java because pyspark over the AWS's spark java implementation.

So I am looking to override the endpoint url as an environment variable so that the underlying java sdk points to localstack instead iof AWS

I see an open issue to implement this across aws other clients, but don't see java sdk there. So just wanted to check if it's already implemented.
aws/aws-sdk#229

[debora-ito]

Environment variable endpoints are not supported yet.