Redshift feature: Include STS session token in COPY CREDENTIALS.

[timH6502]

Description

Enable Redshift COPY with temporary STS credentials by including the session token in the CREDENTIALS clause when present. If not provided, the old behavior is preserved.

Without the session token, COPY using temporary STS creds fails.
This only affects the S3 -> Redshift COPY path when credentials are temporary

More information can be found here:
https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-authorization.html

Specifically this section:
To use temporary token credentials, you must provide the temporary access key ID, the temporary secret access key, and the temporary token. The credentials-args string is in the following format.

CREDENTIALS
'aws_access_key_id=<temporary-access-key-id>;aws_secret_access_key=<temporary-secret-access-key>;token=<temporary-token>'

[timH6502]

@rudolfix I've updated the code, linting should (hopefully) not be an issue any more. May you please re-run the pipeline?

[rudolfix]

LGTM! last question: how do you pass STS to dlt? do you create configuration in code? could you paste code sample?

[timH6502]

LGTM! last question: how do you pass STS to dlt? do you create configuration in code? could you paste code sample?

The last commit is required. I've done that on the train and is was all a bit messy. Now it works again.
The key can be set via the aws_session_token in the secrets.toml (or via environment variables) as in the example below.

[destination.filesystem.credentials]
aws_access_key_id = ""
aws_secret_access_key = ""
aws_session_token = ""
region_name = ""
aws_iam_role = ""

[timH6502]

@rudolfix Is there something to be done, or will an autoformatter handle the rest?

[rudolfix]

@timH6502 auto format will not fix the code after it is pushed. you need to do make format and push the change. pls. try. otherwise the code is good

[timH6502]

@rudolfix Not sure what formatter/settings are required. However, the log indicates that only some identation had to be changed. Hope it works now.

Black (default settings vscode plugin) would change lines 131, 132, 139 and 140.

[rudolfix]

LGTM