Skip to content

Commit 4aaf639

Browse files
general-kroll-4-lifeCopilot
andcommitted
identify-federation
Summary: - OIDC support for: - `google`. - `aws`. - `azure`. - Added robot test `ID Fed AWS S3 Buckets Traffic Light Canonical`. - Added robot test `ID Fed Azure Public Keys Traffic Light Canonical`. - Added robot test `ID Fed Google Container Agg Desc Traffic Light Canonical`. Co-authored-by: Copilot <copilot@github.com>
1 parent d1f204e commit 4aaf639

5 files changed

Lines changed: 170 additions & 20 deletions

File tree

go.mod

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ require (
1818
github.com/spf13/cobra v1.4.0
1919
github.com/spf13/pflag v1.0.5
2020
github.com/spf13/viper v1.10.1
21-
github.com/stackql/any-sdk v0.5.3-alpha05
21+
github.com/stackql/any-sdk v0.5.3-alpha07
2222
github.com/stackql/go-suffix-map v0.0.1-alpha01
2323
github.com/stackql/psql-wire v0.1.2-beta01
2424
github.com/stackql/stackql-parser v0.0.16-alpha02
@@ -68,12 +68,14 @@ require (
6868
github.com/aws/aws-sdk-go-v2/service/sts v1.28.6 // indirect
6969
github.com/aws/smithy-go v1.20.2 // indirect
7070
github.com/clbanning/mxj/v2 v2.7.0 // indirect
71+
github.com/coreos/go-oidc/v3 v3.18.0 // indirect
7172
github.com/danieljoos/wincred v1.2.2 // indirect
7273
github.com/davecgh/go-spew v1.1.1 // indirect
7374
github.com/dvsekhvalnov/jose2go v1.6.0 // indirect
7475
github.com/fsnotify/fsnotify v1.5.1 // indirect
7576
github.com/gabriel-vasile/mimetype v1.4.7 // indirect
7677
github.com/ghodss/yaml v1.0.0 // indirect
78+
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
7779
github.com/go-openapi/jsonpointer v0.19.5 // indirect
7880
github.com/go-openapi/swag v0.21.1 // indirect
7981
github.com/goccy/go-json v0.10.4 // indirect
@@ -127,7 +129,7 @@ require (
127129
golang.org/x/crypto v0.39.0 // indirect
128130
golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 // indirect
129131
golang.org/x/net v0.41.0 // indirect
130-
golang.org/x/oauth2 v0.26.0 // indirect
132+
golang.org/x/oauth2 v0.36.0 // indirect
131133
golang.org/x/sys v0.33.0 // indirect
132134
golang.org/x/term v0.32.0 // indirect
133135
golang.org/x/text v0.26.0 // indirect

go.sum

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,8 @@ github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnht
150150
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
151151
github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
152152
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
153+
github.com/coreos/go-oidc/v3 v3.18.0 h1:V9orjXynvu5wiC9SemFTWnG4F45v403aIcjWo0d41+A=
154+
github.com/coreos/go-oidc/v3 v3.18.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26KVTCI9SrY4=
153155
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
154156
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
155157
github.com/cpuguy83/go-md2man/v2 v2.0.1/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
@@ -184,6 +186,8 @@ github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeME
184186
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
185187
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
186188
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
189+
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
190+
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
187191
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
188192
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
189193
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
@@ -467,8 +471,8 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
467471
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
468472
github.com/spf13/viper v1.10.1 h1:nuJZuYpG7gTj/XqiUwg8bA0cp1+M2mC3J4g5luUYBKk=
469473
github.com/spf13/viper v1.10.1/go.mod h1:IGlFPqhNAPKRxohIzWpI5QEy4kuI7tcl5WvR+8qy1rU=
470-
github.com/stackql/any-sdk v0.5.3-alpha05 h1:Be5d0k5o1fLotY06ToKseNxp/JTFp2P4LpSxspedGw4=
471-
github.com/stackql/any-sdk v0.5.3-alpha05/go.mod h1:yGAr3eEtY6UpFMq8fF5BTvZhdqYGpyS90LP6KY5AKZs=
474+
github.com/stackql/any-sdk v0.5.3-alpha07 h1:EuOx9AMFEW4SkO10kQ4E0FXcyzsmhzDQQz0kTgoXqII=
475+
github.com/stackql/any-sdk v0.5.3-alpha07/go.mod h1:jTU9+GetqZ4lbcSF0D1Cf034nfxiAy0uFKzdi7hx7EE=
472476
github.com/stackql/go-suffix-map v0.0.1-alpha01 h1:TDUDS8bySu41Oo9p0eniUeCm43mnRM6zFEd6j6VUaz8=
473477
github.com/stackql/go-suffix-map v0.0.1-alpha01/go.mod h1:QAi+SKukOyf4dBtWy8UMy+hsXXV+yyEE4vmBkji2V7g=
474478
github.com/stackql/psql-wire v0.1.2-beta01 h1:+0xeLLam3nbI1iv84pWJYGYw5VO5S/+eICRbdu6+dj8=
@@ -657,8 +661,8 @@ golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914/go.mod h1:KelEdhl1UZF7XfJ
657661
golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
658662
golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
659663
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
660-
golang.org/x/oauth2 v0.26.0 h1:afQXWNNaeC4nvZ0Ed9XvCCzXM6UHJG7iCg0W4fPqSBE=
661-
golang.org/x/oauth2 v0.26.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
664+
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
665+
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
662666
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
663667
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
664668
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=

test/python/stackql_test_tooling/StackQLInterfaces.py

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -888,7 +888,7 @@ def should_stackql_exec_inline_equal_both_streams(
888888
*args,
889889
**cfg
890890
)
891-
return self._verify_both_streams(result, expected_output, expected_stderr_output, **cfg)
891+
self._verify_both_streams(result, expected_output, expected_stderr_output, **cfg)
892892

893893

894894
@keyword
@@ -996,19 +996,21 @@ def should_stackql_exec_inline_contain_both_streams(
996996
*args,
997997
**cfg
998998
):
999-
result = self._run_stackql_exec_command(
1000-
stackql_exe,
1001-
okta_secret_str,
1002-
github_secret_str,
1003-
k8s_secret_str,
1004-
registry_cfg,
1005-
auth_cfg_str,
1006-
sql_backend_cfg_str,
1007-
query,
1008-
*args,
1009-
**cfg
1010-
)
1011-
return self._contain_both_streams(result, expected_output, expected_output_stderr)
999+
repeat_count = int(cfg.pop('repeat_count', 1))
1000+
for _ in range(repeat_count):
1001+
result = self._run_stackql_exec_command(
1002+
stackql_exe,
1003+
okta_secret_str,
1004+
github_secret_str,
1005+
k8s_secret_str,
1006+
registry_cfg,
1007+
auth_cfg_str,
1008+
sql_backend_cfg_str,
1009+
query,
1010+
*args,
1011+
**cfg
1012+
)
1013+
self._contain_both_streams(result, expected_output, expected_output_stderr)
10121014

10131015

10141016
@keyword

test/python/stackql_test_tooling/flask/oauth2/token_srv.py

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@
99
import logging
1010

1111
from typing import List
12+
from datetime import datetime, timedelta, timezone
13+
from itertools import count
1214

1315

1416
"""
@@ -64,6 +66,18 @@ def generate_google_response_dict(self, default_scopes: List[str]=["https://www.
6466
"expires_in": 3600
6567
}
6668

69+
_AZURE_FEDERATED_TOKEN_COUNTER = count(1)
70+
_GCP_STS_TOKEN_COUNTER = count(1)
71+
_GCP_IMPERSONATED_TOKEN_COUNTER = count(1)
72+
73+
74+
def _utc_now() -> datetime:
75+
return datetime.now(timezone.utc)
76+
77+
78+
def _isoformat_z(ts: datetime) -> str:
79+
return ts.astimezone(timezone.utc).replace(microsecond=0).isoformat().replace('+00:00', 'Z')
80+
6781
def _form_to_str(form: ImmutableMultiDict) -> str:
6882
return json.dumps(form.to_dict(flat=False), sort_keys=True)
6983

@@ -84,3 +98,66 @@ def contrived_simple_token():
8498
request_data = request.form
8599
app.logger.info(f'POST /contrived/simple/token request data: {_form_to_str(request_data)}')
86100
return json.dumps(_SIMPLE_RESPONSE, sort_keys=True)
101+
102+
103+
@app.route("/aws/sts", methods=['POST'])
104+
def aws_sts_assume_role_with_web_identity():
105+
request_data = request.form
106+
action = request_data.get('Action')
107+
token = request_data.get('WebIdentityToken')
108+
if action != 'AssumeRoleWithWebIdentity' or not token:
109+
return json.dumps({"msg": "malformed aws_web_identity request"}, sort_keys=True), 400
110+
expiration = _isoformat_z(_utc_now() + timedelta(seconds=1))
111+
response_xml = f"""<?xml version=\"1.0\" encoding=\"UTF-8\"?>
112+
<AssumeRoleWithWebIdentityResponse xmlns=\"https://sts.amazonaws.com/doc/2011-06-15/\">
113+
<AssumeRoleWithWebIdentityResult>
114+
<Credentials>
115+
<AccessKeyId>ASIA_MOCK_IDFED</AccessKeyId>
116+
<SecretAccessKey>mock-idfed-secret</SecretAccessKey>
117+
<SessionToken>mock-idfed-session-token</SessionToken>
118+
<Expiration>{expiration}</Expiration>
119+
</Credentials>
120+
</AssumeRoleWithWebIdentityResult>
121+
</AssumeRoleWithWebIdentityResponse>"""
122+
return response_xml, 200, {"Content-Type": "text/xml"}
123+
124+
125+
@app.route("/gcp/sts/token", methods=['POST'])
126+
def gcp_workload_identity_sts_token():
127+
request_data = request.form
128+
subject_token = request_data.get('subject_token')
129+
if not subject_token:
130+
return json.dumps({"msg": "missing subject_token"}, sort_keys=True), 400
131+
token_index = next(_GCP_STS_TOKEN_COUNTER)
132+
response = {
133+
"access_token": f"gcp-fed-token-{token_index}",
134+
"issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
135+
"token_type": "Bearer",
136+
"expires_in": 1,
137+
}
138+
return json.dumps(response, sort_keys=True)
139+
140+
141+
@app.route("/gcp/iamcredentials/generateAccessToken", methods=['POST'])
142+
def gcp_workload_identity_impersonation():
143+
token_index = next(_GCP_IMPERSONATED_TOKEN_COUNTER)
144+
response = {
145+
"accessToken": f"gcp-impersonated-token-{token_index}",
146+
"expireTime": _isoformat_z(_utc_now() + timedelta(seconds=1)),
147+
}
148+
return json.dumps(response, sort_keys=True)
149+
150+
151+
@app.route("/azure/federated/token", methods=['POST'])
152+
def azure_federated_token():
153+
request_data = request.form
154+
subject_token = request_data.get('subject_token')
155+
if not subject_token:
156+
return json.dumps({"msg": "missing subject_token"}, sort_keys=True), 400
157+
token_index = next(_AZURE_FEDERATED_TOKEN_COUNTER)
158+
response = {
159+
"access_token": f"azure-fed-token-{token_index}",
160+
"token_type": "Bearer",
161+
"expires_in": 1,
162+
}
163+
return json.dumps(response, sort_keys=True)

test/robot/functional/stackql_mocked_from_cmd_line.robot

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4508,6 +4508,71 @@ Oauth2 CLient Credentials Auth Should Fail with Invalid Config
45084508
... stdout=${CURDIR}/tmp/Oauth2-CLient-Credentials-Auth-Should-Fail-with-Invalid-Config.tmp
45094509
... stderr=${CURDIR}/tmp/Oauth2-CLient-Credentials-Auth-Should-Fail-with-Invalid-Config-stderr.tmp
45104510

4511+
ID Fed AWS S3 Buckets Traffic Light Canonical
4512+
[Tags] id_fed_traffic_light registry tls_proxied
4513+
Pass Execution If "${RUN_ID_FED_TRAFFIC_LIGHTS}" != "true" Running only in dedicated id federation traffic-light CI step
4514+
Create File ${CURDIR}/tmp/id-fed-aws-subject-token.jwt mock-oidc-subject-token-aws
4515+
${authCfg} = Catenate
4516+
... {"aws":{"type":"aws_web_identity","aws_role_arn":"arn:aws:iam::123456789012:role/mock-idfed-role","aws_sts_region":"us-east-1","aws_sts_endpoint":"https://${LOCAL_HOST_ALIAS}:${MOCKSERVER_PORT_OAUTH_CLIENT_CREDENTIALS_TOKEN}/aws/sts","oidc_subject_token_file":"${CURDIR}/tmp/id-fed-aws-subject-token.jwt"}}
4517+
Should Stackql Exec Inline Contain Both Streams
4518+
... ${STACKQL_EXE}
4519+
... ${OKTA_SECRET_STR}
4520+
... ${GITHUB_SECRET_STR}
4521+
... ${K8S_SECRET_STR}
4522+
... ${REGISTRY_NO_VERIFY_CFG_STR}
4523+
... ${authCfg}
4524+
... ${SQL_BACKEND_CFG_STR_CANONICAL}
4525+
... ${SELECT_AWS_S3_BUCKETS}
4526+
... CreationDate
4527+
... ${EMPTY}
4528+
... stdout=${CURDIR}/tmp/ID-Fed-AWS-S3-Buckets-Traffic-Light-Canonical-stdout.tmp
4529+
... stderr=${CURDIR}/tmp/ID-Fed-AWS-S3-Buckets-Traffic-Light-Canonical-stderr.tmp
4530+
... repeat_count=4
4531+
4532+
ID Fed Azure Public Keys Traffic Light Canonical
4533+
[Tags] id_fed_traffic_light registry tls_proxied
4534+
Pass Execution If "${RUN_ID_FED_TRAFFIC_LIGHTS}" != "true" Running only in dedicated id federation traffic-light CI step
4535+
Create File ${CURDIR}/tmp/id-fed-azure-subject-token.jwt mock-oidc-subject-token-azure
4536+
${authCfg} = Catenate
4537+
... {"azure":{"type":"azure_federated","azure_tenant_id":"00000000-0000-0000-0000-000000000000","client_id":"11111111-1111-1111-1111-111111111111","scopes":["https://management.azure.com/.default"],"azure_federated_endpoint":"https://${LOCAL_HOST_ALIAS}:${MOCKSERVER_PORT_OAUTH_CLIENT_CREDENTIALS_TOKEN}/azure/federated/token","oidc_subject_token_file":"${CURDIR}/tmp/id-fed-azure-subject-token.jwt"}}
4538+
${inputStr} = Catenate
4539+
... select name from azure.network.virtual_networks where subscriptionId = 'subid' order by name asc;
4540+
Should Stackql Exec Inline Contain Both Streams
4541+
... ${STACKQL_EXE}
4542+
... ${OKTA_SECRET_STR}
4543+
... ${GITHUB_SECRET_STR}
4544+
... ${K8S_SECRET_STR}
4545+
... ${REGISTRY_NO_VERIFY_CFG_STR}
4546+
... ${authCfg}
4547+
... ${SQL_BACKEND_CFG_STR_CANONICAL}
4548+
... ${inputStr}
4549+
... name
4550+
... ${EMPTY}
4551+
... stdout=${CURDIR}/tmp/ID-Fed-Azure-Public-Keys-Traffic-Light-Canonical-stdout.tmp
4552+
... stderr=${CURDIR}/tmp/ID-Fed-Azure-Public-Keys-Traffic-Light-Canonical-stderr.tmp
4553+
... repeat_count=4
4554+
4555+
ID Fed Google Container Agg Desc Traffic Light Canonical
4556+
[Tags] id_fed_traffic_light registry tls_proxied
4557+
Pass Execution If "${RUN_ID_FED_TRAFFIC_LIGHTS}" != "true" Running only in dedicated id federation traffic-light CI step
4558+
Create File ${CURDIR}/tmp/id-fed-gcp-subject-token.jwt mock-oidc-subject-token-gcp
4559+
${authCfg} = Catenate
4560+
... {"google":{"type":"gcp_workload_identity","gcp_workload_identity_audience":"//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/mock-pool/providers/mock-provider","gcp_workload_identity_token_url":"https://${LOCAL_HOST_ALIAS}:${MOCKSERVER_PORT_OAUTH_CLIENT_CREDENTIALS_TOKEN}/gcp/sts/token","gcp_service_account_impersonation_url":"https://${LOCAL_HOST_ALIAS}:${MOCKSERVER_PORT_OAUTH_CLIENT_CREDENTIALS_TOKEN}/gcp/iamcredentials/generateAccessToken","scopes":["https://www.googleapis.com/auth/cloud-platform"],"oidc_subject_token_file":"${CURDIR}/tmp/id-fed-gcp-subject-token.jwt"}}
4561+
Should Stackql Exec Inline Contain Both Streams
4562+
... ${STACKQL_EXE}
4563+
... ${OKTA_SECRET_STR}
4564+
... ${GITHUB_SECRET_STR}
4565+
... ${K8S_SECRET_STR}
4566+
... ${REGISTRY_NO_VERIFY_CFG_STR}
4567+
... ${authCfg}
4568+
... ${SQL_BACKEND_CFG_STR_CANONICAL}
4569+
... ${SELECT_CONTAINER_SUBNET_AGG_DESC}
4570+
... ipCidrRange
4571+
... ${EMPTY}
4572+
... stdout=${CURDIR}/tmp/ID-Fed-Google-Container-Agg-Desc-Traffic-Light-Canonical-stdout.tmp
4573+
... stderr=${CURDIR}/tmp/ID-Fed-Google-Container-Agg-Desc-Traffic-Light-Canonical-stderr.tmp
4574+
... repeat_count=4
4575+
45114576
HTTP Log enabled regression test
45124577
Should Horrid HTTP Log Enabled Query StackQL Inline Equal
45134578
... ${STACKQL_EXE}

0 commit comments

Comments
 (0)