Skip to content

Commit 7ff1e95

Browse files
general-kroll-4-lifegeneral-kroll-4-life
committed
id-fed
Summary: - Support for OIDC to 3 major hyperscalars. - Added robot test `ID Fed AWS S3 Buckets List`. - Added robot test `ID Fed Azure VNETs List`. - Added robot test `ID Fed Google Buckets List`.
1 parent 55ff2da commit 7ff1e95

8 files changed

Lines changed: 200 additions & 0 deletions

File tree

‎.github/workflows/build.yml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -546,6 +546,9 @@ jobs:
546546
needs:
547547
- linuxbuild
548548
- test_python_package_build
549+
permissions:
550+
id-token: write
551+
contents: read
549552
runs-on: ubuntu-22-04-m
550553
strategy:
551554
matrix:
@@ -759,6 +762,58 @@ jobs:
759762
run: |
760763
cat ./test/robot/foreign-integration-traffic-lights/tmp/* || true
761764
765+
- name: ID Fed - configure AWS credentials (OIDC)
766+
if: ((startsWith(github.ref_name, 'build-traffic-lights') && github.ref_type == 'tag') || (github.repository == 'stackql/stackql' && github.event_name == 'push' && github.ref == 'refs/heads/main')) && matrix.registry == 'test/registry'
767+
uses: aws-actions/configure-aws-credentials@v4
768+
with:
769+
role-to-assume: ${{ secrets.STACKQL_ID_FED_AWS_ROLE_ARN }}
770+
aws-region: us-east-1
771+
772+
- name: ID Fed - authenticate to Google Cloud (OIDC)
773+
if: ((startsWith(github.ref_name, 'build-traffic-lights') && github.ref_type == 'tag') || (github.repository == 'stackql/stackql' && github.event_name == 'push' && github.ref == 'refs/heads/main')) && matrix.registry == 'test/registry'
774+
uses: google-github-actions/auth@v2
775+
with:
776+
workload_identity_provider: ${{ secrets.STACKQL_ID_FED_GCP_WORKLOAD_IDENTITY_PROVIDER }}
777+
service_account: ${{ secrets.STACKQL_ID_FED_GCP_SERVICE_ACCOUNT }}
778+
779+
- name: ID Fed - login to Azure (OIDC)
780+
if: ((startsWith(github.ref_name, 'build-traffic-lights') && github.ref_type == 'tag') || (github.repository == 'stackql/stackql' && github.event_name == 'push' && github.ref == 'refs/heads/main')) && matrix.registry == 'test/registry'
781+
uses: azure/login@v2
782+
with:
783+
client-id: ${{ secrets.STACKQL_ID_FED_AZURE_CLIENT_ID }}
784+
tenant-id: ${{ secrets.STACKQL_ID_FED_AZURE_TENANT_ID }}
785+
subscription-id: ${{ secrets.AZURE_INTEGRATION_TESTING_SUB_ID }}
786+
787+
- name: Run id-fed traffic light robot integration tests
788+
if: ((startsWith(github.ref_name, 'build-traffic-lights') && github.ref_type == 'tag') || (github.repository == 'stackql/stackql' && github.event_name == 'push' && github.ref == 'refs/heads/main')) && matrix.registry == 'test/registry'
789+
env:
790+
PYTHONPATH: '${{ env.PYTHONPATH }}:${{ github.workspace }}/test/python'
791+
AZURE_TARGET_SUBSCRIPTION_ID: ${{ secrets.AZURE_INTEGRATION_TESTING_SUB_ID }}
792+
GODEBUG: netdns=go
793+
run: |
794+
echo "## Stray flask apps to be killed before robot tests ##"
795+
pgrep -f flask | xargs kill -9 || true
796+
echo "## End ##"
797+
if python cicd/python/build.py --robot-test-id-fed-traffic-lights-integration; then
798+
echo "✅ ID-FED traffic light robot integration tests **all** passed"
799+
else
800+
rv="$?"
801+
echo "🟡 **some** ID-FED traffic light robot integration tests failed code = $rv"
802+
fi
803+
{
804+
echo "ID_FED_TRAFFIC_LIGHTS_COMPLETED=true"
805+
} >> "$GITHUB_ENV"
806+
807+
- name: Output from id-fed traffic lights integration tests
808+
if: env.ID_FED_TRAFFIC_LIGHTS_COMPLETED == 'true'
809+
run: |
810+
cat ./test/robot/reports-id-fed-traffic-lights/output.xml || true
811+
812+
- name: Output from id-fed traffic lights tmp dir
813+
if: env.ID_FED_TRAFFIC_LIGHTS_COMPLETED == 'true'
814+
run: |
815+
cat ./test/robot/id-fed-traffic-lights/tmp/* || true
816+
762817
- name: Generate hosts file and nginx materials
763818
env:
764819
BUILDCOMMITSHA: ${{github.sha}}

‎cicd/python/build.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,16 @@ def run_robot_foreign_integration_traffic_lights_tests_stackql(*args, **kwargs)
9292
shell=True
9393
)
9494

95+
def run_robot_id_fed_integration_traffic_lights_tests_stackql(*args, **kwargs) -> int:
96+
variables = ' '.join([f'--variable {key}:{sanitise_val(value)} ' for key, value in kwargs.get("variables", {}).items()])
97+
return subprocess.call(
98+
'robot '
99+
f'{variables} '
100+
'-d test/robot/reports-id-fed-traffic-lights '
101+
'test/robot/id-fed-traffic-lights',
102+
shell=True
103+
)
104+
95105
def main():
96106
parser = argparse.ArgumentParser()
97107
parser.add_argument('--verbose', action='store_true')
@@ -101,6 +111,7 @@ def main():
101111
parser.add_argument('--robot-test', action='store_true')
102112
parser.add_argument('--robot-test-integration', action='store_true')
103113
parser.add_argument('--robot-test-traffic-lights-integration', action='store_true')
114+
parser.add_argument('--robot-test-id-fed-traffic-lights-integration', action='store_true')
104115
parser.add_argument('--robot-test-foreign-traffic-lights-integration', action='store_true')
105116
parser.add_argument('--config', type=json.loads, default={})
106117
args = parser.parse_args()
@@ -133,6 +144,10 @@ def main():
133144
ret_code = run_robot_foreign_integration_traffic_lights_tests_stackql(**args.config)
134145
if ret_code != 0:
135146
exit(ret_code)
147+
if args.robot_test_id_fed_traffic_lights_integration:
148+
ret_code = run_robot_id_fed_integration_traffic_lights_tests_stackql(**args.config)
149+
if ret_code != 0:
150+
exit(ret_code)
136151
exit(ret_code)
137152

138153

‎docs/auth.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,3 +66,28 @@ source cicd/vol/vendor-secrets/foreign_to_stackql_user.sh
6666
stackql --auth '{"aws":{"type":"aws_assume_role","keyIDenvvar":"AWS_ACCESS_KEY_ID","credentialsenvvar":"AWS_SECRET_ACCESS_KEY","aws_role_arn":"'"${STACKQL_AUDIT_ROLE_ARN}"'"}}' shell
6767
```
6868

69+
## OIDC from github
70+
71+
GHA → cloud federation via each provider's official action; stackql then uses its standard auth types — **no federation-specific `--auth` string needed**.
72+
73+
Workflow needs `permissions: id-token: write`, then per cloud:
74+
75+
**AWS** — [GitHub docs](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). IAM OIDC provider + role; `aws-actions/configure-aws-credentials@v4` with `role-to-assume`. Session token (`AWS_SESSION_TOKEN`) is picked up from env automatically — no extra field needed.
76+
```
77+
--auth '{"aws":{"type":"aws_signing_v4","keyIDenvvar":"AWS_ACCESS_KEY_ID","credentialsenvvar":"AWS_SECRET_ACCESS_KEY"}}'
78+
```
79+
80+
**GCP** — [GitHub docs](https://github.com/google-github-actions/auth#setup). Workload Identity **Pool** + Provider, SA bound via `roles/iam.workloadIdentityUser`; `google-github-actions/auth@v2` with `workload_identity_provider` + `service_account`.
81+
```
82+
--auth '{"google":{"type":"service_account","credentialsfilepath":"'"${GOOGLE_APPLICATION_CREDENTIALS}"'"}}'
83+
```
84+
85+
**Azure** — [action docs](https://github.com/Azure/login#login-with-openid-connect-oidc-recommended). Entra app + federated credential (one per org — Azure won't accept `repository_owner` matching); `azure/login@v2` with client/tenant/subscription IDs.
86+
```
87+
--auth '{"azure":{"type":"azure_default"}}'
88+
```
89+
90+
### Gotchas
91+
- GCP: use **Workload** Identity Federation (project-scoped), not Workforce. Impersonation needs `roles/iam.workloadIdentityUser`. GH secret = full provider resource name (`projects/.../providers/...`).
92+
- Azure: federated credentials for GHA disallow `repository_owner` matching → create one credential per org, or regex on `claims['sub']`.
93+

‎test/robot/id-fed-traffic-lights/__init__.robot‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
*** Settings ***
2+
Resource ${CURDIR}/stackql.resource
3+
4+

‎test/robot/id-fed-traffic-lights/stackql.resource‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
*** Variables ***
2+
${LOCAL_LIB_HOME} ${CURDIR}${/}..${/}..${/}python
3+
${REPOSITORY_ROOT} ${CURDIR}${/}..${/}..${/}..
4+
${EXECUTION_PLATFORM} native # to be overridden from command line, eg "docker"
5+
${SQL_BACKEND} sqlite_embedded # to be overridden from command line, eg "postgres_tcp"
6+
${IS_WSL} false # to be overridden from command line, with string "true"
7+
${USE_STACKQL_PREINSTALLED} false # to be overridden from command line, with string "true"
8+
${SUNDRY_CONFIG} {} # to be overridden from command line, with string value
9+
${STACKQL_INTERFACE_LIBRARY} stackql_test_tooling.StackQLInterfaces
10+
${CLOUD_INTEGRATION_LIBRARY} stackql_test_tooling.CloudIntegration
11+
12+
*** Settings ***
13+
Library Process
14+
Library OperatingSystem
15+
Variables ${LOCAL_LIB_HOME}/stackql_test_tooling/stackql_context.py ${REPOSITORY_ROOT} ${EXECUTION_PLATFORM} ${SQL_BACKEND} ${USE_STACKQL_PREINSTALLED}
16+
... ${SUNDRY_CONFIG}
17+
Library Process
18+
Library OperatingSystem
19+
Library String
20+
Library ${STACKQL_INTERFACE_LIBRARY} ${EXECUTION_PLATFORM} ${SQL_BACKEND}
21+
Library ${CLOUD_INTEGRATION_LIBRARY}

‎test/robot/id-fed-traffic-lights/stackql_id_fed_traffic_light_integration_from_cmd_line.robot‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
2+
3+
4+
*** Settings ***
5+
Resource ${CURDIR}/stackql.resource
6+
7+
*** Test Cases ***
8+
9+
ID Fed AWS S3 Buckets List
10+
Sleep 2s
11+
${awsAuthCfg} = Catenate
12+
... { "aws": { "type":"aws_signing_v4", "keyIDenvvar": "AWS_ACCESS_KEY_ID", "credentialsenvvar": "AWS_SECRET_ACCESS_KEY" } }
13+
${bucketsListQuery} = Catenate
14+
... select * from aws.pseudo_s3.buckets_list_only where region = 'ap-southeast-2';
15+
${result} = Run Process
16+
... ${STACKQL_EXE}
17+
... \-\-auth
18+
... ${awsAuthCfg}
19+
... \-\-registry
20+
... { "url": "file://${REPOSITORY_ROOT}/test/registry", "localDocRoot": "${REPOSITORY_ROOT}/test/registry", "verifyConfig": { "nopVerify": true } }
21+
... exec
22+
... ${bucketsListQuery}
23+
... cwd=${REPOSITORY_ROOT}
24+
... stdout=${CURDIR}/tmp/ID-Fed-AWS-S3-Buckets-List.tmp
25+
... stderr=${CURDIR}/tmp/ID-Fed-AWS-S3-Buckets-List-stderr.tmp
26+
Should Be Equal As Integers ${result.rc} 0
27+
Should Be Empty ${result.stderr}
28+
Should Contain ${result.stdout} stackql\-trial\-bucket\-02
29+
30+
31+
ID Fed Azure VNETs List
32+
Sleep 2s
33+
${azureTargetSubscription} = OperatingSystem.Get Environment Variable AZURE_TARGET_SUBSCRIPTION_ID
34+
Should Not Be Empty ${azureTargetSubscription}
35+
${azureAuthCfg} = Catenate
36+
... { "azure": { "type":"azure_default" } }
37+
${bucketsListQuery} = Catenate
38+
... select location, name from azure.network.virtual_networks where subscriptionId = '${azureTargetSubscription}';
39+
${result} = Run Process
40+
... ${STACKQL_EXE}
41+
... \-\-auth
42+
... ${azureAuthCfg}
43+
... \-\-registry
44+
... { "url": "file://${REPOSITORY_ROOT}/test/registry", "localDocRoot": "${REPOSITORY_ROOT}/test/registry", "verifyConfig": { "nopVerify": true } }
45+
... exec
46+
... ${bucketsListQuery}
47+
... cwd=${REPOSITORY_ROOT}
48+
... stdout=${CURDIR}/tmp/ID-Fed-Azure-VNETs-List.tmp
49+
... stderr=${CURDIR}/tmp/ID-Fed-Azure-VNETs-List-stderr.tmp
50+
Should Be Equal As Integers ${result.rc} 0
51+
Should Be Empty ${result.stderr}
52+
Should Contain ${result.stdout} inspector\-network
53+
54+
55+
ID Fed Google Buckets List
56+
Sleep 2s
57+
${gcpCredentialsFile} = OperatingSystem.Get Environment Variable GOOGLE_APPLICATION_CREDENTIALS
58+
Should Not Be Empty ${gcpCredentialsFile}
59+
${gcpAuthCfg} = Catenate
60+
... { "google": { "type":"service_account", "credentialsfilepath": "${gcpCredentialsFile}" } }
61+
${bucketsListQuery} = Catenate
62+
... select location, name from google.storage.buckets where project = 'stackql-demo';
63+
${result} = Run Process
64+
... ${STACKQL_EXE}
65+
... \-\-auth
66+
... ${gcpAuthCfg}
67+
... \-\-registry
68+
... { "url": "file://${REPOSITORY_ROOT}/test/registry", "localDocRoot": "${REPOSITORY_ROOT}/test/registry", "verifyConfig": { "nopVerify": true } }
69+
... exec
70+
... ${bucketsListQuery}
71+
... cwd=${REPOSITORY_ROOT}
72+
... stdout=${CURDIR}/tmp/ID-Fed-Google-Buckets-List.tmp
73+
... stderr=${CURDIR}/tmp/ID-Fed-Google-Buckets-List-stderr.tmp
74+
Should Be Equal As Integers ${result.rc} 0
75+
Should Be Empty ${result.stderr}
76+
Should Contain ${result.stdout} stackql\-demo\-bucket

‎test/robot/id-fed-traffic-lights/tmp/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*
2+
!.gitignore

‎test/robot/reports-id-fed-traffic-lights/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*
2+
!.gitignore

0 commit comments

Comments
 (0)