Skip to content

Commit 4256aab

Browse files
LucaCinquiniLucaCinquini
authored
Merge pull request #249 from unity-sds/246-removing-jpl-internal-albs
Commenting jpl-internal ingress
2 parents e4cc4e4 + 1f8778a commit 4256aab

File tree

9 files changed

+105
-29
lines changed

9 files changed

+105
-29
lines changed

‎terraform-unity/modules/terraform-unity-sps-airflow/README.md‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,6 @@ No modules.
4242
| [aws_s3_bucket.airflow_logs](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/s3_bucket) | resource |
4343
| [aws_s3_bucket_policy.airflow_logs_s3_policy](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/s3_bucket_policy) | resource |
4444
| [aws_security_group.airflow_efs](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/security_group) | resource |
45-
| [aws_security_group.airflow_ingress_sg](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/security_group) | resource |
4645
| [aws_security_group.airflow_ingress_sg_internal](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/security_group) | resource |
4746
| [aws_security_group_rule.airflow_efs](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/security_group_rule) | resource |
4847
| [aws_ssm_parameter.airflow_api_health_check_endpoint](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
@@ -51,11 +50,9 @@ No modules.
5150
| [aws_ssm_parameter.airflow_ui_health_check_endpoint](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
5251
| [aws_ssm_parameter.airflow_ui_url](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
5352
| [aws_ssm_parameter.unity_proxy_airflow_ui](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
54-
| [aws_vpc_security_group_ingress_rule.airflow_ingress_sg_jpl_rule](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/vpc_security_group_ingress_rule) | resource |
5553
| [aws_vpc_security_group_ingress_rule.airflow_ingress_sg_proxy_rule](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/vpc_security_group_ingress_rule) | resource |
5654
| [helm_release.airflow](https://registry.terraform.io/providers/hashicorp/helm/2.15.0/docs/resources/release) | resource |
5755
| [helm_release.keda](https://registry.terraform.io/providers/hashicorp/helm/2.15.0/docs/resources/release) | resource |
58-
| [kubernetes_ingress_v1.airflow_ingress](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/ingress_v1) | resource |
5956
| [kubernetes_ingress_v1.airflow_ingress_internal](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/ingress_v1) | resource |
6057
| [kubernetes_namespace.keda](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/namespace) | resource |
6158
| [kubernetes_persistent_volume.airflow_deployed_dags](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/persistent_volume) | resource |
@@ -69,7 +66,6 @@ No modules.
6966
| [kubernetes_storage_class.efs](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/storage_class) | resource |
7067
| [null_resource.remove_keda_finalizers](https://registry.terraform.io/providers/hashicorp/null/3.2.3/docs/resources/resource) | resource |
7168
| [random_id.airflow_webserver_secret](https://registry.terraform.io/providers/hashicorp/random/3.6.1/docs/resources/id) | resource |
72-
| [time_sleep.wait_after_ssm](https://registry.terraform.io/providers/hashicorp/time/0.12.1/docs/resources/sleep) | resource |
7369
| [time_sleep.wait_for_efs_mount_target_dns_propagation](https://registry.terraform.io/providers/hashicorp/time/0.12.1/docs/resources/sleep) | resource |
7470
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/caller_identity) | data source |
7571
| [aws_db_instance.db](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/db_instance) | data source |
@@ -78,10 +74,12 @@ No modules.
7874
| [aws_lambda_functions.lambda_check_all](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/lambda_functions) | data source |
7975
| [aws_secretsmanager_secret_version.db](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/secretsmanager_secret_version) | data source |
8076
| [aws_security_groups.venue_proxy_sg](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/security_groups) | data source |
81-
| [aws_ssm_parameter.ssl_cert_arn](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
77+
| [aws_ssm_parameter.shared_services_account](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
78+
| [aws_ssm_parameter.shared_services_domain](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
79+
| [aws_ssm_parameter.shared_services_region](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
8280
| [aws_ssm_parameter.subnet_ids](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
81+
| [aws_ssm_parameter.venue_proxy_baseurl](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
8382
| [aws_vpc.cluster_vpc](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/vpc) | data source |
84-
| [kubernetes_ingress_v1.airflow_ingress](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/data-sources/ingress_v1) | data source |
8583
| [kubernetes_ingress_v1.airflow_ingress_internal](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/data-sources/ingress_v1) | data source |
8684
| [kubernetes_namespace.service_area](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/data-sources/namespace) | data source |
8785

@@ -110,5 +108,6 @@ No modules.
110108
|------|-------------|
111109
| <a name="output_airflow_deployed_dags_pvc"></a> [airflow\_deployed\_dags\_pvc](#output\_airflow\_deployed\_dags\_pvc) | n/a |
112110
| <a name="output_airflow_urls"></a> [airflow\_urls](#output\_airflow\_urls) | SSM parameter IDs and URLs for the various Airflow endpoints. |
111+
| <a name="output_airflow_venue_urls"></a> [airflow\_venue\_urls](#output\_airflow\_venue\_urls) | URLs for the various Airflow endpoints at venue-proxy level. |
113112
| <a name="output_s3_buckets"></a> [s3\_buckets](#output\_s3\_buckets) | SSM parameter IDs and bucket names for the various buckets used in the pipeline. |
114113
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎terraform-unity/modules/terraform-unity-sps-airflow/data.tf‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,13 @@ data "kubernetes_namespace" "service_area" {
1818
}
1919
}
2020

21+
/* Note: re-enable this to allow access via the JPL network
2122
data "kubernetes_ingress_v1" "airflow_ingress" {
2223
metadata {
2324
name = kubernetes_ingress_v1.airflow_ingress.metadata[0].name
2425
namespace = data.kubernetes_namespace.service_area.metadata[0].name
2526
}
26-
}
27+
}*/
2728

2829
data "kubernetes_ingress_v1" "airflow_ingress_internal" {
2930
metadata {
@@ -44,6 +45,23 @@ data "aws_efs_file_system" "efs" {
4445
file_system_id = var.efs_file_system_id
4546
}
4647

48+
/* Note: re-enable this to allow access via the JPL network
4749
data "aws_ssm_parameter" "ssl_cert_arn" {
4850
name = "/unity/account/network/ssl"
51+
}*/
52+
53+
data "aws_ssm_parameter" "shared_services_account" {
54+
name = "/unity/shared-services/aws/account"
55+
}
56+
57+
data "aws_ssm_parameter" "shared_services_region" {
58+
name = "/unity/shared-services/aws/account/region"
59+
}
60+
61+
data "aws_ssm_parameter" "shared_services_domain" {
62+
name = "arn:aws:ssm:${data.aws_ssm_parameter.shared_services_region.value}:${data.aws_ssm_parameter.shared_services_account.value}:parameter/unity/shared-services/domain"
63+
}
64+
65+
data "aws_ssm_parameter" "venue_proxy_baseurl" {
66+
name = "/unity/${var.project}/${var.venue}/management/httpd/loadbalancer-url"
4967
}

‎terraform-unity/modules/terraform-unity-sps-airflow/main.tf‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -430,6 +430,7 @@ resource "helm_release" "airflow" {
430430
]
431431
}
432432

433+
/* Note: re-enable this to allow access via the JPL network
433434
resource "aws_security_group" "airflow_ingress_sg" {
434435
name = "${var.project}-${var.venue}-airflow-ingress-sg"
435436
description = "SecurityGroup for Airflow LoadBalancer ingress"
@@ -439,7 +440,7 @@ resource "aws_security_group" "airflow_ingress_sg" {
439440
Component = "airflow"
440441
Stack = "airflow"
441442
})
442-
}
443+
}*/
443444

444445
resource "aws_security_group" "airflow_ingress_sg_internal" {
445446
name = "${var.project}-${var.venue}-airflow-internal-ingress-sg"
@@ -452,6 +453,7 @@ resource "aws_security_group" "airflow_ingress_sg_internal" {
452453
})
453454
}
454455

456+
/* Note: re-enable this to allow access via the JPL network
455457
#tfsec:ignore:AVD-AWS-0107
456458
resource "aws_vpc_security_group_ingress_rule" "airflow_ingress_sg_jpl_rule" {
457459
for_each = toset(["128.149.0.0/16", "137.78.0.0/16", "137.79.0.0/16"])
@@ -461,7 +463,7 @@ resource "aws_vpc_security_group_ingress_rule" "airflow_ingress_sg_jpl_rule" {
461463
from_port = local.load_balancer_port
462464
to_port = local.load_balancer_port
463465
cidr_ipv4 = each.key
464-
}
466+
}*/
465467

466468
data "aws_security_groups" "venue_proxy_sg" {
467469
filter {
@@ -484,6 +486,7 @@ resource "aws_vpc_security_group_ingress_rule" "airflow_ingress_sg_proxy_rule" {
484486
referenced_security_group_id = data.aws_security_groups.venue_proxy_sg.ids[0]
485487
}
486488

489+
/* Note: re-enable this to allow access via the JPL network
487490
resource "kubernetes_ingress_v1" "airflow_ingress" {
488491
metadata {
489492
name = "airflow-ingress"
@@ -521,7 +524,7 @@ resource "kubernetes_ingress_v1" "airflow_ingress" {
521524
}
522525
wait_for_load_balancer = true
523526
depends_on = [helm_release.airflow]
524-
}
527+
}*/
525528

526529
resource "kubernetes_ingress_v1" "airflow_ingress_internal" {
527530
metadata {
@@ -564,12 +567,13 @@ resource "aws_ssm_parameter" "airflow_ui_url" {
564567
name = format("/%s", join("/", compact(["", var.project, var.venue, var.service_area, "processing", "airflow", "ui_url"])))
565568
description = "The URL of the Airflow UI."
566569
type = "String"
567-
value = "https://${data.kubernetes_ingress_v1.airflow_ingress.status[0].load_balancer[0].ingress[0].hostname}:5000"
570+
value = "https://www.${data.aws_ssm_parameter.shared_services_domain.value}:4443/${var.project}/${var.venue}/sps/"
568571
tags = merge(local.common_tags, {
569572
Name = format(local.resource_name_prefix, "endpoints-airflow_ui")
570573
Component = "SSM"
571574
Stack = "SSM"
572575
})
576+
depends_on = [aws_ssm_parameter.unity_proxy_airflow_ui]
573577
}
574578

575579
resource "aws_ssm_parameter" "airflow_ui_health_check_endpoint" {
@@ -578,8 +582,8 @@ resource "aws_ssm_parameter" "airflow_ui_health_check_endpoint" {
578582
type = "String"
579583
value = jsonencode({
580584
"componentName" : "Airflow UI"
581-
"healthCheckUrl" : "http://${data.kubernetes_ingress_v1.airflow_ingress_internal.status[0].load_balancer[0].ingress[0].hostname}:5000/health"
582-
"landingPageUrl" : "http://${data.kubernetes_ingress_v1.airflow_ingress_internal.status[0].load_balancer[0].ingress[0].hostname}:5000"
585+
"healthCheckUrl" : "https://www.${data.aws_ssm_parameter.shared_services_domain.value}:4443/${var.project}/${var.venue}/sps/health"
586+
"landingPageUrl" : "https://www.${data.aws_ssm_parameter.shared_services_domain.value}:4443/${var.project}/${var.venue}/sps/"
583587
})
584588
tags = merge(local.common_tags, {
585589
Name = format(local.resource_name_prefix, "health-check-endpoints-airflow_ui")
@@ -589,18 +593,20 @@ resource "aws_ssm_parameter" "airflow_ui_health_check_endpoint" {
589593
lifecycle {
590594
ignore_changes = [value]
591595
}
596+
depends_on = [aws_ssm_parameter.unity_proxy_airflow_ui]
592597
}
593598

594599
resource "aws_ssm_parameter" "airflow_api_url" {
595600
name = format("/%s", join("/", compact(["", var.project, var.venue, var.service_area, "processing", "airflow", "api_url"])))
596601
description = "The URL of the Airflow REST API."
597602
type = "String"
598-
value = "https://${data.kubernetes_ingress_v1.airflow_ingress.status[0].load_balancer[0].ingress[0].hostname}:5000/api/v1"
603+
value = "https://www.${data.aws_ssm_parameter.shared_services_domain.value}:4443/${var.project}/${var.venue}/sps/api/v1"
599604
tags = merge(local.common_tags, {
600605
Name = format(local.resource_name_prefix, "endpoints-airflow_api")
601606
Component = "SSM"
602607
Stack = "SSM"
603608
})
609+
depends_on = [aws_ssm_parameter.unity_proxy_airflow_ui]
604610
}
605611

606612
resource "aws_ssm_parameter" "airflow_api_health_check_endpoint" {
@@ -609,8 +615,8 @@ resource "aws_ssm_parameter" "airflow_api_health_check_endpoint" {
609615
type = "String"
610616
value = jsonencode({
611617
"componentName" : "Airflow API"
612-
"healthCheckUrl" : "http://${data.kubernetes_ingress_v1.airflow_ingress_internal.status[0].load_balancer[0].ingress[0].hostname}:5000/api/v1/health"
613-
"landingPageUrl" : "http://${data.kubernetes_ingress_v1.airflow_ingress_internal.status[0].load_balancer[0].ingress[0].hostname}:5000/api/v1"
618+
"healthCheckUrl" : "https://www.${data.aws_ssm_parameter.shared_services_domain.value}:4443/${var.project}/${var.venue}/sps/api/v1/health"
619+
"landingPageUrl" : "https://www.${data.aws_ssm_parameter.shared_services_domain.value}:4443/${var.project}/${var.venue}/sps/api/v1"
614620
})
615621
tags = merge(local.common_tags, {
616622
Name = format(local.resource_name_prefix, "health-check-endpoints-airflow_api")
@@ -620,6 +626,7 @@ resource "aws_ssm_parameter" "airflow_api_health_check_endpoint" {
620626
lifecycle {
621627
ignore_changes = [value]
622628
}
629+
depends_on = [aws_ssm_parameter.unity_proxy_airflow_ui]
623630
}
624631

625632
resource "aws_ssm_parameter" "unity_proxy_airflow_ui" {

‎terraform-unity/modules/terraform-unity-sps-airflow/outputs.tf‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,18 @@ output "airflow_urls" {
1212
}
1313
}
1414

15+
output "airflow_venue_urls" {
16+
description = "URLs for the various Airflow endpoints at venue-proxy level."
17+
value = {
18+
"ui" = {
19+
"url" = nonsensitive(replace(data.aws_ssm_parameter.venue_proxy_baseurl.value, "management/ui", "sps/"))
20+
}
21+
"rest_api" = {
22+
"url" = nonsensitive(replace(data.aws_ssm_parameter.venue_proxy_baseurl.value, "management/ui", "sps/api/v1"))
23+
}
24+
}
25+
}
26+
1527
output "s3_buckets" {
1628
description = "SSM parameter IDs and bucket names for the various buckets used in the pipeline."
1729
value = {

‎terraform-unity/modules/terraform-unity-sps-ogc-processes-api/README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,17 +23,14 @@ No modules.
2323
| Name | Type |
2424
|------|------|
2525
| [aws_lambda_invocation.unity_proxy_lambda_invocation](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/lambda_invocation) | resource |
26-
| [aws_security_group.ogc_ingress_sg](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/security_group) | resource |
2726
| [aws_security_group.ogc_ingress_sg_internal](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/security_group) | resource |
2827
| [aws_ssm_parameter.ogc_processes_api_health_check_endpoint](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
2928
| [aws_ssm_parameter.ogc_processes_api_url](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
3029
| [aws_ssm_parameter.ogc_processes_ui_url](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
3130
| [aws_ssm_parameter.unity_proxy_ogc_api](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/ssm_parameter) | resource |
32-
| [aws_vpc_security_group_ingress_rule.ogc_ingress_sg_jpl_rule](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/vpc_security_group_ingress_rule) | resource |
3331
| [aws_vpc_security_group_ingress_rule.ogc_ingress_sg_proxy_rule](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/resources/vpc_security_group_ingress_rule) | resource |
3432
| [kubernetes_deployment.ogc_processes_api](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/deployment) | resource |
3533
| [kubernetes_deployment.redis](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/deployment) | resource |
36-
| [kubernetes_ingress_v1.ogc_processes_api_ingress](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/ingress_v1) | resource |
3734
| [kubernetes_ingress_v1.ogc_processes_api_ingress_internal](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/ingress_v1) | resource |
3835
| [kubernetes_service.ogc_processes_api](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/service) | resource |
3936
| [kubernetes_service.redis](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/resources/service) | resource |
@@ -42,10 +39,12 @@ No modules.
4239
| [aws_lambda_functions.lambda_check_all](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/lambda_functions) | data source |
4340
| [aws_secretsmanager_secret_version.db](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/secretsmanager_secret_version) | data source |
4441
| [aws_security_groups.venue_proxy_sg](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/security_groups) | data source |
45-
| [aws_ssm_parameter.ssl_cert_arn](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
42+
| [aws_ssm_parameter.shared_services_account](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
43+
| [aws_ssm_parameter.shared_services_domain](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
44+
| [aws_ssm_parameter.shared_services_region](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
4645
| [aws_ssm_parameter.subnet_ids](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
46+
| [aws_ssm_parameter.venue_proxy_baseurl](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/ssm_parameter) | data source |
4747
| [aws_vpc.cluster_vpc](https://registry.terraform.io/providers/hashicorp/aws/5.67.0/docs/data-sources/vpc) | data source |
48-
| [kubernetes_ingress_v1.ogc_processes_api_ingress](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/data-sources/ingress_v1) | data source |
4948
| [kubernetes_ingress_v1.ogc_processes_api_ingress_internal](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/data-sources/ingress_v1) | data source |
5049
| [kubernetes_namespace.service_area](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/data-sources/namespace) | data source |
5150
| [kubernetes_persistent_volume_claim.airflow_deployed_dags](https://registry.terraform.io/providers/hashicorp/kubernetes/2.32.0/docs/data-sources/persistent_volume_claim) | data source |
@@ -73,4 +72,5 @@ No modules.
7372
| Name | Description |
7473
|------|-------------|
7574
| <a name="output_ogc_processes_urls"></a> [ogc\_processes\_urls](#output\_ogc\_processes\_urls) | SSM parameter IDs and URLs for the various OGC Processes endpoints. |
75+
| <a name="output_ogc_processes_venue_urls"></a> [ogc\_processes\_venue\_urls](#output\_ogc\_processes\_venue\_urls) | URLs for the various OGC Processes endpoints at venue-proxy level. |
7676
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

0 commit comments

Comments
 (0)