Use PubSub JWT auth for security instead of the hardcoded static key

Author: JoshuaFox

README.md

@@ -83,21 +83,29 @@ names start `_gcp_`. The part of the function name after `_gcp_` is used for the
 
 ## Before deploying
 
-### Org and IAM
+### In which Project 
 
 * You can deploy Iris in any project within your Google Cloud organization, but we recommend using a
   [new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project).
 
+### Needed roles for deployment
+#### Organization-leve roles
+
 * Here are the required organization-level roles for you, the deployer, to allow the deploy script to set up roles and  log sink. (Note that *Organization Owner* is not enough).
     * *Organization Role Administrator*  so the deployment script can create a custom IAM role for Iris that allows it to  get and set labels.
     * *Security Admin* so the deployment script can grant the needed role bindings, e.g., to the App Engine service account.
     * *Logs Configuration Writer* so the deployment script can create an organization log sink that sends logs to
       PubSub.
 
-* The required project-level roles: *Project Owner* or *Project Editor* on the project where Iris is deployed, so that the deployment script can create role bindings, topics and subscriptions, and deploy App Engine. Fine-granted "predefined roles" are not possible because deploying Cloud Scheduler cron requires at least Editor or Owner, per GCP docs.
+#### Project-level roles
+* The required project-level roles: *Project Owner* or *Project Editor* on the project where Iris is deployed, so that the deployment script can 
+  * create role bindings, topics and subscriptions
+  * deploy App Engine. 
+  * `actAs` the serivice account `iris-msg-sender` for deploying it to allow JWT auth.
 
-### App Engine Defaults
+* Fine-granted "predefined roles" are not possible because deploying Cloud Scheduler cron requires at least Editor or Owner, per GCP docs.
 
+### App Engine Defaults
 
 ## Deployment
 
@@ -159,6 +167,7 @@ names start `_gcp_`. The part of the function name after `_gcp_` is used for the
     * Another topic is a dead-letter topic.
 * PubSub subscriptions
     * There is one for each topic: These direct the messages to `/label_one` and `/do_label` in `main.py`, respectively.
+    * For security, these two PubSub subscriptions [use JWT auth](These https://cloud.google.com/pubsub/docs/authenticate-push-subscriptions). The deployment script sets this up for you.
     * A dead-letter subscription. This is a pull subscription. By default, it just accumulates the messages. You can use      it to see statistics, or you can pull messages from it.
 
 # Local Development

TODO.md

@@ -5,21 +5,10 @@
 * P2 Memory consumption: Even an empty AppEngine app (not Iris, just a Hello World with 3 lines of code in total) crashes on out-of-memory for the smalled AppEngine instance. Google has confirmed this. See if there is a workaround.  This will save money.
 
 * P2 PubSub push endpoint security:
-  Note: The token by itself is not very secure, though  
-  Google has at times recommended this approach.
-
-  Alternatives:
-    - Replace the `PUBSUB_VERIFICATION_TOKEN` with random value in `deploy.sh`
-    - Or better: [Use JWT](https://cloud.google.com/pubsub/docs/push)
+  Note: The token by itself is not very secure, though   Google has at times recommended this approach.
 
-* P3 In `integration_test.sh`
-    - Test more labels (in addition to `iris3_name` which is now tested)
-    - Test the copying of labels from the project.
-    - Support testing of the cron-based labeling, which would also allow testing of Cloud SQL and of attachment of
-      Disks. In this test:
-        1. Modify cron to run 1 minute after the deployment is launched (and restore it at the end of the test.)
-        1. Call `deploy.sh` using with the `-c` switch to disable event-based labeling 1. Wait 1.5 minutes after deploy
-           before checking that the labels are there
+  [Use JWT](https://cloud.google.com/pubsub/docs/push)
+
 
 * P3 Label immediately after an event in certain cases, as opposed to using a daily cron as is now done.
     * Cloud SQL Instances
@@ -41,7 +30,6 @@
 
 * P3 Rethink the need for title case in class names. This is clumsy for `Cloudsql`.
 
-
 * P4 Implement new labels, for example using ideas from
   the [GCP Auto Tag project](https://github.com/doitintl/gcp-auto-tag/)
   But **don't add too many**: There are *a lot* of fields on resources.

config.yaml.original

@@ -51,15 +51,3 @@ from_project: True
 
 label_all_on_cron: False
 
-# Optionally change this token before first deployment for added security in
-# communication between PubSub and the Iris App on App Engine.
-# You could even re-generate a new token per deployment.
-# Note that this token-based approach is not very secure, though it  was once recommended by Google.
-# However, so long as the GCP project running the Iris3 AppEngine service is otherwise secure,
-# setting this token protects against unwanted invocations of labeling.
-# If you then redeploy, don't change the token, as it is used for
-# communication between different parts of the system.
-pubsub_verification_token: 0b0a30cde7e3489f0a9cd74bb51c514d
-
-
-

config.yaml.test.template

@@ -11,5 +11,4 @@ specific_prefixes:
   Buckets: gcs${RUN_ID}
 from_project: True
 label_all_on_cron: False
-# We generate a random token per-test-run as an additional isolation measure
-pubsub_verification_token: ${PUBSUB_TEST_TOKEN}
+

main.py

@@ -2,14 +2,21 @@
 
 import sys
 
-from flask import Response
+print("Initializing ", file=sys.stderr)
+from util.utils import init_logging, sort_dict
 
+# Must init logging before any library code writes logs (which would then just override our config)
+init_logging()
 
-print("Initializing ", file=sys.stderr)
 import flask
+from flask import Response
+from google.auth.transport import requests
+from google.oauth2 import id_token
 from util.gcp.gcp_utils import (
     increment_invocation_count,
     count_invocations_by_path,
+    get_project,
+    current_project_id,
 )
 from util.gcp.detect_gae import detect_gae
 
@@ -19,11 +26,6 @@
 
     app.wsgi_app = google.appengine.api.wrap_wsgi_app(app.wsgi_app)
 
-from util.utils import init_logging, sort_dict
-
-# Must init logging before any library code writes logs (which would then just override our config)
-init_logging()
-
 from functools import lru_cache
 
 from typing import Dict, Type
@@ -49,7 +51,6 @@
 
 from util.config_utils import (
     is_project_enabled,
-    pubsub_token,
     iris_homepage_text,
 )
 from util.utils import log_time, timing
@@ -96,7 +97,7 @@ def schedule():
     """
     Send out a message per-plugin per-project to label all objects of that type and project.
     """
-
+    # Not validated with JWT because validated with cron header (see below)
     increment_invocation_count("schedule")
     with gae_memory_logging("schedule"):
         try:
@@ -172,6 +173,11 @@ def __send_pubsub_per_projectplugin(configured_projects):
 
 @app.route("/label_one", methods=["POST"])
 def label_one():
+    """Message received from PubSub when the log sink detects a new resource"""
+    ok = __check_pubsub_jwt()
+    if not ok:
+        return "JWT Failed", 400
+
     increment_invocation_count("label_one")
     with gae_memory_logging("label_one"):
 
@@ -226,6 +232,37 @@ def label_one():
             return "Error", 500
 
 
+def __check_pubsub_jwt():
+    try:
+        bearer_token = flask.request.headers.get("Authorization")
+        token = bearer_token.split(" ")[1]
+
+        claim = id_token.verify_oauth2_token(token, requests.Request())
+
+        # example claim:  { "aud": "https://iris3-dot-myproj.appspot.com/label_one?token=xxxxxxxxx",
+        #     "azp": "1125239305332910191520",
+        #     "email": "[email protected]",
+        #     "email_verified": True, "exp": 1708281691, "iat": 1708278091,
+        #     "iss": "https://accounts.google.com", "sub": "1125239305332910191520"}
+
+        if not (is_email_verif := claim.get("email_verified")):
+            logging.error(f"Email verified was {is_email_verif}")
+            return False
+
+        logging.info("Claims: " + str(claim))
+        if (
+            email := claim.get("email")
+        ) != f"iris-msg-sender@{current_project_id()}.iam.gserviceaccount.com":
+            logging.error("incorrect email in JWT " + email)
+            return False
+    except Exception as e:
+        logging.exception(f"Invalid JWT token: {e}")
+        return False
+    logging.info("JWT Passed")
+
+    return True
+
+
 def __label_one_0(data, plugin_cls: Type[Plugin]):
     plugin = PluginHolder.get_plugin_instance(plugin_cls)
     gcp_object = plugin.get_gcp_object(data)
@@ -259,7 +296,6 @@ def __label_one_0(data, plugin_cls: Type[Plugin]):
 def __extract_pubsub_content() -> Dict:
     """Take the value at the relevant key in the logging message from PubSub,
     Base64-decode, convert to Python object."""
-    __check_pubsub_verification_token()
 
     envelope = flask.request.get_json()
     msg = envelope.get("message", {})
@@ -283,6 +319,10 @@ def do_label():
     """Receives a push message from PubSub, sent from schedule() above,
     and labels all objects of a given plugin and project_id.
     """
+
+    ok = __check_pubsub_jwt()
+    if not ok:
+        return "JWT Failed", 400
     increment_invocation_count("do_label")
     with gae_memory_logging("do_label"):
 
@@ -319,21 +359,6 @@ def do_label():
             return "Error", 500
 
 
-def __check_pubsub_verification_token():
-    """Token verifying that only PubSub accesses PubSub push endpoints"""
-    expected_token = pubsub_token()
-    if not expected_token:
-        raise FlaskException(
-            "Should define expected token in configuration.",
-            400,
-        )
-
-    token_from_args = flask.request.args.get("token", "")
-    if expected_token != token_from_args:
-        logging.info("Token was %s but expected %s", token_from_args, expected_token)
-        raise FlaskException(f'Access denied: Invalid token "{expected_token}"', 403)
-
-
 class FlaskException(Exception):
     status_code = 400
 

scripts/_deploy-project.sh

@@ -38,11 +38,9 @@ fi
 
 gae_svc=$(grep "service:" app.yaml | awk '{print $2}')
 
-# The following line depends on the  the export PYTHON_PATH="." above.
-PUBSUB_VERIFICATION_TOKEN=$(python3 ./util/print_pubsub_token.py)
 
-LABEL_ONE_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/label_one?token=${PUBSUB_VERIFICATION_TOKEN}"
-DO_LABEL_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/do_label?token=${PUBSUB_VERIFICATION_TOKEN}"
+LABEL_ONE_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/label_one"
+DO_LABEL_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/do_label?token"
 
 declare -A enabled_services
 while read -r svc _; do
@@ -102,6 +100,16 @@ fi
 project_number=$(gcloud projects describe $PROJECT_ID --format json|jq -r '.projectNumber')
 PUBSUB_SERVICE_ACCOUNT="service-${project_number}@gcp-sa-pubsub.iam.gserviceaccount.com"
 
+msg_sender_sa_name=iris-msg-sender
+
+gcloud iam service-accounts create --project $PROJECT_ID $msg_sender_sa_name ||true
+
+MSGSENDER_SERVICE_ACCOUNT=${msg_sender_sa_name}@${PROJECT_ID}.iam.gserviceaccount.com
+
+gcloud projects add-iam-policy-binding ${PROJECT_ID} \
+ --member="serviceAccount:${MSGSENDER_SERVICE_ACCOUNT}"\
+ --role='roles/iam.serviceAccountTokenCreator'
+
 set +e
 # Allow Pubsub to publish into the deadletter topic
 BINDING_ERR_OUTPUT=$(gcloud pubsub topics add-iam-policy-binding $DEADLETTER_TOPIC \
@@ -136,6 +144,7 @@ if [[ $? -eq 0 ]]; then
   gcloud pubsub subscriptions update "$DO_LABEL_SUBSCRIPTION" \
     --project="$PROJECT_ID" \
     --push-endpoint "$DO_LABEL_SUBSCRIPTION_ENDPOINT" \
+    --push-auth-service-account $MSGSENDER_SERVICE_ACCOUNT  \
     --ack-deadline=$ACK_DEADLINE \
     --max-delivery-attempts=$MAX_DELIVERY_ATTEMPTS \
     --dead-letter-topic=$DEADLETTER_TOPIC \
@@ -147,6 +156,7 @@ else
   gcloud pubsub subscriptions create "$DO_LABEL_SUBSCRIPTION" \
     --topic "$SCHEDULELABELING_TOPIC" --project="$PROJECT_ID" \
     --push-endpoint "$DO_LABEL_SUBSCRIPTION_ENDPOINT" \
+    --push-auth-service-account $MSGSENDER_SERVICE_ACCOUNT  \
     --ack-deadline=$ACK_DEADLINE \
     --max-delivery-attempts=$MAX_DELIVERY_ATTEMPTS \
     --dead-letter-topic=$DEADLETTER_TOPIC \
@@ -179,6 +189,7 @@ else
       echo >&2 "Updating $LABEL_ONE_SUBSCRIPTION"
       gcloud pubsub subscriptions update "$LABEL_ONE_SUBSCRIPTION" --project="$PROJECT_ID" \
         --push-endpoint="$LABEL_ONE_SUBSCRIPTION_ENDPOINT" \
+        --push-auth-service-account $MSGSENDER_SERVICE_ACCOUNT  \
         --ack-deadline=$ACK_DEADLINE \
         --max-delivery-attempts=$MAX_DELIVERY_ATTEMPTS \
         --dead-letter-topic=$DEADLETTER_TOPIC \
@@ -188,6 +199,7 @@ else
   else
       gcloud pubsub subscriptions create "$LABEL_ONE_SUBSCRIPTION" --topic "$LOGS_TOPIC" --project="$PROJECT_ID" \
         --push-endpoint="$LABEL_ONE_SUBSCRIPTION_ENDPOINT" \
+        --push-auth-service-account $MSGSENDER_SERVICE_ACCOUNT  \
         --ack-deadline=$ACK_DEADLINE \
         --max-delivery-attempts=$MAX_DELIVERY_ATTEMPTS \
         --dead-letter-topic=$DEADLETTER_TOPIC \

test_scripts/utils_for_tests.py

@@ -10,8 +10,6 @@
 from urllib.error import URLError
 from urllib.parse import urlencode
 
-from util.config_utils import pubsub_token
-
 logging.basicConfig(format="%(levelname)s: %(message)s", level=logging.INFO)
 logging.getLogger("googleapiclient.discovery_cache").setLevel(logging.ERROR)
 
@@ -42,9 +40,8 @@ def do_local_http(
         host_and_port = f"localhost:{LOCAL_PORT}"
         args_s = ""
         if extra_args:
-            args_s = "&"
             args_s += urlencode(extra_args)
-        url_ = f"http://{host_and_port}/{path}?token={pubsub_token()}{args_s}"
+        url_ = f"http://{host_and_port}/{path}?{args_s}"
         req = request.Request(
             url_,
             data=data_bytes,

uninstall_scripts/_uninstall-for-project.sh

@@ -16,6 +16,8 @@ LABEL_ONE_SUBSCRIPTION=label_one
 
 project_number=$(gcloud projects describe $PROJECT_ID --format json|jq -r '.projectNumber')
 PUBSUB_SERVICE_ACCOUNT="service-${project_number}@gcp-sa-pubsub.iam.gserviceaccount.com"
+msg_sender_sa_name=iris-msg-sender
+MSGSENDER_SERVICE_ACCOUNT=${msg_sender_sa_name}@${PROJECT_ID}.iam.gserviceaccount.com
 
 gcloud pubsub topics remove-iam-policy-binding $DEADLETTER_TOPIC \
         --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
@@ -29,6 +31,11 @@ gcloud pubsub subscriptions remove-iam-policy-binding $LABEL_ONE_SUBSCRIPTION \
       --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
       --role="roles/pubsub.subscriber" --project $PROJECT_ID ||true
 
+gcloud projects remove-iam-policy-binding --project ${PROJECT_ID} \
+ --member="serviceAccount:${MSGSENDER_SERVICE_ACCOUNT}"\
+ --role='roles/iam.serviceAccountTokenCreator'
+
+
 gcloud pubsub subscriptions delete $DEADLETTER_SUB --project="$PROJECT_ID" -q || true
 gcloud pubsub subscriptions delete "$DO_LABEL_SUBSCRIPTION" -q --project="$PROJECT_ID" ||true
 gcloud pubsub subscriptions delete "$LABEL_ONE_SUBSCRIPTION" --project="$PROJECT_ID" 2>/dev/null || true

util/config_utils.py

@@ -61,19 +61,6 @@ def label_all_on_cron() -> bool:
     return ret
 
 
-def pubsub_token() -> str:
-    config = get_config()
-    ret = config.get("pubsub_verification_token")
-
-    return ret
-
-
-def get_config_redact_token():
-    c = get_config().copy()
-    c["pubsub_verification_token"] = "[REDACTED]"
-    return c
-
-
 @functools.lru_cache
 def get_config() -> typing.Dict:
     test_config = "config-test.yaml"