Provision keys safely

[phbnf]

Currently, both the AWS and GCP secret manager implementation store unencrypted private keys in the Terraform state file. This is ok for test logs, but we need a different solution for production logs.

[roger2hk]

Note that AWS/GCP Secret(s) Manager has already encrypted the secrets with the corrsponding KMS key(s). The insecure part is the hashicorp/tls resource which is primarily intended for easily bootstrapping throwaway development environments.

Extracting the tls_private_key into its own module and naming the module as "unsafe/insecure" could greatly avoid the mis-use of hashicorp/tls resource.

[phbnf]

Yes, 100%. I'm thinking as well that we could carve out the module that creates the private key, and provide instructions about how to deploy this module safely and smoothly to have a secure new log creation customer journey.

[phbnf]

@haydentherapper gave a shot at it for GCP in #282 using Tink. Let's consider this for production GCP logs.