fix: if unable to calculate local md5 hash use old value of detect_md5hash #12032

gurusai-voleti · 2024-10-17T08:58:34Z

if unable to calculate local md5 hash use old value of detect_md5hash
Fixes: hashicorp/terraform-provider-google#18618

Release Note Template for Downstream PRs (will be copied)

storage: if unable to calculate local md5 hash use old value of detect_md5hash

github-actions · 2024-10-17T08:59:14Z

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

@roaks3, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

modular-magician · 2024-10-17T09:10:31Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 1 file changed, 3 insertions(+))
google-beta provider: Diff ( 1 file changed, 3 insertions(+))

modular-magician · 2024-10-17T09:18:53Z

Tests analytics

Total tests: 109
Passed tests: 100
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

modular-magician · 2024-10-17T12:22:33Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 51 insertions(+))
google-beta provider: Diff ( 2 files changed, 51 insertions(+))

modular-magician · 2024-10-17T12:31:18Z

Tests analytics

Total tests: 110
Passed tests: 100
Skipped tests: 9
Affected tests: 1

Click here to see the affected service packages

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

Get to know how VCR tests work

modular-magician · 2024-10-17T12:31:55Z

🟢 Tests passed during RECORDING mode:
TestAccStorageObject_detect_md5hash_nofile[Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

🟢 All tests passed!

View the build log or the debug log for each test

kautikdk · 2024-10-17T13:26:58Z

Hi @gurusai-voleti, This change appears to be the risky fix as we are setting localMd5Hash = old in DiffSuppressFunc when the file is not present. This will prevent terraform diff generation for google_storage_bucket_object resource when the source file is removed. Eventually it will make unnecessary diff in the next terraform plan.
Setting Hash to previous one also prevents diff for file content change in-between two terraform apply if the files are removed locally and file content is changed in terraform config.
Let me know if I am missing something.

PS: Added test appears to be testing different behavior. I suggest, writing test steps and test configs by referring bug reproductions steps.
cc: @roaks3

gurusai-voleti · 2024-10-18T04:50:28Z

with this fix in terraform plan wont show any difference and in terraform apply it will create files and wont do anything else unless there are really infra changes to deploy, this is working as expected without this change also with out file delete.
added test case used the same example which is reported by customer

kautikdk · 2024-10-18T05:30:19Z

But if there is a resource dependency then Ideally customer expects a difference. If the source file is being removed, Ideally there should be some difference, Right?

gurusai-voleti · 2024-10-18T09:07:36Z

yes if the file is created through a resource block it will show up in the plan, only thing it will not display in the plan is md5 hash difference with this fix, which is the real problem of the issue between plan and apply, I can modify my test case more

kautikdk · 2024-10-18T11:45:26Z

Hi @roaks3, Is there any way to handle intermediate value change of attribute during terraform apply operation? I believe, The error is coming because of two different values of same attribute during one apply operation.

@gurusai-voleti can you modify PR description as per this guidelines:https://googlecloudplatform.github.io/magic-modules/contribute/create-pr/#requirements. Currently there is no link between this PR and the issue that it is going to fix.

modular-magician · 2024-10-21T06:53:59Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 51 insertions(+))
google-beta provider: Diff ( 2 files changed, 51 insertions(+))

modular-magician · 2024-10-21T07:02:19Z

Tests analytics

Total tests: 110
Passed tests: 101
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

modular-magician · 2024-10-22T05:20:31Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 51 insertions(+))
google-beta provider: Diff ( 2 files changed, 51 insertions(+))

modular-magician · 2024-10-22T05:28:49Z

Tests analytics

Total tests: 110
Passed tests: 101
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

github-actions · 2024-10-22T09:41:53Z

@roaks3 This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

roaks3 · 2024-10-22T21:19:07Z

Yea, this is relatively complicated. The detect_md5hash solution seems to fundamentally be at odds with Terraform paradigms (see https://developer.hashicorp.com/terraform/plugin/sdkv2/guides/terraform-0.12-compatibility#inaccurate-plans), in that it is explicitly communicating a planned value of different hash to the user, but then expects to change it to the actual hash later. I suspect this worked fine with ForceNew for the ~6 years it has been like this (although I'm not certain exactly why), but when it was changed to be mutable in #10038, I think we broke some assumptions and are now seeing this error.

Per the link, you might have luck working with CustomizeDiff to handle the detect_md5hash value properly, but even that could result in an unintended breaking change.

I would recommend narrowing in on the exact failure case, as it is odd that the source file can be removed and then apply works the first time, but then not on subsequent times. Hopefully there is a specific edge case that can be accounted for.

As mentioned by @kautikdk , I'm a bit concerned about the proposed solution in the PR being too broad, and impacting behavior that is working as intended.

modular-magician · 2024-10-23T11:42:31Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 51 insertions(+))
google-beta provider: Diff ( 2 files changed, 51 insertions(+))

modular-magician · 2024-10-23T11:51:44Z

Tests analytics

Total tests: 110
Passed tests: 101
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

roaks3

To follow up: I was able to confirm with the team the CustomizeDiff was either unavailable, or rarely used, when this functionality was originally introduced, so that is likely a path we can take to fix this issue and possibly simplify the behavior.

modular-magician · 2025-02-20T06:12:11Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))
google-beta provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))

modular-magician · 2025-02-20T06:18:12Z

Tests analytics

Total tests: 117
Passed tests: 108
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

wiktorn · 2025-02-20T06:47:46Z

mmv1/third_party/terraform/services/storage/resource_storage_bucket_object_test.go

+  name   = "%s"
+  bucket = google_storage_bucket.bucket.name
+  source = "%s"
+  source_md5hash = "%s"


Providing exact md5 hash value externally is not idiomatic in terraform. This should be calculated within terraform either by using md5(...), using local_file resource attributes or in a similar matter.

Using md5(....) will fail on the files that do change during the apply, so the best way is to use local_file and its content_md5 attribute. This ensures that in the plan stage value is passed as "known after apply" and properly calculated during apply

modular-magician · 2025-02-21T09:04:42Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))
google-beta provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))

modular-magician · 2025-02-21T09:10:37Z

Tests analytics

Total tests: 117
Passed tests: 108
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

modular-magician · 2025-02-24T10:59:24Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))
google-beta provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))

modular-magician · 2025-02-24T12:26:53Z

Tests analytics

Total tests: 119
Passed tests: 110
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

wiktorn · 2025-03-02T17:52:15Z

mmv1/third_party/terraform/services/storage/resource_storage_bucket_object_test.go

+	updatedName := getNewTmpTestFile(t, "tf-test")
+	updatedDataMd5 := writeFile(updatedName.Name(), []byte("datum"))


Change of the filename will serve itself as a signal to recreate the file.

Clue of the issue we trying to solve is, that contents of exciting file change. So I recommend just writing additional content to the same file, which is used in first apply.

wiktorn · 2025-03-02T17:53:05Z

mmv1/third_party/terraform/services/storage/resource_storage_bucket_object_test.go

-					}
-				},
-				Config: testGoogleStorageBucketsObjectBasic(bucketName, testFile.Name()),
+				Config: testGoogleStorageBucketsObjectFileMd5(bucketName, updatedName.Name()),


I'd love to have the same configuration used in both steps. The issue is not, that the user is changing the contents and hence the issue, but with the same terraform config, but changed contents of the files, the error occurs.

wiktorn · 2025-03-02T17:57:04Z

mmv1/third_party/terraform/services/storage/resource_storage_bucket_object_test.go

+resource "google_storage_bucket" "bucket" {
+  name     = "%s"
+  location = "US"
+}
+
+resource "google_storage_bucket_object" "object" {
+  name   = "%s"
+  bucket = google_storage_bucket.bucket.name
+  source = "%s"
+  source_md5hash = filemd5("%s")
+}
+`, bucketName, objectName, sourceFilename, sourceFilename)
+}


This case is good, as it covers the case of file that is managed outside of Terraform code (for example - uploading config file).

I recommend add additional case, that I mentioned in this comment - that is verifying behavior, when content of the file is generated by terraform as a part of apply stage.

Adding such cases will ensure, that this problem will not re-surface, and will give confidence, that issue is indeed fixed

modular-magician · 2025-03-03T10:33:09Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))
google-beta provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))

modular-magician · 2025-03-03T12:00:14Z

Tests analytics

Total tests: 119
Passed tests: 110
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

modular-magician · 2025-03-06T04:14:57Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))
google-beta provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))

modular-magician · 2025-03-06T05:41:32Z

Tests analytics

Total tests: 119
Passed tests: 110
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

modular-magician · 2025-03-10T04:07:55Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))
google-beta provider: Diff ( 2 files changed, 42 insertions(+), 35 deletions(-))

modular-magician · 2025-03-10T05:34:52Z

Tests analytics

Total tests: 119
Passed tests: 110
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

gurusai-voleti · 2025-03-10T14:59:20Z

mmv1/third_party/terraform/services/storage/resource_storage_bucket_object.go

+		d.SetNewComputed("md5hash")
+		d.SetNewComputed("generation")
+		// if any change in source_md5hash, override change of detect_md5hash
+		d.ForceNew("detect_md5hash")


@kautikdk if any change in source_md5hash, ignore the detect_md5hash diffsuppress func at terraform apply time

Why to set ForceNew only when there is a change in source_md5hash?
We should avoid detect_md5hash even when there is no change in source_md5hash. If user specifies this field then ignore detect_md5hash at all operations.

i verified few methods, but this is the only way to avoid the DiffSuppressFunc of detect_md5hash field

I don't think it is going to work because setting up it as ForceNew will assign "different hash" to the detect_md5hash attribute and it will replace object.

You can probably avoid DiffSuppressFunc by modify flattener of the field.
You can try something like this in resourceStorageBucketObjectRead,

if v, ok := d.GetOk("source_md5hash"); ok{ if err := d.Set("source_md5hash", v); err != nil { return fmt.Errorf("Error setting source_md5hash: %s", err) } if err := d.Set("detect_md5hash", d.Get("detect_md5hash")); err != nil { return fmt.Errorf("Error setting detect_md5hash: %s", err) } } else { if err := d.Set("detect_md5hash", res.Md5Hash); err != nil { return fmt.Errorf("Error setting detect_md5hash: %s", err) } }

This will keep existing value of detect_md5hash as it is if the source_md5hash is specified which should prevent the error. We can probably document this behavior as well that if the source_md5hash is specified then detect_md5hash can not be used and rely on it.
This is not fully tested approach but worth try!

I tested its not generating any inconsistent final plan, as forceNew is called in CustomizeDiff it acts as new resource, this completely overrides the diffsuppressfunc and assign the default value "different hash" and once create/update completed in read function detect_md5hash assigned with latest md5hash and same is stored in state file

+1 to this concern, especially if the google_storage_bucket_object is set to create_before_destroy through propagation from other resources. Then apply results in no file in the bucket. In-place updates are really the only way to avoid such problems / bugs.

yes agree this is recreating the object which is not a good approach as inplace updates are preferable

I don't think it is going to work because setting up it as ForceNew will assign "different hash" to the detect_md5hash attribute and it will replace object.

You can probably avoid DiffSuppressFunc by modify flattener of the field. You can try something like this in resourceStorageBucketObjectRead,

if v, ok := d.GetOk("source_md5hash"); ok{ if err := d.Set("source_md5hash", v); err != nil { return fmt.Errorf("Error setting source_md5hash: %s", err) } if err := d.Set("detect_md5hash", d.Get("detect_md5hash")); err != nil { return fmt.Errorf("Error setting detect_md5hash: %s", err) } } else { if err := d.Set("detect_md5hash", res.Md5Hash); err != nil { return fmt.Errorf("Error setting detect_md5hash: %s", err) } }

This will keep existing value of detect_md5hash as it is if the source_md5hash is specified which should prevent the error. We can probably document this behavior as well that if the source_md5hash is specified then detect_md5hash can not be used and rely on it. This is not fully tested approach but worth try!

You can try the suggestion I mentioned in this message. Probably we can avoid DiffSupress mechanism by setting detect_md5hash everytime with the previous value. It will only take effect when user specifies source_md5hash so it won't break any userflows and will handle introduction of source_md5hash as well so that we can gracefully deprecate and remove detect_md5hash field.

in CustomizeDiff there is no chance of setting the value, either setnewcomputed or forcenew is only possible it seems

tried the suggested solution in create/update still facing issue of inconsistent final plan, its not ignoring the diffsuppressfunc of detect_md5hash

Can you please push those changes as well so that we can take a look?

modular-magician · 2025-03-10T15:08:01Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 44 insertions(+), 35 deletions(-))
google-beta provider: Diff ( 2 files changed, 44 insertions(+), 35 deletions(-))

modular-magician · 2025-03-10T16:35:14Z

Tests analytics

Total tests: 119
Passed tests: 110
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

modular-magician · 2025-03-12T14:12:57Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 43 insertions(+), 43 deletions(-))
google-beta provider: Diff ( 2 files changed, 43 insertions(+), 43 deletions(-))

modular-magician · 2025-03-12T15:40:25Z

Tests analytics

Total tests: 119
Passed tests: 110
Skipped tests: 9
Affected tests: 0

Click here to see the affected service packages

🟢 All tests passed!

View the build log

set old value if unable to calculate local md5 hash

096a13a

github-actions bot requested a review from roaks3 October 17, 2024 08:59

modular-magician added the service/storage label Oct 17, 2024

modular-magician requested a review from kautikdk October 17, 2024 09:10

added test case

23bc382

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

6ab4bf2

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

6fdb1c8

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

914c94c

roaks3 reviewed Oct 23, 2024

View reviewed changes

add customdiff

a1298f2

github-actions bot requested a review from roaks3 October 24, 2024 05:16

gurusai-voleti marked this pull request as draft February 19, 2025 07:42

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

f79c3f2

wiktorn reviewed Feb 20, 2025

View reviewed changes

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

8c0741b

update md5

47694ca

wiktorn reviewed Mar 2, 2025

View reviewed changes

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

a9006a7

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

69fb57e

Merge branch 'GoogleCloudPlatform:main' into inconsis_tf_plan

7c751cc

Update resource_storage_bucket_object.go

6d74d41

gurusai-voleti commented Mar 10, 2025

View reviewed changes

wiktorn mentioned this pull request Mar 11, 2025

Fix Inconsistent Final Plan issue for Google Storage Bucket Object hashicorp/terraform-provider-google#18618

Open

gurusai-voleti added 2 commits March 12, 2025 11:27

update

e1df59a

update

64f9889

		updatedName := getNewTmpTestFile(t, "tf-test")
		updatedDataMd5 := writeFile(updatedName.Name(), []byte("datum"))

fix: if unable to calculate local md5 hash use old value of detect_md5hash #12032

Are you sure you want to change the base?

fix: if unable to calculate local md5 hash use old value of detect_md5hash #12032

Conversation

gurusai-voleti commented Oct 17, 2024 • edited Loading

github-actions bot commented Oct 17, 2024

modular-magician commented Oct 17, 2024

Diff report

modular-magician commented Oct 17, 2024

Tests analytics

modular-magician commented Oct 17, 2024

Diff report

modular-magician commented Oct 17, 2024

Tests analytics

Action taken

modular-magician commented Oct 17, 2024

kautikdk commented Oct 17, 2024 • edited Loading

gurusai-voleti commented Oct 18, 2024 • edited Loading

kautikdk commented Oct 18, 2024

gurusai-voleti commented Oct 18, 2024 • edited Loading

kautikdk commented Oct 18, 2024 • edited Loading

modular-magician commented Oct 21, 2024

Diff report

modular-magician commented Oct 21, 2024

Tests analytics

modular-magician commented Oct 22, 2024

Diff report

modular-magician commented Oct 22, 2024

Tests analytics

github-actions bot commented Oct 22, 2024

roaks3 commented Oct 22, 2024

modular-magician commented Oct 23, 2024

Diff report

modular-magician commented Oct 23, 2024

Tests analytics

roaks3 left a comment

Choose a reason for hiding this comment

modular-magician commented Feb 20, 2025

Diff report

modular-magician commented Feb 20, 2025

Tests analytics

Choose a reason for hiding this comment

modular-magician commented Feb 21, 2025

Diff report

modular-magician commented Feb 21, 2025

Tests analytics

modular-magician commented Feb 24, 2025

Diff report

modular-magician commented Feb 24, 2025

Tests analytics

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

modular-magician commented Mar 3, 2025

Diff report

modular-magician commented Mar 3, 2025

Tests analytics

modular-magician commented Mar 6, 2025

Diff report

modular-magician commented Mar 6, 2025

Tests analytics

modular-magician commented Mar 10, 2025

Diff report

modular-magician commented Mar 10, 2025

Tests analytics

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kautikdk Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kautikdk Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kautikdk Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

modular-magician commented Mar 10, 2025

Diff report

gurusai-voleti commented Oct 17, 2024 •

edited

Loading

kautikdk commented Oct 17, 2024 •

edited

Loading

gurusai-voleti commented Oct 18, 2024 •

edited

Loading

gurusai-voleti commented Oct 18, 2024 •

edited

Loading

kautikdk commented Oct 18, 2024 •

edited

Loading

kautikdk Mar 11, 2025 •

edited

Loading

kautikdk Mar 11, 2025 •

edited

Loading

kautikdk Mar 12, 2025 •

edited

Loading