Spin up checks required variables for some Providers

[seun-ja]

New logic only allows no default variable before runtime if the Provider is of a Dynamic type

Closes #3115

The PR checks if the variable default isn't provided when required. A new hook handles the check: VariablesValidatorHook.

Inspired by @itowlson's #3115 (comment), new metadata has been added to Providers.

It runs a validation where there's no Dynamic variable provider. One should get an error like this

Error: provider error: no provider resolved required variables: ["foo"]

Caused by:
      no provider resolved required variable(s): ["foo"]

NB. It's variables now, as there's a possibility of having multiple variables

[seun-ja]

I noticed something unusual about the test_vault_config_provider test

It is meant to target VaultVariablesProvider; however, it ends up targeting StaticVariablesProvider.

With the new setup, VaultVariablesProvider is treated as a Dynamic variable provider, while StaticVariablesProvider is regarded as a Static variable provider. Therefore, even without providing the data in spin.toml, the test should still pass because the check should be bypassed, as it is assumed to be available at runtime. When debugging the reason for the initial test failure, I realised the test was targeting StaticVariablesProvider.

Is it by design that it was treated as StaticVariablesProvider?

cc: @rylev

[itowlson]

I noted some problems in the comments, but one other thing is that this would really benefit from tests - those would have provided assurance around what I currently suspect are logic errors. At minimum we should have these test cases:

• The application uses no variables (expected: valid)
• The application uses one variable, is equipped with a single static provider, and:
  • the provider has the variable (expected: valid)
  • the provider does not have the variable (expected: invalid)
• The application uses one variable, and is equipped with a single dynamic provider (expected: valid)
• The application uses one variable, is equipped with both a dynamic and a static provider, and:
  • the static provider has the variable (expected: valid)
  • the static provider does not have the variable (expected: valid)
• The application uses one variable, is equipped with two static providers, and:
  • the first provider we check has the variable (expected: valid)
  • the first provider we check does not have the variable, but the second does (expected: valid)
  • neither provider has the variable (expected: invalid)

I am not sure if we write tests for cases where there are multiple variables - in theory we do but in practice the logic there is fairly simple.

[itowlson]

Technically this doesn't need &self for functionality, but I guess maybe it needs it so it can be called polymorphically via a boxed reference?

[itowlson]

Not sure what pre_runtime_prepare means in this context. My understanding is that you intend to check if variables references are doomed to fail: can we find a name that expresses that?

[itowlson]

I am pretty sure the logic here is wrong. Suppose I have a variable myvar and there are two static providers in play. myvar exists in Provider B but not in Provider A. If Provider A is hit first, it will say myvar doesn't exist, even though Provider B would have actually satisfied it.

[itowlson]

I am pretty sure the logic here is wrong. If there is any dynamic provider in play, then we cannot assert that the variable is doomed. We do not go on and try to find a static provider (and we definitely do not fail if no static provider has the variable).

[itowlson]

It would be good for the PR description to show the output on failure - I am not sure what message this would produce and whether it would be suitable for startup time.

[itowlson]

I'm looking at test_vault_config_provider and it appears to bring in the Vault provider via runtime config (

spin/tests/integration.rs

Line 552 in d1eb9ac

spin_up_args: vec!["--runtime-config-file".into(), "runtime_config.toml".into()],

, https://github.com/spinframework/spin/blob/main/tests/testcases/vault-variables-test/runtime_config.toml). Can you say more about what you're seeing that makes you think it's bypassing Vault?

If it's only this:

With the new setup, VaultVariablesProvider is treated as a Dynamic variable provider, while StaticVariablesProvider is regarded as a Static variable provider. Therefore, even without providing the data in spin.toml, the test should still pass because the check should be bypassed, as it is assumed to be available at runtime.

then that I think could be an error in your validation code (where you check the static providers even if dynamic providers are in play).

You did also mention that in debugging you saw it going to the environment provider - that's not unexpected in itself because the providers are checked in priority order - you should be concerned only if you don't see it going to the Vault provider.

[itowlson]

I should maybe expand that penultimate comment. When a developer makes a change, and an existing test fails (especially one in the area they're changing), they should treat that as "I broke existing behaviour" until proven otherwise. They should not assume that their changes are correct and have shown up a flaw in the test.

That's not to say that existing tests never have flaws. They do! They have loads of flaws! And sometimes it turns out that the code change is correct and the test had a hidden flaw. But it's wise to start from a position of "the problem is likely in the new, untested code" before resorting to "fixing" the test.

[seun-ja]

Can you say more about what you're seeing that makes you think it's bypassing Vault?

As you rightly pointed out, the validation logic was flawed. So the reason for seeing a Static was an expected occurrence, as it is part of the available Providers. The logic sees Static first and just returns an error, as it couldn't find a variable.

The new logic seems to have resolved that issue and should do what is expected. However, there needs to be some updates to the logic, as some integration_tests tests (different from the previous ones). Will continue on that tomorrow.

[itowlson]

Before we go further could you write some unit tests please? For example, I'm guessing the change to explicitly check for default values was motivated by discovering a case where a variable should have been allowed but wasn't. Which is a good catch... but... there's no test to identify the case and prevent a regression. Test cases allow the reviewer to look at the covered cases and go either "okay I can see this all works and am reviewing only for maintainability" or "huh but what happens in this weird case, I need to review for correctness." Thanks.