Configuration quoted and unquoted map keys

[radcortez]

When a map key does not contain dots, it can be written in both quoted and unquoted forms:

map.key.url=https://example.com
map.key.timeout=2000

or

map."key".url=https://example.com
map."key".timeout=2000

Both forms are valid and equivalent for simple keys (no dots). However, when both forms are present in the configuration for the same logical key, properties from one form are silently lost:

map."key".url=https://example.com
map.key.timeout=2000

Instead of producing one map entry with both url and timeout populated, only properties from one notation are bound. The other form's values are silently ignored, and which form wins depends on iteration order, hence our recommendation to use the same form: https://smallrye.io/smallrye-config/Latest/config/mappings/#maps.

Root Cause

There is a mismatch between how property names are compared and how map keys are extracted:

• Property name comparison treats map.key.url and map."key".url as two different names — quotes are only significant when the key contains dots.
• Map key extraction strips quotes, so both forms resolve to the same key, to store the unquoted key in the Map

When building a map with a complex value type (a nested object), the code commits to one path form per key and constructs the nested object in a single pass using that path. Properties defined with the other form are never found because the property name comparison considers them different.

In many cases, extensions generate additional configurations that may not follow the user syntax (usually, we don't add quotes to single names, but the user may use them).

Comparison with Spring Boot

Spring Boot uses bracket notation (map[key].url) instead of quotes. For simple keys, both map.key.url and map[key].url are valid. When mixed for a complex object, Spring also binds properties from only one notation — the other is silently ignored. However, Spring's behavior is deterministic (bracket notation takes precedence), while in SmallRye Config the result depends on property name iteration order.

Possible Solutions

Multiple Quarkus extensions have independently worked around this at the extension level:

• Produce both quoted and unquoted variants of schema management config #54492
• Improve compatibility of the REST Client configuration #42932
• Fix dev service always restarting on named datasource configuration change #41736

Ideally, this should be fixed in SmallRye Config directly:

• When building a map, always use the unquoted form as the canonical path for simple keys. This ensures the nested object is always constructed against a consistent, predictable path.
• Fallback lookup: When a property is not found under the canonical (unquoted) name, try the quoted form. This handles the case where a config source only stores the value under the quoted key. Since config sources are provided by external implementations and cannot be modified, the fallback must happen at the lookup layer controlled by SmallRye Config.

Unfortunately, I don't see any other solution than a fallback lookup. I've always refrained from implementing it, because it would potentially cause an explosion of lookups, as explained here: #52690 (comment).

The only other solution I see is to force the use of quotes (or even brackets) on any Map segment, either simple or composed. This actually has the benefit of visually making it clear that a specific segment of a configuration is actually a user-defined name (or dynamic name).

Related

• Inconsistency using quotation marks in named datasources #52690 — Inconsistency using quotation marks in named datasources
• Produce both quoted and unquoted variants of schema management config #54492 — Produce both quoted and unquoted variants of schema management config
• Dev services (new model?) may not apply "additional configuration provider"'s config properly #52699 — Dev services may not apply additional configuration provider's config properly
• Fix dev service always restarting on named datasource configuration change #41736 — Fix dev service always restarting on named datasource configuration change
• Improve compatibility of the REST Client configuration #42932 — Improve compatibility of the REST Client configuration
• Stork environment variable resolution not working properly #41344 — Stork environment variable resolution not working properly
• SRCFG00014 when defining map value from env var #50442 — SRCFG00014 when defining map value from env var
• Problem with environment variables when using multiple datasources #12968 — Problem with environment variables when using multiple datasources
• Cache name with hyphen/dash loses the hyphen during configuration binding in Quarkus Cache Caffeine #54940 — Cache name with hyphen loses the hyphen during configuration binding

[radcortez]

//cc @gsmet

[gsmet]

I know it's a relatively massive breaking change but I would be in favor of having a clear and distinctive format for maps. The fact that the current behavior is ambiguous has come with so many issues.

Given we started with quotes, I would go with quotes - as long as it solves the problems we had a long the way with the syntax being unclear (typically the lookup optimizations).

Now just to reiterate, that would be a relatively massive breaking change. But I think the config machinery will be able to tell people that the elements are not used? And maybe if we get unused elements, we could check if they match a quoted path in a slow path and provide a nice warning message guiding the user?

/cc @geoand @cescoffier

[dmlloyd]

Ambiguity is an unavoidable side-effect of any valid quoting scheme. Escaping is necessary in order to allow any possible input. Quoting is a convenience to make multi-level escaping less confusing. The quotes don't have an inherent meaning (like "this is a map key"). They are a syntactic mechanism which is only valid in the context of properties.

To illustrate why this distinction is important, if you consider Spring-style brackets [/] to be a semantic mechanism ("this is a map key"), you still have the problem of how to handle things that need to be escaped ("my map key includes a [ or ] character") and precedence (foo[bar].baz[zap] could be interpreted as foo with a subkey of bar].baz[zap). So in this case you still have to have escaping. If you have escaping, you really need quoting for usability.

We can't avoid ambiguity in our current reality. The best thing we can do is handle it consistently.

[gsmet]

We can't avoid ambiguity in our current reality. The best thing we can do is handle it consistently.

@dmlloyd I fail to see why we can't avoid it? If we were enforcing quotes for map keys (+ escaping), we would know if something is a map key.

And I really don't see why we would continue on the ambiguity path, while we know it's problematic at several levels (and less clear).
Now, for sure, that would break a lot of configurations, but I wonder if we could have OpenRewrite recipes to solve most of the issues (at least for the core properties).

Now, I might be missing something but in this case, I would like an explanation with some examples. And also would like to understand why continuing with ambiguity is a good thing.

[dmlloyd]

So what is the actual proposal then? Disallowing quoting and escapes altogether? Creating a Spring-style syntax for map keys? How would you make foo[bar] and foo.bar not be equivalent?

[radcortez]

I don't think the proposal was to disallow quoting, but to require quoting whenever you refer to a Map key, even for single segments or non-dotted names, which is, in fact, a big breaking change. We could probably have some OpenRewrite to migrate configuration sources we know. I guess my concern there is how to migrate configurations we don't have access to in obscure places :)

A less intrusive alternative would be to properly define in which order we read the keys (foo or "foo") with a configuration where you can warn or error if you find competing keys.

[dmlloyd]

I was really responding to the idea of eliminating ambiguity. It's one thing to say "map keys have to be quoted". But, config sources don't have knowledge of the config mapping schema which is not one but two layers higher. So you need a complete lexicographic rule set for the properties config source that's internally self-consistent, before you can map that to the general config layer, before you can map to the mapping layer.

So to make ambiguous config impossible, you'd need some way to know which segments must be quoted.

[dmlloyd]

This is a parsing problem, really, and one that the proposed hierarchical design was supposed to solve by semantically interpreting ambiguous segments at the config source level.

To me, the expected behavior would be that I can use whatever syntax I like, but if I have a conflicting key in the same file, then I should get a warning or (better yet) an error when that file is loaded. First one wins, second one is rejected loudly.

There are a lot of reasons why this might be hard with the current implementation. With a hierarchical config source design, everything would be handled segment-wise, so it should be much easier (and should resolve several issues where maps and non-maps behave differently, which has always been a problem).

I don't know where I'm going with this, other than just putting out my opinion (once again) that every second we spend trying to get the original implementation to somehow fit the newer requirements - instead of burning it all down and starting from scratch, which is probably much easier - is another second wasted. If we did this two years ago we'd have been done a year and a half ago and working on much more interesting things.

[radcortez]

To me, the expected behavior would be that I can use whatever syntax I like, but if I have a conflicting key in the same file, then I should get a warning or (better yet) an error when that file is loaded. First one wins, second one is rejected loudly.

We can certainly implement such behavior.

There are a lot of reasons why this might be hard with the current implementation. With a hierarchical config source design, everything would be handled segment-wise, so it should be much easier (and should resolve several issues where maps and non-maps behave differently, which has always been a problem).

I don't think it is hard to fix. Adding a warning or error should be straightforward, since we already know the map keys (either single or quoted in a given segment). We just didn't want to break the current behavior (well, I guess that we could have added a warning, which wouldn't break).