Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce behavior-change configs as an alternative to feature configs #1124

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@
import java.util.Objects;
import java.util.stream.Stream;
import org.apache.polaris.core.PolarisCallContext;
import org.apache.polaris.core.PolarisConfigurationStore;
import org.apache.polaris.core.PolarisDefaultDiagServiceImpl;
import org.apache.polaris.core.PolarisDiagnostics;
import org.apache.polaris.core.config.PolarisConfigurationStore;
import org.apache.polaris.core.entity.PolarisPrincipalSecrets;
import org.apache.polaris.core.persistence.BasePolarisMetaStoreManagerTest;
import org.apache.polaris.core.persistence.PolarisTestMetaStoreManager;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,11 @@
/**
* @implSpec This test expects the server to be configured with the following features configured:
* <ul>
* <li>{@link org.apache.polaris.core.PolarisConfiguration#ALLOW_OVERLAPPING_CATALOG_URLS}:
* <li>{@link
* org.apache.polaris.core.config.FeatureConfiguration#ALLOW_OVERLAPPING_CATALOG_URLS}:
* {@code true}
* <li>{@link
* org.apache.polaris.core.PolarisConfiguration#SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION}:
* org.apache.polaris.core.config.FeatureConfiguration#SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION}:
* {@code true}
* </ul>
* The server must also be configured to reject request body sizes larger than 1MB (1000000
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,11 @@
* @implSpec @implSpec This test expects the server to be configured with the following features
* configured:
* <ul>
* <li>{@link org.apache.polaris.core.PolarisConfiguration#ALLOW_OVERLAPPING_CATALOG_URLS}:
* <li>{@link
* org.apache.polaris.core.config.FeatureConfiguration#ALLOW_OVERLAPPING_CATALOG_URLS}:
* {@code true}
* <li>{@link
* org.apache.polaris.core.PolarisConfiguration#ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING}:
* org.apache.polaris.core.config.FeatureConfiguration#ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING}:
* {@code true}
* </ul>
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@
import org.apache.iceberg.rest.RESTCatalog;
import org.apache.iceberg.rest.responses.ErrorResponse;
import org.apache.iceberg.types.Types;
import org.apache.polaris.core.PolarisConfiguration;
import org.apache.polaris.core.admin.model.AwsStorageConfigInfo;
import org.apache.polaris.core.admin.model.Catalog;
import org.apache.polaris.core.admin.model.CatalogGrant;
Expand All @@ -76,6 +75,8 @@
import org.apache.polaris.core.admin.model.TablePrivilege;
import org.apache.polaris.core.admin.model.ViewGrant;
import org.apache.polaris.core.admin.model.ViewPrivilege;
import org.apache.polaris.core.config.FeatureConfiguration;
import org.apache.polaris.core.config.PolarisConfiguration;
import org.apache.polaris.core.entity.CatalogEntity;
import org.apache.polaris.core.entity.PolarisEntityConstants;
import org.apache.polaris.service.it.env.CatalogApi;
Expand Down Expand Up @@ -449,7 +450,7 @@ public void testCreateTableWithOverriddenBaseLocation() {
Catalog catalog = managementApi.getCatalog(currentCatalogName);
Map<String, String> catalogProps = new HashMap<>(catalog.getProperties().toMap());
catalogProps.put(
PolarisConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "false");
FeatureConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "false");
managementApi.updateCatalog(catalog, catalogProps);

restCatalog.createNamespace(Namespace.of("ns1"));
Expand Down Expand Up @@ -477,7 +478,7 @@ public void testCreateTableWithOverriddenBaseLocationCannotOverlapSibling() {
Catalog catalog = managementApi.getCatalog(currentCatalogName);
Map<String, String> catalogProps = new HashMap<>(catalog.getProperties().toMap());
catalogProps.put(
PolarisConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "false");
FeatureConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "false");
managementApi.updateCatalog(catalog, catalogProps);

restCatalog.createNamespace(Namespace.of("ns1"));
Expand Down Expand Up @@ -514,7 +515,7 @@ public void testCreateTableWithOverriddenBaseLocationMustResideInNsDirectory() {
Catalog catalog = managementApi.getCatalog(currentCatalogName);
Map<String, String> catalogProps = new HashMap<>(catalog.getProperties().toMap());
catalogProps.put(
PolarisConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "false");
FeatureConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "false");
managementApi.updateCatalog(catalog, catalogProps);

restCatalog.createNamespace(Namespace.of("ns1"));
Expand Down Expand Up @@ -563,7 +564,7 @@ public void testLoadTableWithAccessDelegationForExternalCatalogWithConfigDisable
.isInstanceOf(ForbiddenException.class)
.hasMessageContaining("Access Delegation is not enabled for this catalog")
.hasMessageContaining(
PolarisConfiguration.ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING.catalogConfig());
FeatureConfiguration.ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING.catalogConfig());
} finally {
resolvingFileIO.deleteFile(fileLocation);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@
import java.util.Map;
import org.apache.iceberg.rest.RESTCatalog;
import org.apache.iceberg.view.ViewCatalogTests;
import org.apache.polaris.core.PolarisConfiguration;
import org.apache.polaris.core.admin.model.Catalog;
import org.apache.polaris.core.admin.model.CatalogProperties;
import org.apache.polaris.core.admin.model.PolarisCatalog;
import org.apache.polaris.core.admin.model.PrincipalWithCredentials;
import org.apache.polaris.core.admin.model.StorageConfigInfo;
import org.apache.polaris.core.config.FeatureConfiguration;
import org.apache.polaris.core.entity.CatalogEntity;
import org.apache.polaris.service.it.env.ClientCredentials;
import org.apache.polaris.service.it.env.IcebergHelper;
Expand All @@ -50,8 +50,8 @@
* client.
*
* @implSpec This test expects the server to be configured with {@link
* org.apache.polaris.core.PolarisConfiguration#SUPPORTED_CATALOG_STORAGE_TYPES} set to the
* appropriate storage type.
* org.apache.polaris.core.config.FeatureConfiguration#SUPPORTED_CATALOG_STORAGE_TYPES} set to
* the appropriate storage type.
*/
@ExtendWith(PolarisIntegrationTestExtension.class)
public abstract class PolarisRestCatalogViewIntegrationBase extends ViewCatalogTests<RESTCatalog> {
Expand Down Expand Up @@ -99,9 +99,9 @@ public void before(TestInfo testInfo) {
CatalogProperties.builder(defaultBaseLocation)
.addProperty(
CatalogEntity.REPLACE_NEW_LOCATION_PREFIX_WITH_CATALOG_DEFAULT_KEY, "file:")
.addProperty(PolarisConfiguration.ALLOW_EXTERNAL_TABLE_LOCATION.catalogConfig(), "true")
.addProperty(FeatureConfiguration.ALLOW_EXTERNAL_TABLE_LOCATION.catalogConfig(), "true")
.addProperty(
PolarisConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "true")
FeatureConfiguration.ALLOW_UNSTRUCTURED_TABLE_LOCATION.catalogConfig(), "true")
.build();
Catalog catalog =
PolarisCatalog.builder()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,10 @@
* @implSpec This test expects the server to be configured with the following features enabled:
* <ul>
* <li>{@link
* org.apache.polaris.core.PolarisConfiguration#SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION}:
* org.apache.polaris.core.config.FeatureConfiguration#SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION}:
* {@code true}
* <li>{@link org.apache.polaris.core.PolarisConfiguration#ALLOW_OVERLAPPING_CATALOG_URLS}:
* <li>{@link
* org.apache.polaris.core.config.FeatureConfiguration#ALLOW_OVERLAPPING_CATALOG_URLS}:
* {@code true}
* </ul>
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import jakarta.annotation.Nonnull;
import java.time.Clock;
import java.time.ZoneId;
import org.apache.polaris.core.config.PolarisConfigurationStore;
import org.apache.polaris.core.persistence.transactional.TransactionalPersistence;

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@
import java.util.Set;
import java.util.stream.Collectors;
import org.apache.iceberg.exceptions.ForbiddenException;
import org.apache.polaris.core.PolarisConfiguration;
import org.apache.polaris.core.PolarisConfigurationStore;
import org.apache.polaris.core.config.FeatureConfiguration;
import org.apache.polaris.core.config.PolarisConfigurationStore;
import org.apache.polaris.core.context.CallContext;
import org.apache.polaris.core.entity.PolarisBaseEntity;
import org.apache.polaris.core.entity.PolarisEntityConstants;
Expand Down Expand Up @@ -510,7 +510,7 @@ public void authorizeOrThrow(
boolean enforceCredentialRotationRequiredState =
featureConfig.getConfiguration(
CallContext.getCurrentContext().getPolarisCallContext(),
PolarisConfiguration.ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING);
FeatureConfiguration.ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING);
if (enforceCredentialRotationRequiredState
&& authenticatedPrincipal
.getPrincipalEntity()
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.polaris.core.config;

import java.util.Optional;

/**
* Configurations for non-feature beheavior changes within Polaris. These configurations are not
* intended for use by end users and govern nuanced behavior changes and bugfixes. The
* configurations never expose user-facing catalog-level configurations. These configurations are
* not stable and may be removed at any time.
*
* @param <T> The type of the configuration
*/
public class BehaviorChangeConfiguration<T> extends PolarisConfiguration<T> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need different types to support this? I think probably adding a field to PolarisConfiguration would be sufficient

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking we don't need a new type, but I chose to add one because I think it's valuable to separate the actual config instances from one another. Besides just having them clearly separated & organized, it means the caller is forced to recognize which type they are relying on (e.g. BehaviorChangeConfiguration.UNSTABLE_FLAG vs FeatureConfiguration.ACTUAL_FEATURE). It also allows us to override e.g. catalogConfig which shouldn't be valid for these flags.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also allows us to override e.g. catalogConfig which shouldn't be valid for these flags.

That seems reasonable, but given that we use a builder, can't we just enforce that at construction? I do like having the different config types housed in different namespaces (as in your examples BehaviorChangeConfiguration.UNSTABLE_FLAG vs FeatureConfiguration.ACTUAL_FEATURE)...

I guess I would just like to ensure that the call sites that are checking for config values don't need to know what type the configuration is at runtime. I can just imagine cases where someone accidentally references one type and the config field is another or someone tries to write some generic code that only works for one config type but not the other...


protected BehaviorChangeConfiguration(
String key, String description, T defaultValue, Optional<String> catalogConfig) {
super(key, description, defaultValue, catalogConfig);
}

public static final BehaviorChangeConfiguration<Boolean> VALIDATE_VIEW_LOCATION_OVERLAP =
PolarisConfiguration.<Boolean>builder()
.key("STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS")
.description("If true, validate that view locations don't overlap when views are created")
.defaultValue(true)
.buildBehaviorChangeConfiguration();
}
Loading