-
Notifications
You must be signed in to change notification settings - Fork 17
Add virtual platform constraint plugins for automatic version alignment across group modules #1436
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
FinlayRJW
wants to merge
51
commits into
develop
Choose a base branch
from
finlayw/VP
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
51 commits
Select commit
Hold shift + click to select a range
427f958
nearly working
FinlayRJW 4a69fd3
working but I don't like it
FinlayRJW c4f3931
working for buildScript
FinlayRJW d4039d8
tidy tests
FinlayRJW 86711c1
tidy tests
FinlayRJW 3a4e8de
not working
FinlayRJW 15cb035
fix so can debug
FinlayRJW f8b6eca
check GMM directly
FinlayRJW 9122a6a
read GMM directly
FinlayRJW 32a4b6c
tidy up
FinlayRJW e95fad7
tidy up
FinlayRJW e053db8
tidy up
FinlayRJW 801d0ac
make tests cleaner
FinlayRJW 5ae4125
tidy tests
FinlayRJW 3a4fa16
tidy up
FinlayRJW a608475
add virtual platform
FinlayRJW cb7d2ec
tidy up
FinlayRJW 1ca7286
tidy up
FinlayRJW f827d2c
spotless
FinlayRJW 0ab6ccc
Update src/main/java/com/palantir/gradle/versions/VirtualPlatformSett…
FinlayRJW 896f366
Update src/main/java/com/palantir/gradle/versions/AddVirtualPlatformP…
FinlayRJW 3851352
fix throw
FinlayRJW baeac65
only care about POM
FinlayRJW 232402e
properly defined input factory
FinlayRJW 01f7c5e
tidy up
FinlayRJW 5b9dd60
two transitives
FinlayRJW de77ba0
apply from settings plugn
FinlayRJW 16210df
remove gradle 6 + CC
FinlayRJW 4e632e1
no need for gradle property
FinlayRJW c1ffec2
parse pom
FinlayRJW dc9b80c
explode
FinlayRJW 9990c29
explode
FinlayRJW 543e76b
rename
FinlayRJW 8aaed15
Update src/main/java/com/palantir/gradle/versions/ExternallyConsisten…
FinlayRJW e1d9d97
actually collect all the vps
FinlayRJW 15f27e7
before project
FinlayRJW 6500038
add CC test
FinlayRJW b94709e
create consistent-versions project
FinlayRJW 507778b
move to settings project
FinlayRJW c3d54b1
make private
FinlayRJW 417c3ac
make private
FinlayRJW cb34c21
tidy up
FinlayRJW dbc3948
this is fine
FinlayRJW 0cdf53d
tidy up
FinlayRJW f61f3cc
rename classes
FinlayRJW 3b16fe2
update readme
FinlayRJW 7568021
improve readme
FinlayRJW 0552c0b
cacheable rule!
FinlayRJW df73149
filter to only java
FinlayRJW aa7c874
OBO error
FinlayRJW 73e729d
Merge remote-tracking branch 'origin/develop' into finlayw/VP
FinlayRJW File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| apply plugin: 'com.palantir.external-publish-jar' | ||
| apply plugin: 'java-gradle-plugin' | ||
| apply plugin: 'com.palantir.gradle-plugin-testing' | ||
| apply plugin: 'groovy' | ||
| apply plugin: 'com.palantir.shadow-jar' | ||
|
|
||
| dependencies { | ||
| shadeTransitively 'com.fasterxml.jackson.dataformat:jackson-dataformat-xml' | ||
| shadeTransitively 'com.fasterxml.jackson.datatype:jackson-datatype-guava' | ||
| shadeTransitively 'com.fasterxml.jackson.datatype:jackson-datatype-jdk8' | ||
|
|
||
| testImplementation 'com.netflix.nebula:nebula-test' | ||
|
|
||
| annotationProcessor "org.immutables:value" | ||
| compileOnly "org.immutables:value::annotations" | ||
| } | ||
|
|
||
| gradlePlugin { | ||
| website = 'https://github.com/palantir/gradle-consistent-versions' | ||
| vcsUrl = 'https://github.com/palantir/gradle-consistent-versions' | ||
|
|
||
| plugins { | ||
| externallyConsistentVersions { | ||
| id = 'com.palantir.externally-consistent-versions' | ||
| implementationClass = 'com.palantir.gradle.versions.ConstraintConsumerSettingsPlugin' | ||
| displayName = 'Settings plugin to automatically assign dependencies to a virtual platforms' | ||
| description = displayName | ||
| tags.set(['versions']) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| test { | ||
| jvmArgs '--add-opens', 'java.base/java.lang.invoke=ALL-UNNAMED' | ||
| jvmArgs '--add-opens', 'java.base/java.util=ALL-UNNAMED' | ||
| } | ||
|
|
||
| gradleTestUtils { | ||
| gradleVersions = ['8.14.3'] | ||
| } |
165 changes: 165 additions & 0 deletions
165
...versions/src/main/java/com/palantir/gradle/versions/ConstraintConsumerSettingsPlugin.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,165 @@ | ||
| /* | ||
| * (c) Copyright 2025 Palantir Technologies Inc. All rights reserved. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.palantir.gradle.versions; | ||
|
|
||
| import com.ctc.wstx.stax.WstxInputFactory; | ||
| import com.ctc.wstx.stax.WstxOutputFactory; | ||
| import com.fasterxml.jackson.annotation.JsonIgnoreProperties; | ||
| import com.fasterxml.jackson.databind.ObjectMapper; | ||
| import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
| import com.fasterxml.jackson.dataformat.xml.XmlMapper; | ||
| import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper; | ||
| import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty; | ||
| import com.fasterxml.jackson.datatype.jdk8.Jdk8Module; | ||
| import java.io.IOException; | ||
| import java.io.InputStream; | ||
| import java.util.List; | ||
| import java.util.Optional; | ||
| import java.util.stream.Collectors; | ||
| import javax.inject.Inject; | ||
| import org.gradle.api.Plugin; | ||
| import org.gradle.api.UncheckedIOException; | ||
| import org.gradle.api.artifacts.CacheableRule; | ||
| import org.gradle.api.artifacts.ComponentMetadataContext; | ||
| import org.gradle.api.artifacts.ComponentMetadataRule; | ||
| import org.gradle.api.artifacts.ModuleVersionIdentifier; | ||
| import org.gradle.api.artifacts.repositories.RepositoryResourceAccessor; | ||
| import org.gradle.api.initialization.Settings; | ||
| import org.gradle.api.logging.Logger; | ||
| import org.gradle.api.logging.Logging; | ||
| import org.gradle.api.plugins.JavaPlugin; | ||
| import org.immutables.value.Value; | ||
|
|
||
| public class ConstraintConsumerSettingsPlugin implements Plugin<Settings> { | ||
|
|
||
| private static final Logger log = Logging.getLogger(ConstraintConsumerSettingsPlugin.class); | ||
|
|
||
| @Override | ||
| public final void apply(Settings settings) { | ||
| settings.getGradle().rootProject(rootProject -> { | ||
| rootProject.getPluginManager().apply(ConstraintProducerPlugin.class); | ||
| }); | ||
|
|
||
| settings.getGradle().beforeProject(project -> { | ||
| project.getDependencies().getComponents().all(VirtualPlatformRule.class); | ||
| project.getBuildscript().getDependencies().getComponents().all(VirtualPlatformRule.class); | ||
| }); | ||
| } | ||
|
|
||
| @CacheableRule | ||
| public abstract static class VirtualPlatformRule implements ComponentMetadataRule { | ||
| private static final ObjectMapper XML_MAPPER = | ||
| new XmlMapper(new WstxInputFactory(), new WstxOutputFactory()).registerModule(new Jdk8Module()); | ||
|
|
||
| @Inject | ||
| protected abstract RepositoryResourceAccessor getRepositoryResourceAccessor(); | ||
|
|
||
| @Override | ||
| public final void execute(ComponentMetadataContext context) { | ||
| ModuleVersionIdentifier id = context.getDetails().getId(); | ||
| processVariant(context, id, JavaPlugin.RUNTIME_ELEMENTS_CONFIGURATION_NAME); | ||
| processVariant(context, id, JavaPlugin.API_ELEMENTS_CONFIGURATION_NAME); | ||
| } | ||
|
|
||
| private void processVariant(ComponentMetadataContext context, ModuleVersionIdentifier id, String variantName) { | ||
| context.getDetails().withVariant(variantName, _variant -> { | ||
| String pomPath = buildPomPath(id); | ||
| getRepositoryResourceAccessor().withResource(pomPath, resource -> { | ||
| parsePom(resource) | ||
| .flatMap(Metadata::extractDependencies) | ||
| .map(VirtualPlatformRule::extractVirtualPlatforms) | ||
| .filter(platforms -> !platforms.isEmpty()) | ||
| .ifPresent(platforms -> assignToPlatforms(context, id, platforms)); | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| private static Optional<Metadata> parsePom(InputStream resource) { | ||
| try { | ||
| return Optional.of(XML_MAPPER.readValue(resource, Metadata.class)); | ||
| } catch (IOException e) { | ||
| throw new UncheckedIOException("Failed to parse POM: ", e); | ||
| } | ||
| } | ||
|
|
||
| private static List<String> extractVirtualPlatforms(List<Dependency> dependencies) { | ||
| return dependencies.stream() | ||
| .map(VirtualPlatformRule::buildPlatformCoordinate) | ||
| .<String>mapMulti(Optional::ifPresent) | ||
| .collect(Collectors.toList()); | ||
| } | ||
|
|
||
| private static Optional<String> buildPlatformCoordinate(Dependency dependency) { | ||
| return dependency | ||
| .group() | ||
| .filter(group -> group.startsWith(ConstraintProducerPlugin.VIRTUAL_PLATFORM_PREFIX)) | ||
| .map(group -> group.substring(ConstraintProducerPlugin.VIRTUAL_PLATFORM_PREFIX.length() + 1)) | ||
| .flatMap(extractedGroup -> dependency.module().map(module -> extractedGroup + ":" + module)); | ||
| } | ||
|
|
||
| private static String buildPomPath(ModuleVersionIdentifier id) { | ||
| String groupPath = id.getGroup().replace('.', '/'); | ||
| return String.format( | ||
| "%s/%s/%s/%s-%s.pom", groupPath, id.getName(), id.getVersion(), id.getName(), id.getVersion()); | ||
| } | ||
CRogers marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| private static void assignToPlatforms( | ||
| ComponentMetadataContext context, ModuleVersionIdentifier id, List<String> platformCoordinates) { | ||
| platformCoordinates.forEach(platform -> { | ||
| log.debug("Assigning component {} to virtual platform {}", id, platform); | ||
| context.getDetails().belongsTo(platform + ":" + id.getVersion(), true); | ||
| }); | ||
| } | ||
| } | ||
|
|
||
| @Value.Immutable | ||
| @JsonDeserialize(as = ImmutableMetadata.class) | ||
| @JsonIgnoreProperties(ignoreUnknown = true) | ||
| interface Metadata { | ||
| Optional<DependencyManagement> dependencyManagement(); | ||
|
|
||
| default Optional<List<Dependency>> extractDependencies() { | ||
| return dependencyManagement().map(DependencyManagement::dependencies); | ||
| } | ||
| } | ||
|
|
||
| @Value.Immutable | ||
| @JsonDeserialize(as = ImmutableDependencyManagement.class) | ||
| @JsonIgnoreProperties(ignoreUnknown = true) | ||
| interface DependencyManagement { | ||
| @JacksonXmlElementWrapper(localName = "dependencies") | ||
| @JacksonXmlProperty(localName = "dependency") | ||
| @Value.Default | ||
| default List<Dependency> dependencies() { | ||
| return List.of(); | ||
| } | ||
| } | ||
|
|
||
| @Value.Immutable | ||
| @JsonDeserialize(as = ImmutableDependency.class) | ||
| @JsonIgnoreProperties(ignoreUnknown = true) | ||
| interface Dependency { | ||
| @JacksonXmlProperty(localName = "groupId") | ||
| Optional<String> group(); | ||
|
|
||
| @JacksonXmlProperty(localName = "artifactId") | ||
| Optional<String> module(); | ||
|
|
||
| @JacksonXmlProperty(localName = "version") | ||
| Optional<String> version(); | ||
| } | ||
| } | ||
116 changes: 116 additions & 0 deletions
116
...sistent-versions/src/main/java/com/palantir/gradle/versions/ConstraintProducerPlugin.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| /* | ||
| * (c) Copyright 2025 Palantir Technologies Inc. All rights reserved. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.palantir.gradle.versions; | ||
|
|
||
| import com.google.common.collect.ImmutableList; | ||
| import java.util.Set; | ||
| import java.util.stream.Collectors; | ||
| import org.gradle.api.Named; | ||
| import org.gradle.api.NamedDomainObjectProvider; | ||
| import org.gradle.api.Plugin; | ||
| import org.gradle.api.Project; | ||
| import org.gradle.api.artifacts.Configuration; | ||
| import org.gradle.api.plugins.JavaPlugin; | ||
| import org.gradle.api.publish.Publication; | ||
| import org.gradle.api.publish.PublishingExtension; | ||
| import org.gradle.api.publish.ivy.IvyPublication; | ||
| import org.gradle.api.publish.maven.MavenPublication; | ||
|
|
||
| public class ConstraintProducerPlugin implements Plugin<Project> { | ||
|
|
||
| static final String VIRTUAL_PLATFORM_PREFIX = "consistent-versions.external-virtual-platform"; | ||
|
|
||
| @Override | ||
| public final void apply(Project rootProject) { | ||
| if (!rootProject.equals(rootProject.getRootProject())) { | ||
| throw new IllegalStateException("ConstraintProducerPlugin must be applied to the root project"); | ||
| } | ||
|
|
||
| rootProject.afterEvaluate(project -> { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is this still configuration cacheable ?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think so - I've added a basic test def "check does not break configuration cache"() {
expect:
runTasksWithConfigurationCache('--write-locks')
runTasksWithConfigurationCache('build')
}
Which I think would blow up if there where any configuration cache issues |
||
| project.getAllprojects().stream() | ||
| .filter(ConstraintProducerPlugin::isJavaLibrary) | ||
| .collect(Collectors.groupingBy(proj -> String.valueOf(proj.getGroup()), Collectors.toSet())) | ||
| .forEach((groupName, projects) -> { | ||
| if (projects.size() > 1) { | ||
| applyVirtualPlatformPerGroup(groupName, projects); | ||
| } | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| private void applyVirtualPlatformPerGroup(String groupName, Set<Project> projectGroup) { | ||
| String platformCoordinates = VIRTUAL_PLATFORM_PREFIX + "." + groupName + ":_"; | ||
|
|
||
| projectGroup.stream() | ||
| .filter(project -> project.getPluginManager().hasPlugin("java")) | ||
| .forEach(project -> addVirtualPlatformToProject(project, platformCoordinates)); | ||
| } | ||
|
|
||
| private void addVirtualPlatformToProject(Project project, String platformCoordinates) { | ||
| NamedDomainObjectProvider<Configuration> constraints = project.getConfigurations() | ||
| .register("virtualPlatformConstraints", config -> { | ||
| config.setDescription( | ||
| "Enforces version alignment via virtual platform for modules within the same group"); | ||
| config.setCanBeResolved(false); | ||
| config.setCanBeConsumed(false); | ||
| config.setVisible(false); | ||
|
|
||
| config.getDependencyConstraints() | ||
| .add(project.getDependencies().getConstraints().create(platformCoordinates, constraint -> { | ||
| constraint.version( | ||
| v -> v.require(project.getVersion().toString())); | ||
| constraint.because("Virtual platform for version alignment across group when using " | ||
| + "com.palantir.externally-consistent-versions"); | ||
| })); | ||
| }); | ||
|
|
||
| project.getConfigurations() | ||
| .named(JavaPlugin.API_ELEMENTS_CONFIGURATION_NAME) | ||
| .configure(conf -> conf.extendsFrom(constraints.get())); | ||
|
|
||
| project.getConfigurations() | ||
| .named(JavaPlugin.RUNTIME_ELEMENTS_CONFIGURATION_NAME) | ||
| .configure(conf -> conf.extendsFrom(constraints.get())); | ||
| } | ||
|
|
||
| static boolean isJavaLibrary(Project project) { | ||
| if (project.getPluginManager().hasPlugin("nebula.maven-publish")) { | ||
| // 'nebula.maven-publish' creates publications lazily which causes inconsistencies based | ||
| // on ordering. | ||
| return true; | ||
| } | ||
| PublishingExtension publishing = project.getExtensions().findByType(PublishingExtension.class); | ||
| if (publishing == null) { | ||
| return false; | ||
| } | ||
| ImmutableList<String> jarPublications = publishing.getPublications().stream() | ||
| .filter(ConstraintProducerPlugin::isLibraryPublication) | ||
| .map(Named::getName) | ||
| .collect(ImmutableList.toImmutableList()); | ||
| return !jarPublications.isEmpty(); | ||
| } | ||
|
|
||
| private static boolean isLibraryPublication(Publication publication) { | ||
| if (publication instanceof MavenPublication mavenPublication) { | ||
| return mavenPublication.getArtifacts().stream().anyMatch(artifact -> "jar".equals(artifact.getExtension())); | ||
| } | ||
| if (publication instanceof IvyPublication ivyPublication) { | ||
| return ivyPublication.getArtifacts().stream().anyMatch(artifact -> "jar".equals(artifact.getExtension())); | ||
| } | ||
| return true; | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should test the performance of this on a large internal repo wrt to dependency resolution times. There is the option to cache these rules, we should see if we need to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Testing on an internal project seems to indicate that on CI no more network requests are made when resolving the
runtimeClasspathandcompileClasspathsee discussion here https://pl.ntr/2xp