Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-19256 Support Conditional Writes in S3A #7329

Open
wants to merge 29 commits into
base: trunk
Choose a base branch
from
Open

HADOOP-19256 Support Conditional Writes in S3A #7329

wants to merge 29 commits into from
+1,581 −103

Conversation

saikatroy038
Copy link
Contributor

@saikatroy038 saikatroy038 commented Jan 27, 2025 •
edited
Loading

Description of PR

Cherry-pick of #7011, and additional changes on top of it

How was this patch tested?

integration tests ran on ap-south-1
Failing tests:
ITestAwsSdkWorkarounds.testNoisyLogging
ITestConnectionTimeouts.testObjectUploadTimeouts
TestRawLocalContractVectoredRead.testVectoredReadAfterNormalRead - java.lang.AssertionError: Resource not found contract/rawlocal.xml

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+1 💚 mvninstall 39m 2s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 0m 42s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 36s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 33s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 0m 39s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 0m 41s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 33s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 9s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 37m 7s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 30s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 27s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 27s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 22s /results-checkstyle-hadoop-tools_hadoop-aws.txt hadoop-tools/hadoop-aws: The patch generated 71 new + 5 unchanged - 0 fixed = 76 total (was 5)
+1 💚 mvnsite 0m 31s the patch passed
-1 ❌ javadoc 0m 28s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ javadoc 0m 25s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 spotbugs 1m 6s the patch passed
+1 💚 shadedclient 38m 0s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 47s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
129m 9s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 506c09f684c8 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 29ff04a
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/testReport/
Max. process+thread count 529 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 50s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+1 💚 mvninstall 39m 52s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 0m 44s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 35s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 32s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 0m 41s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 0m 41s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 33s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 8s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 52s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 29s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 21s /results-checkstyle-hadoop-tools_hadoop-aws.txt hadoop-tools/hadoop-aws: The patch generated 70 new + 5 unchanged - 0 fixed = 75 total (was 5)
+1 💚 mvnsite 0m 34s the patch passed
-1 ❌ javadoc 0m 29s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ javadoc 0m 25s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 spotbugs 1m 6s the patch passed
+1 💚 shadedclient 37m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 44s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
131m 25s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 88b6aaed7d3f 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / aa1defc
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/testReport/
Max. process+thread count 641 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran requested changes Feb 5, 2025
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catching up with this.

I can see it is still part way through doing those first changes I'd suggested. I don't think we need that map of options, I'd rather they get extracted.

we have two create options then

  • reject overwrite
  • later: require match + etag

the code is very, very similar, all that is different is that the second one needs an etag

we need to think of the best way to get both of these down.

For testing large files, I also want to add an option for "always do a multipart write" -but that can go in as a header with a new option defined.

@steveloughran steveloughran self-assigned this Feb 6, 2025
@steveloughran steveloughran mentioned this pull request Feb 6, 2025
Draft
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+1 💚 mvninstall 40m 46s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 0m 45s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 0m 34s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 0m 32s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 0m 41s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 0m 40s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 32s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 1m 9s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 29s the patch passed
+1 💚 compile 0m 37s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 0m 37s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 0m 26s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 21s /results-checkstyle-hadoop-tools_hadoop-aws.txt hadoop-tools/hadoop-aws: The patch generated 65 new + 5 unchanged - 0 fixed = 70 total (was 5)
+1 💚 mvnsite 0m 31s the patch passed
-1 ❌ javadoc 0m 29s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ javadoc 0m 26s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 spotbugs 1m 6s the patch passed
+1 💚 shadedclient 37m 54s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 45s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
132m 1s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 3212193e2d22 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / b19da55
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/testReport/
Max. process+thread count 528 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@github-actions github-actions bot added the Common label Feb 7, 2025
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 55s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 20s Maven dependency ordering for branch
+1 💚 mvninstall 35m 11s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 18m 44s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 17m 23s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 4m 41s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 34s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 5s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 38s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 3m 51s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 39m 30s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 1m 27s the patch passed
+1 💚 compile 18m 15s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 18m 15s the patch passed
+1 💚 compile 17m 10s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 17m 10s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 26s /results-checkstyle-root.txt root: The patch generated 133 new + 23 unchanged - 0 fixed = 156 total (was 23)
+1 💚 mvnsite 2m 30s the patch passed
-1 ❌ javadoc 0m 50s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
-1 ❌ javadoc 0m 46s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)
+1 💚 spotbugs 4m 10s the patch passed
+1 💚 shadedclient 38m 42s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 38s hadoop-common in the patch passed.
-1 ❌ unit 2m 55s /patch-unit-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
252m 37s
Reason Tests
Failed junit tests hadoop.fs.s3a.impl.TestRequestFactory
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 23f3f779377a 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 26811e6
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/testReport/
Max. process+thread count 2734 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 28s Maven dependency ordering for branch
+1 💚 mvninstall 34m 19s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 19m 15s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 18m 2s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 26s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 27s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 1m 58s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 30s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 49s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 37m 58s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 1m 25s the patch passed
+1 💚 compile 17m 55s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 17m 55s the patch passed
+1 💚 compile 17m 6s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 17m 6s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 31s /results-checkstyle-root.txt root: The patch generated 142 new + 23 unchanged - 0 fixed = 165 total (was 23)
+1 💚 mvnsite 2m 26s the patch passed
-1 ❌ javadoc 1m 10s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 50s /patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 46s /patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
+1 💚 spotbugs 4m 9s the patch passed
+1 💚 shadedclient 39m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 35s hadoop-common in the patch passed.
+1 💚 unit 2m 55s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
250m 42s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 9d450b01ee8e 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 3e59b74
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/testReport/
Max. process+thread count 1255 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 51s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 5m 57s Maven dependency ordering for branch
+1 💚 mvninstall 34m 52s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 18m 43s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 17m 12s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 33s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 30s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 6s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 38s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 50s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 1m 23s the patch passed
+1 💚 compile 17m 42s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 17m 42s the patch passed
+1 💚 compile 17m 14s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 17m 14s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 25s /results-checkstyle-root.txt root: The patch generated 244 new + 23 unchanged - 0 fixed = 267 total (was 23)
+1 💚 mvnsite 2m 29s the patch passed
-1 ❌ javadoc 1m 10s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 50s /patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 45s /patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
-1 ❌ spotbugs 1m 30s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 38m 53s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 40s hadoop-common in the patch passed.
+1 💚 unit 2m 59s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
250m 6s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1598]
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux aa90e18fae55 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 294b14d
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/testReport/
Max. process+thread count 1255 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 34s Maven dependency ordering for branch
-1 ❌ mvninstall 47m 43s /branch-mvninstall-root.txt root in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ compile 0m 31s /branch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt root in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ compile 0m 31s /branch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt root in HADOOP-19256-s3-conditional-writes failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
+1 💚 checkstyle 5m 47s HADOOP-19256-s3-conditional-writes passed
-1 ❌ mvnsite 0m 33s /branch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ mvnsite 0m 42s /branch-mvnsite-hadoop-tools_hadoop-aws.txt hadoop-aws in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ javadoc 0m 31s /branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 31s /branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 31s /branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
-1 ❌ spotbugs 1m 39s /branch-spotbugs-hadoop-common-project_hadoop-common.txt hadoop-common in HADOOP-19256-s3-conditional-writes failed.
-1 ❌ spotbugs 0m 31s /branch-spotbugs-hadoop-tools_hadoop-aws.txt hadoop-aws in HADOOP-19256-s3-conditional-writes failed.
+1 💚 shadedclient 13m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 44s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 49s /patch-mvninstall-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ mvninstall 0m 30s /patch-mvninstall-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ compile 1m 20s /patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javac 1m 20s /patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ compile 0m 13s /patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
-1 ❌ javac 0m 13s /patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 14s /buildtool-patch-checkstyle-root.txt The patch fails to run checkstyle in root
-1 ❌ mvnsite 0m 55s /patch-mvnsite-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ mvnsite 0m 24s /patch-mvnsite-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ javadoc 1m 10s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 26s /patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 0m 35s /results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06 with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06 generated 24 new + 0 unchanged - 0 fixed = 24 total (was 0)
-1 ❌ spotbugs 0m 24s /patch-spotbugs-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ spotbugs 0m 24s /patch-spotbugs-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ shadedclient 16m 45s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 27s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch failed.
-1 ❌ unit 0m 32s /patch-unit-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ asflicense 0m 44s /results-asflicense.txt The patch generated 1 ASF License warnings.
88m 22s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux f7f7dd227502 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / b007664
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/testReport/
Max. process+thread count 84 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 18m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 13s Maven dependency ordering for branch
+1 💚 mvninstall 35m 10s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 18m 50s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 17m 17s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 39s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 32s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 4s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 36s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 3m 50s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 38m 27s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 1m 23s the patch passed
+1 💚 compile 18m 34s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 18m 34s the patch passed
+1 💚 compile 17m 16s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 17m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 31s /results-checkstyle-root.txt root: The patch generated 244 new + 23 unchanged - 0 fixed = 267 total (was 23)
+1 💚 mvnsite 2m 28s the patch passed
-1 ❌ javadoc 1m 11s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 1m 37s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
-1 ❌ spotbugs 1m 31s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 38m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 35s hadoop-common in the patch passed.
+1 💚 unit 2m 55s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 1s The patch does not generate ASF License warnings.
269m 8s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1598]
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 23195b073f37 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / b007664
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/testReport/
Max. process+thread count 2927 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@saikatroy038 saikatroy038 force-pushed the s3-conditional-writes branch from 19f3d58 to 4423ab0 Compare March 5, 2025 06:15
@saikatroy038 saikatroy038 force-pushed the s3-conditional-writes branch from 4423ab0 to 19f3d58 Compare March 5, 2025 06:39
@saikatroy038 saikatroy038 changed the base branch from HADOOP-19256-s3-conditional-writes to trunk March 5, 2025 06:39
@saikatroy038 saikatroy038 force-pushed the s3-conditional-writes branch from 17c0ad9 to 19f3d58 Compare March 5, 2025 06:44
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ patch 0m 20s #7329 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
Subsystem Report/Notes
GITHUB PR #7329
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/18/console
versions git=2.34.1
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 56s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ HADOOP-19256-s3-conditional-writes Compile Tests _
+0 🆗 mvndep 6m 15s Maven dependency ordering for branch
+1 💚 mvninstall 34m 55s HADOOP-19256-s3-conditional-writes passed
+1 💚 compile 20m 3s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 18m 7s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 4m 33s HADOOP-19256-s3-conditional-writes passed
+1 💚 mvnsite 2m 31s HADOOP-19256-s3-conditional-writes passed
+1 💚 javadoc 2m 5s HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 1m 35s HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 4m 14s HADOOP-19256-s3-conditional-writes passed
+1 💚 shadedclient 40m 39s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 32s Maven dependency ordering for patch
+1 💚 mvninstall 1m 27s the patch passed
+1 💚 compile 19m 23s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 19m 23s the patch passed
+1 💚 compile 18m 58s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 18m 58s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 32s /results-checkstyle-root.txt root: The patch generated 329 new + 23 unchanged - 0 fixed = 352 total (was 23)
+1 💚 mvnsite 2m 29s the patch passed
-1 ❌ javadoc 1m 12s /patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚 javadoc 1m 39s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
-1 ❌ spotbugs 1m 31s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚 shadedclient 38m 39s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 20m 36s hadoop-common in the patch passed.
+1 💚 unit 2m 54s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 3s The patch does not generate ASF License warnings.
258m 26s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Unread field:S3ABlockOutputStream.java:[line 1609]
Unread field:S3ABlockOutputStream.java:[line 1603]
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/artifact/out/Dockerfile
GITHUB PR #7329
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 067cfe6c4e5a 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HADOOP-19256-s3-conditional-writes / 4423ab0
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/testReport/
Max. process+thread count 1256 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@saikatroy038
Copy link
Contributor Author

with skipHead and skipList in S3AFileSystem.innerCreateFile(), the file will be overridden if etag is passed.. even if we set fsBuilder.overwrite(false). is that expected behaviour?

Saikat Roy added 3 commits March 5, 2025 21:46
HADOOP-19256: Add tests
4739e17
HADOOP-19256: Update tests
a8612e7
HADOOP-19256: Remove incorrect tests.
fe5b9f9 
- testIfMatchCreateFileWithoutOverwrite, testIfMatchCreateFileWithoutOverwriteWithPerformanceFlag
- with the new design, overwrite is ignored if etag is passed
@steveloughran
Copy link
Contributor

good question. options

  1. overwrite takes priority. Good; consistent with the rest. Bad: people explicitly setting etag may be surprised
  2. etag priority. Good: resolves ambiguity about etag flag.
  3. reject as an illegal state.

for 1 and 2 we log at debug too, obviously

I think 1 or 3 is correct, with 3 being the strict one.

@saikatroy038 saikatroy038 changed the title HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile HADOOP-19256 Support Conditional Writes in S3A Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steveloughran steveloughran steveloughran requested changes

Assignees

@steveloughran steveloughran

Labels
AWS build Common DistCp Infra MapReduce TOOLS trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@saikatroy038 @hadoop-yetus @steveloughran