HADOOP-19258: Upgrade aws sdk v2 to 2.27.14 #7015

diljotgrewal · 2024-08-28T19:59:53Z

Description of PR

Upgrading aws sdk to 2.27.14

How was this patch tested?

Tested with maven in us-east-1

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-08-29T15:29:26Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 58s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 9s		Maven dependency ordering for branch
+1 💚	mvninstall	37m 5s		trunk passed
+1 💚	compile	19m 39s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	18m 10s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	mvnsite	26m 36s		trunk passed
+1 💚	javadoc	9m 13s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	8m 7s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	55m 57s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	2m 19s		Maven dependency ordering for patch
+1 💚	mvninstall	34m 50s		the patch passed
+1 💚	compile	19m 34s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	19m 34s		the patch passed
+1 💚	compile	18m 13s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	18m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	20m 29s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	9m 38s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	7m 52s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	shadedclient	57m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	820m 6s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 39s		The patch does not generate ASF License warnings.
		1168m 11s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
	hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
	hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
	hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown
	hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
	hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
	hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/1/artifact/out/Dockerfile
GITHUB PR	#7015
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux f3cb20736a6e 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `efa0fc6`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/1/testReport/
Max. process+thread count	3680 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2024-08-30T11:01:56Z

what command line options did you pass to the test run. e.g (-Dscale, ...)
did you do a test with s3 express or with any third party store
session credential/role tests (set fs.s3a.assumed.role.arn to a valid ARN For this)
did you do the full testing.md stuff about command line manual testing, and looking through the logs for new error messages
did any new dependencies sneak out even though bundle.jar is meant to be shaded?

we have to be really rigorous here as there are many regressions which can sneak past the test suites (new log messages on abort(), performance slow downs) which aren't picked up by junit itself. And needless exported dependencies is a recurrent PITA.

steveloughran · 2024-08-30T11:02:36Z

+rebase or merge branch to resolve the conflicts

hadoop-yetus · 2024-08-31T16:57:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 30s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 54s		trunk passed
+1 💚	compile	17m 50s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	16m 18s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	4m 23s		trunk passed
+1 💚	mvnsite	25m 55s		trunk passed
+1 💚	javadoc	9m 30s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	8m 9s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 21s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
-1 ❌	spotbugs	31m 40s	/branch-spotbugs-root-warnings.html	root in trunk has 2 extant spotbugs warnings.
+1 💚	shadedclient	66m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 51s		Maven dependency ordering for patch
+1 💚	mvninstall	30m 56s		the patch passed
+1 💚	compile	17m 13s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	17m 13s		the patch passed
+1 💚	compile	16m 1s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	16m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 23s		the patch passed
+1 💚	mvnsite	19m 27s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 45s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	8m 9s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+0 🆗	spotbugs	0m 20s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	64m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	757m 25s		root in the patch passed.
+1 💚	asflicense	1m 42s		The patch does not generate ASF License warnings.
		1136m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/2/artifact/out/Dockerfile
GITHUB PR	#7015
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle shellcheck shelldocs
uname	Linux 666f9c344838 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d8530d3`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/2/testReport/
Max. process+thread count	3964 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-aws . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

diljotgrewal · 2024-09-03T03:22:37Z

Testing against a Regular S3 bucket in us-east-1:

auth-keys

<configuration>
  <property>
    <name>test.fs.s3a.name</name>
    <value>s3a://<REDACTED>/</value>
  </property>

  <property>
    <name>fs.contract.test.fs.s3a</name>
    <value>s3a://<REDACTED>/</value>
  </property>

  <property>
    <name>fs.s3a.access.key</name>
    <description>AWS access key ID. Omit for IAM role-based authentication.</description>
    <value><REDACTED></value>
  </property>

  <property>
    <name>fs.s3a.secret.key</name>
    <description>AWS secret key. Omit for IAM role-based authentication.</description>
    <value><REDACTED></value>
  </property>

  <property>
  <name>fs.s3a.encryption.key</name>
  <value>arn:aws:kms:us-east-1:<REDACTED>:key/<REDACTED></value>
  </property>

  <property>
    <name>fs.s3a.assumed.role.arn</name>
    <value>arn:aws:iam::<REDACTED>:role/<REDACTED></value>
  </property>

  <property>
        <name>test.fs.s3a.sts.enabled</name>
        <value>false</value>
    </property>
</configuration>

Scale Tests

 mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale

[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 3, Time elapsed: 8.89 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AConfiguration
[ERROR] testRequestTimeout(org.apache.hadoop.fs.s3a.ITestS3AConfiguration)  Time elapsed: 0.029 s  <<< ERROR!
java.lang.IllegalArgumentException: Cannot locate field clientConfiguration on class software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionSyncClient
	at org.apache.commons.lang3.Validate.isTrue(Validate.java:158)
	at org.apache.commons.lang3.reflect.FieldUtils.readField(FieldUtils.java:469)
	at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.getField(ITestS3AConfiguration.java:535)
	at org.apache.hadoop.fs.s3a.ITestS3AConfiguration.testRequestTimeout(ITestS3AConfiguration.java:449)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)

[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 169.585 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[ERROR] testThreadPoolCoolDown(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)  Time elapsed: 143.941 s  <<< FAILURE!
java.lang.AssertionError: Failed to find threads in active FS - test is flawed. Actual: 0
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failEquals(Assert.java:187)
	at org.junit.Assert.assertNotEquals(Assert.java:201)
	at org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.testThreadPoolCoolDown(ITestS3AConcurrentOps.java:206)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)

[ERROR] Failures:
[ERROR]   ITestS3AConcurrentOps.testThreadPoolCoolDown:206->Assert.assertNotEquals:201->Assert.failEquals:187->Assert.fail:89 Failed to find threads in active FS - test is flawed. Actual: 0
[ERROR] Errors:
[ERROR]   ITestS3AConfiguration.testRequestTimeout:449->getField:535 » IllegalArgument C...
[INFO]
[ERROR] Tests run: 1327, Failures: 1, Errors: 1, Skipped: 127

adding access point:

<property>
    <name>fs.s3a.bucket.sample-bucket.accesspoint.arn</name>
    <value> <REDACTED> </value>
    <description>Configure S3a traffic to use this AccessPoint</description>
</property>

mvn clean verify -Dit.test=ITest* -Dtest=none

Getting a lot of errors, currently debugging:

several parsing errors

grep "Failed to parse b-00" itest_log| sort | uniq -c
    132 java.lang.Exception: Failed to parse b-00

[^[[1;31mERROR^[[m] testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)  Time elapsed: 2.179 s  <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but was:<512>

[^[[1;31mERROR^[[m] testClosedInstrumentation(org.apache.hadoop.fs.s3a.ITestS3AClosedFS)  Time elapsed: 0.333 s  <<< FAILURE!
org.junit.ComparisonFailure: [S3AInstrumentation.hasMetricSystem()] expected:<[fals]e> but was:<[tru]e>
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

few encryption errors:

[^[[1;31mERROR^[[m] testEncryptionOverRename(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 2.502 s  <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected expected:<kms+context> but was:<null>

[^[[1;31mERROR^[[m] testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 3.498 s  <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
  <[0, 1, 239, 4079]>
to contain exactly in any order:
  <[0, 1, 255, 4095]>
elements not found:
  <[255, 4095]>
and elements not expected:
  <[239, 4079]>

[^[[1;31mERROR^[[m] testBytesReadWithStream(org.apache.hadoop.fs.s3a.statistics.ITestS3AFileSystemStatistic)  Time elapsed: 1.646 s  <<< FAILURE!
java.lang.AssertionError: Mismatch in number of FS bytes read by InputStreams expected:<2048> but was:<196167>

[^[[1;31mERROR^[[m] testCustomSignerAndInitializer[simple-delete](org.apache.hadoop.fs.s3a.auth.ITestCustomSigner)  Time elapsed: 1.138 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://s3atestingdsgrewal/job-00/test/testCustomSignerAndInitializer[simple-delete]/customsignerpath1: software.amazon.awssdk.services.s3.model.S3Exception: The authorization mechanism you have provided is not supported. Please use Signature Version 4. (Service: S3, Status Code: 400, Request ID: Z68EWFD946XZRPW9, Extended Request ID: 4oF/6V21FWJZdR5osr6DcNYC6RLxEQZrNxEf3/X8ur0Tm/aeVe31aNtZXT/VU/yrG17ZMUOAZK4=):InvalidRequest: The authorization mechanism you have provided is not supported. Please use Signature Version 4. (Service: S3, Status Code: 400, Request ID: Z68EWFD946XZRPW9, Extended Request ID: 4oF/6V21FWJZdR5osr6DcNYC6RLxEQZrNxEf3/X8ur0Tm/aeVe31aNtZXT/VU/yrG17ZMUOAZK4=)

[^[[1;31mERROR^[[m] testComplexDirActions(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List)  Time elapsed: 16.345 s  <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>

[^[[1;31mERROR^[[m] testRenamePopulatesFileAncestors2(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 2.337 s  <<< FAILURE!
java.lang.AssertionError: Number of bytes copied: files_copied_bytes expected:<16384> but was:<16368>

[^[[1;31mERROR^[[m] ^[[1;31m  ITestS3AClosedFS.testClosedInstrumentation:111 [S3AInstrumentation.hasMetricSystem()] expected:<[fals]e> but was:<[tru]e>^[[m
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4235] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])

ILoadTest

mvn verify -Dtest=skip -Dit.test=ILoadTest* -Dscale > loadtest.txt

[INFO] Running org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 200.663 s - in org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.692 s - in org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials
[INFO] Running org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.984 s - in org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials

steveloughran · 2024-09-03T20:27:29Z

I fixed the assume role issue in #7021

steveloughran · 2024-09-03T20:34:33Z

regarding the others

ITestS3AConcurrentOps. hit that in HADOOP-19221. S3A: Unable to recover from failure of multipart block upload attempt #6938 where I'd assumed it was my change that did it: maybe it just guaranteed it would also surface.
ITestS3AConfiguration looks like its due to a private field change. If you can see what it was changed to we could migrate the test case -if not, it'll need cutting.
Other ones I'm unsure about -could be race conditions based on where you are, or its library changes. Assume for now that they're test regressions as some assumptions are invalid, rather than library regressions. (I think you can see why we like to isolate SDK updates from everything else...)

Can you just do a checkout of trunk with no changes, run the same tests and see what the results are?

diljotgrewal · 2024-09-04T04:54:13Z

re-ran the scale tests with and without sdk update commit. I am seeing similar failures in both. Error samples included below.

I'll rebase this branch with #7021. I'll also look into ITestS3AConfiguration tests next.

without AWS SDK update:

command: mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale > scale_8_core_old_aws

[ERROR] testCostOfCreatingMagicFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost)  Time elapsed: 2.922 s  <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0004/test/testCostOfCreatingMagicFile/__magic_job-123/__base/file.txt.pending': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"]): Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"])

[ERROR] test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob)  Time elapsed: 37.374 s  <<< FAILURE!
java.lang.AssertionError:
Job job_1725412925883_0003 failed in state FAILED with cause Job commit failed: org.apache.hadoop.fs.PathIOException: `s3a://s3atestingdsgrewal/job-00-fork-0008/test/ITestS3ACommitterMRJob-execute-magic/__magic_job-3cbf58ed-920a-48a7-9a68-2b888fb9ce11/job-3cbf58ed-920a-48a7-9a68-2b888fb9ce11/01/task_1725412925883_0003_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4270] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4270] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])

[ERROR] Tests run: 5, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 20.563 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms
[ERROR] testEncryptionOverRename(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 5.037 s  <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected expected:<kms+context> but was:<null>

[ERROR] testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 3.812 s  <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
  <[0, 1, 239, 4079]>
to contain exactly in any order:
  <[0, 1, 255, 4095]>
elements not found:
  <[255, 4095]>
and elements not expected:
  <[239, 4079]>

[ERROR] testEncryptionEnabledAndDisabledFS(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 1.837 s  <<< FAILURE!
java.lang.AssertionError: SecurityException should be thrown: "Exception should be raised if unencrypted data is read by a CSE enabled FS"

[ERROR] Tests run: 18, Failures: 0, Errors: 10, Skipped: 1, Time elapsed: 21.693 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion
[ERROR] testCentralEndpointAndNullRegionFipsWithCRUD(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)  Time elapsed: 2.665 s  <<< ERROR!
java.lang.IllegalArgumentException: Non central endpoint cannot be set when fs.s3a.endpoint.fips is true : https://s3-accesspoint.us-east-1.amazonaws.com

[ERROR] testCentralEndpoint(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)  Time elapsed: 0.948 s  <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid configuration: region from ARN `us-east-1` does not match client region `us-east-2` and UseArnRegion is `false`

[^[[1;31mERROR^[[m] testEndpointOverride(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)  Time elapsed: 0.998 s  <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid configuration: region from ARN `us-east-1` does not match client region `us-east-2` and UseArnRegion is `false`

[ERROR] testCommitterWithDuplicatedCommit[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol)  Time elapsed: 9.857 s  <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0002/test/ITestMagicCommitProtocol-testCommitterWithDuplicatedCommit[track-commit-in-memory-false]/__magic_job-job_202409040534_0002/job-job_202409040534_0002/01/task_202409040534_0002_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4331] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4331] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])

[ERROR] testMapFileOutputCommitter[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol)  Time elapsed: 11.016 s  <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0002/test/ITestMagicCommitProtocol-testMapFileOutputCommitter[track-commit-in-memory-false]/__magic_job-job_202409040503_0002/job-job_202409040503_0002/01/task_202409040503_0002_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 5604] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 5604] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])

[ERROR] testComplexDirActions(org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List)  Time elapsed: 16.843 s  <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>

[ERROR] testRenamePopulatesFileAncestors2(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 4.976 s  <<< FAILURE!
java.lang.AssertionError: Number of bytes copied: files_copied_bytes expected:<16384> but was:<16368>

[ERROR] testLargeFilesFromRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp)  Time elapsed: 5.72 s  <<< ERROR!
java.io.FileNotFoundException: File file:/home/dsgrewal/hadoop/hadoop-tools/hadoop-aws/target/test-dir/2/ITestS3AContractDistCp/testLargeFilesFromRemote/local/outputDir/inputDir/file1 does not exist

[ERROR] test_110_teragen[magic-memory=false](org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A)  Time elapsed: 20.265 s  <<< FAILURE!
java.lang.AssertionError: teragen(1000, s3a://s3atestingdsgrewal/job-00/test/terasort-magic-false/sortin) failed expected:<0> but was:<1>

[ERROR] testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)  Time elapsed: 2.27 s  <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but was:<512>

[ERROR] test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass)  Time elapsed: 0.572 s  <<< FAILURE!
org.junit.ComparisonFailure: [List file status length S3AFileStatus{path=s3a://s3atestingdsgrewal/job-00/test/tests3ascale/array/src/hugefile; isDirectory=false; length=10485744; replication=1; blocksize=33554432; modification_time=1725414230000; access_time=0; owner=dsgrewal; group=dsgrewal; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="8f247ca05cb4324d2bd3e1ec1a88e481-2" versionId=null] expected:<104857[60]L> but was:<104857[44]L>

With SDK update

command : dsgrewal@5671d360aa8a:~/hadoop/hadoop-tools/hadoop-aws$ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale > scale_8_core_new_aws

error samples:

[ERROR] testCostOfCreatingMagicFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperationCost)  Time elapsed: 2.842 s  <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0003/test/testCostOfCreatingMagicFile/__magic_job-123/__base/file.txt.pending': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"]): Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 640])
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 3629] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit["iostatistics"])

[ERROR] test_200_execute[magic](org.apache.hadoop.fs.s3a.commit.integration.ITestS3ACommitterMRJob)  Time elapsed: 37.368 s  <<< FAILURE!
java.lang.AssertionError:
Job job_1725417665118_0003 failed in state FAILED with cause Job commit failed: org.apache.hadoop.fs.PathIOException: `s3a://s3atestingdsgrewal/job-00-fork-0001/test/ITestS3ACommitterMRJob-execute-magic/__magic_job-95d105a2-9a3d-4a55-b1d6-7cade7a2b755/job-95d105a2-9a3d-4a55-b1d6-7cade7a2b755/01/task_1725417665118_0003_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4267] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4267] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])

[ERROR] testDirectoryListingFileLengths(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 3.855 s  <<< FAILURE!
java.lang.AssertionError:
[File lengths aren't the same as expected from FileStatus dir. listing]
Expecting:
  <[0, 1, 239, 4079]>
to contain exactly in any order:
  <[0, 1, 255, 4095]>
elements not found:
  <[255, 4095]>
and elements not expected:
  <[239, 4079]>

[ERROR] testEncryptionEnabledAndDisabledFS(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 1.826 s  <<< FAILURE!
java.lang.AssertionError: SecurityException should be thrown: "Exception should be raised if unencrypted data is read by a CSE enabled FS"

[ERROR] testEncryption(org.apache.hadoop.fs.s3a.ITestS3AClientSideEncryptionKms)  Time elapsed: 1.657 s  <<< FAILURE!
java.lang.AssertionError: Key wrap algo isn't same as expected expected:<kms+context> but was:<null>

[ERROR] testCentralEndpointAndNullRegionFipsWithCRUD(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)  Time elapsed: 2.916 s  <<< ERROR!
java.lang.IllegalArgumentException: Non central endpoint cannot be set when fs.s3a.endpoint.fips is true : https://s3-accesspoint.us-east-1.amazonaws.com

[ERROR] testCentralEndpoint(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)  Time elapsed: 0.967 s  <<< ERROR!
software.amazon.awssdk.core.exception.SdkClientException: Invalid configuration: region from ARN `us-east-1` does not match client region `us-east-2` and UseArnRegion is `false`

[ERROR] testCommitterWithDuplicatedCommit[track-commit-in-memory-false](org.apache.hadoop.fs.s3a.commit.magic.ITestMagicCommitProtocol)  Time elapsed: 9.582 s  <<< ERROR!
org.apache.hadoop.fs.PathIOException:
`s3a://s3atestingdsgrewal/job-00-fork-0007/test/ITestMagicCommitProtocol-testCommitterWithDuplicatedCommit[track-commit-in-memory-false]/__magic_job-job_202409040635_0007/job-job_202409040635_0007/01/task_202409040635_0007_m_000000.pendingset': Failed to read JSON file com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4326] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"]): Unexpected end-of-input in field name
 at [Source: (org.apache.hadoop.fs.FSDataInputStream); line: 1, column: 4326] (through reference chain: org.apache.hadoop.fs.s3a.commit.files.PendingSet["iostatistics"]->org.apache.hadoop.fs.statistics.IOStatisticsSnapshot["meanstatistics"])

[ERROR] testComplexDirActions(org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus)  Time elapsed: 16.914 s  <<< FAILURE!
java.lang.AssertionError: getLen expected:<512> but was:<496>

[ERROR] testLargeFilesFromRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp)  Time elapsed: 5.542 s  <<< ERROR!
java.io.FileNotFoundException: File file:/home/dsgrewal/hadoop/hadoop-tools/hadoop-aws/target/test-dir/7/ITestS3AContractDistCp/testLargeFilesFromRemote/local/outputDir/inputDir/file1 does not exist

[ERROR] test_110_teragen[magic-memory=false](org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortOnS3A)  Time elapsed: 20.716 s  <<< FAILURE!
java.lang.AssertionError: teragen(1000, s3a://s3atestingdsgrewal/job-00/test/terasort-magic-false/sortin) failed expected:<0> but was:<1>

[ERROR] test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.commit.magic.ITestS3AHugeMagicCommits)  Time elapsed: 1.798 s  <<< FAILURE!
org.junit.ComparisonFailure: [List file status length S3AFileStatus{path=s3a://s3atestingdsgrewal/job-00/test/tests3ascale/ITestS3AHugeMagicCommits/commit/commit.bin; isDirectory=false; length=10485744; replication=1; blocksize=33554432; modification_time=1725418948000; access_time=0; owner=dsgrewal; group=dsgrewal; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="2c6748a4d4a12430c65b8f7d26665ae6-2" versionId=null] expected:<104857[60]L> but was:<104857[44]L>

[ERROR] testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)  Time elapsed: 2.21 s  <<< FAILURE!
java.lang.AssertionError: Mismatch in bytes written expected:<697> but was:<512>

[ERROR] test_030_postCreationAssertions(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesStorageClass)  Time elapsed: 0.564 s  <<< FAILURE!
org.junit.ComparisonFailure: [List file status length S3AFileStatus{path=s3a://s3atestingdsgrewal/job-00/test/tests3ascale/array/src/hugefile; isDirectory=false; length=10485744; replication=1; blocksize=33554432; modification_time=1725418969000; access_time=0; owner=dsgrewal; group=dsgrewal; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="c6585c77653d07c9320355e92fef70d4-2" versionId=null] expected:<104857[60]L> but was:<104857[44]L>

steveloughran · 2024-09-04T10:41:31Z

BTW, @diljotgrewal does this new SDK fix aws/aws-sdk-java-v2#5247

This is a serious problem and nobody seems be giving it attention. It also flags to me that there seems to be more attention in the library to new features over fundamental issues. It'd be nice if you can draw attention to the SDK team about this -the more internal pressure you can generate the more likely it is that someone may actually look at it.

steveloughran · 2024-09-04T20:30:11Z

This SDK is incomplete. It lets you set the flag on put/complete multipart upload, but not on CopyObjectRequest. We need this for rename. Given copy is just a PUT with a different source, I don't see why it shouldn't be available in S3 -so it's just an SDK omission

Am I correct?

hadoop-yetus · 2024-09-05T13:24:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 53s		Maven dependency ordering for branch
-1 ❌	mvninstall	33m 9s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	17m 53s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	16m 35s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
-1 ❌	mvnsite	12m 36s	/branch-mvnsite-root.txt	root in trunk failed.
-1 ❌	javadoc	10m 7s	/branch-javadoc-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	root in trunk failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌	javadoc	8m 10s	/branch-javadoc-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt	root in trunk failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌	shadedclient	52m 48s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 3s		Maven dependency ordering for patch
-1 ❌	mvninstall	30m 43s	/patch-mvninstall-root.txt	root in the patch failed.
+1 💚	compile	16m 59s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	16m 59s		the patch passed
+1 💚	compile	16m 17s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	16m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-1 ❌	mvnsite	6m 36s	/patch-mvnsite-root.txt	root in the patch failed.
+1 💚	shellcheck	0m 0s		No new issues.
-1 ❌	javadoc	10m 32s	/patch-javadoc-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	root in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌	javadoc	8m 24s	/patch-javadoc-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt	root in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌	shadedclient	55m 21s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	756m 25s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 44s		The patch does not generate ASF License warnings.
		1037m 34s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpc

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/3/artifact/out/Dockerfile
GITHUB PR	#7015
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux bbff94579452 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ebc8913`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/3/testReport/
Max. process+thread count	4278 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7015/3/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

diljotgrewal · 2024-09-05T18:05:19Z

Just talked to the SDK team, InstanceProfileCredentialsProvider issue is not fixed with this SDK release. It is on their radar and they plan to work on it soon.

Re: incomplete SDK: can you please give me some context? Which flag is missing in CopyObjectRequest?

steveloughran · 2024-09-05T19:31:59Z

Just talked to the SDK team, InstanceProfileCredentialsProvider issue is not fixed with this SDK release. It is on their radar and they plan to work on it soon.

happy

Re: incomplete SDK: can you please give me some context? Which flag is missing in CopyObjectRequest?

I couldn't see how to set the If-none-match header on a CopyObjectRequest

steveloughran · 2024-09-13T18:23:39Z

does this address the needless logging of HADOOP-19272? if not, later one is needed and you need to do the before/after checks to make sure things are better not worse -including a rename of a 128+ MB file. Think we should add a large file upload and for anyone testing in EC2 the upload has to go to a s3 store on another continent. if you can think of anything else like this, add it to testing.md as you do it. thanks

FYI, we fear SDK updates for subtle regressions like that. They can get past the automated testing but fill up the logs -which is why we absolutely require that step of manual review of all output files of the test run, manual exploratory testing etc. we're going to have to be even more rigorous now.

steveloughran · 2024-10-09T13:13:10Z

@diljotgrewal I've created a feature branch for this work: HADOOP-19256-s3-conditional-writes

Can you upgrade to the latest SDK, do the usual qualification work and then submit a request against this branch. I don't want to do any SDK updates to trunk itself until I can come up with a even more robust qualification. Which I think will have to include auditing bits of the SDK as well to see if there are new warnings added.

github-actions bot added build trunk labels Aug 28, 2024

diljotgrewal mentioned this pull request Aug 29, 2024

HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile() #7011

Open

4 tasks

diljotgrewal closed this Aug 30, 2024

diljotgrewal force-pushed the sdk_update branch from efa0fc6 to 1655acc Compare August 30, 2024 17:45

diljotgrewal reopened this Aug 30, 2024

github-actions bot added TOOLS AWS labels Aug 30, 2024

diljotgrewal changed the title ~~HADOOP-19256: Upgrade aws sdk v2 to 2.27.14~~ HADOOP-19258: Upgrade aws sdk v2 to 2.27.14 Aug 30, 2024

diljotgrewal closed this Sep 4, 2024

diljotgrewal reopened this Sep 4, 2024

Upgrade aws sdk v2 to 2.27.14

ebc8913

diljotgrewal force-pushed the sdk_update branch from d8530d3 to ebc8913 Compare September 4, 2024 20:05

github-actions bot removed TOOLS AWS labels Sep 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19258: Upgrade aws sdk v2 to 2.27.14 #7015

HADOOP-19258: Upgrade aws sdk v2 to 2.27.14 #7015

diljotgrewal commented Aug 28, 2024

hadoop-yetus commented Aug 29, 2024

steveloughran commented Aug 30, 2024

steveloughran commented Aug 30, 2024

hadoop-yetus commented Aug 31, 2024

diljotgrewal commented Sep 3, 2024

steveloughran commented Sep 3, 2024

steveloughran commented Sep 3, 2024

diljotgrewal commented Sep 4, 2024 •

edited

Loading

steveloughran commented Sep 4, 2024

steveloughran commented Sep 4, 2024

hadoop-yetus commented Sep 5, 2024

diljotgrewal commented Sep 5, 2024

steveloughran commented Sep 5, 2024

steveloughran commented Sep 13, 2024

steveloughran commented Oct 9, 2024

HADOOP-19258: Upgrade aws sdk v2 to 2.27.14 #7015

Are you sure you want to change the base?

HADOOP-19258: Upgrade aws sdk v2 to 2.27.14 #7015

Conversation

diljotgrewal commented Aug 28, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Aug 29, 2024

steveloughran commented Aug 30, 2024

steveloughran commented Aug 30, 2024

hadoop-yetus commented Aug 31, 2024

diljotgrewal commented Sep 3, 2024

Scale Tests

adding access point:

ILoadTest

steveloughran commented Sep 3, 2024

steveloughran commented Sep 3, 2024

diljotgrewal commented Sep 4, 2024 • edited Loading

without AWS SDK update:

With SDK update

steveloughran commented Sep 4, 2024

steveloughran commented Sep 4, 2024

hadoop-yetus commented Sep 5, 2024

diljotgrewal commented Sep 5, 2024

steveloughran commented Sep 5, 2024

steveloughran commented Sep 13, 2024

steveloughran commented Oct 9, 2024

diljotgrewal commented Sep 4, 2024 •

edited

Loading