Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-17609. [FGL] Fix lock mode in some RPCs. #7037

Open
wants to merge 3 commits into
base: HDFS-17384
Choose a base branch
from

Conversation

hfutatzhanghb
Copy link
Contributor

@hfutatzhanghb hfutatzhanghb commented Sep 10, 2024

Description of PR

Fix lock mode in some RPCs and add some assert statement.

  • FSNamesystem#getFilesBlockingDecom : use GLOBAL mode rather than FS mode because of below codes:
for (DatanodeDescriptor dataNode :
        blockManager.getDatanodeManager().getDatanodes()) {

}
  • FSNamesystem#listOpenFiles: use GLOBAL mode because it calls getFilesBlockingDecom method.
  • FSNamesystem#getContentSummary: use GLOBAL mode rather than FS mode because it calls computeFileSize method.
  • BlockManagerSafeMode#leaveSafeMode: use GLOBAL mode rather than BM mode because it calls startSecretManagerIfNecessary which depended upon FS lock. Change to GLOBAL mode is safe here.

@hfutatzhanghb hfutatzhanghb changed the title HDFS-17609. [FGL] Fix lock mode in some RPC. HDFS-17609. [FGL] Fix lock mode in some RPCs. Sep 10, 2024
@hfutatzhanghb
Copy link
Contributor Author

@ZanderXu @ferhui Sir, PTAL if you have free time~ Thanks a lot.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 12m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ HDFS-17384 Compile Tests _
-1 ❌ mvninstall 5m 53s /branch-mvninstall-root.txt root in HDFS-17384 failed.
-1 ❌ compile 1m 35s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in HDFS-17384 failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ compile 0m 44s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in HDFS-17384 failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
+1 💚 checkstyle 1m 9s HDFS-17384 passed
-1 ❌ mvnsite 0m 49s /branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in HDFS-17384 failed.
-1 ❌ javadoc 0m 43s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in HDFS-17384 failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javadoc 0m 35s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in HDFS-17384 failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ spotbugs 0m 45s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in HDFS-17384 failed.
-1 ❌ shadedclient 5m 2s branch has errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 44s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 48s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javac 0m 48s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ compile 0m 43s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ javac 0m 43s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 0s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 215 unchanged - 0 fixed = 216 total (was 215)
-1 ❌ mvnsite 0m 45s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ javadoc 0m 36s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javadoc 0m 37s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ spotbugs 0m 44s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ shadedclient 6m 47s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 45s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
37m 8s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/1/artifact/out/Dockerfile
GITHUB PR #7037
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux d1f94db65e65 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HDFS-17384 / a0e373b
Default Java Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/1/testReport/
Max. process+thread count 88 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 5m 7s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ HDFS-17384 Compile Tests _
-1 ❌ mvninstall 5m 55s /branch-mvninstall-root.txt root in HDFS-17384 failed.
-1 ❌ compile 1m 34s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in HDFS-17384 failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ compile 0m 42s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in HDFS-17384 failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
+1 💚 checkstyle 1m 9s HDFS-17384 passed
-1 ❌ mvnsite 0m 49s /branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in HDFS-17384 failed.
-1 ❌ javadoc 0m 42s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in HDFS-17384 failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javadoc 0m 37s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in HDFS-17384 failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ spotbugs 0m 46s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in HDFS-17384 failed.
-1 ❌ shadedclient 5m 3s branch has errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 44s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 46s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javac 0m 46s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ compile 0m 47s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ javac 0m 47s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 4s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 215 unchanged - 0 fixed = 216 total (was 215)
-1 ❌ mvnsite 0m 46s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ javadoc 0m 36s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javadoc 0m 35s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ spotbugs 0m 44s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ shadedclient 6m 49s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 45s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 30s The patch does not generate ASF License warnings.
30m 18s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/2/artifact/out/Dockerfile
GITHUB PR #7037
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux cd897e0e39e4 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HDFS-17384 / a0e373b
Default Java Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/2/testReport/
Max. process+thread count 88 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/2/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 32s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ HDFS-17384 Compile Tests _
-1 ❌ mvninstall 5m 54s /branch-mvninstall-root.txt root in HDFS-17384 failed.
-1 ❌ compile 1m 34s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in HDFS-17384 failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ compile 0m 41s /branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in HDFS-17384 failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
+1 💚 checkstyle 1m 6s HDFS-17384 passed
-1 ❌ mvnsite 0m 47s /branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in HDFS-17384 failed.
-1 ❌ javadoc 0m 44s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in HDFS-17384 failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javadoc 0m 35s /branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in HDFS-17384 failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ spotbugs 0m 43s /branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in HDFS-17384 failed.
-1 ❌ shadedclient 4m 55s branch has errors when building and testing our client artifacts.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 41s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ compile 0m 44s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javac 0m 44s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ compile 0m 42s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ javac 0m 42s /patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 59s the patch passed
-1 ❌ mvnsite 0m 42s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ javadoc 0m 36s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌ javadoc 0m 37s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ spotbugs 0m 43s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ shadedclient 6m 42s patch has errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 0m 43s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
24m 44s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/Dockerfile
GITHUB PR #7037
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 2e83c811bde4 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision HDFS-17384 / fcd3294
Default Java Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/testReport/
Max. process+thread count 92 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants