HDDS-11462. Enhancing DataNode I/O Monitoring Capabilities. #7206

slfan1989 · 2024-09-16T04:18:55Z

What changes were proposed in this pull request?

In this enhanced functionality, we have added the following features:

DataNode UI now supports displaying heartbeat information.

The DataNode Volume list now shows the number of containers.

DataNode includes new tabs for I/O Stats and Data Scanner.

IO Status

Data Scanner

I/O Stats now supports more detailed performance metrics.

{
    "name" : "Hadoop:service=HddsDatanode,name=VolumeIOStats-/data2/ozonedata/hddsdata",
    "modelerType" : "VolumeIOStats-/data2/ozonedata/hddsdata",
    "tag.StorageDirectory" : "/data2/ozonedata/hddsdata/hdds",
    "tag.Hostname" : "bigdata-s2688-hdp446.apache01",
    "ReadLatency60sNumOps" : 0,
    "ReadLatency60s50thPercentileLatency" : 0,
    "ReadLatency60s75thPercentileLatency" : 0,
    "ReadLatency60s90thPercentileLatency" : 0,
    "ReadLatency60s95thPercentileLatency" : 0,
    "ReadLatency60s99thPercentileLatency" : 0,
    "WriteLatency60sNumOps" : 0,
    "WriteLatency60s50thPercentileLatency" : 0,
    "WriteLatency60s75thPercentileLatency" : 0,
    "WriteLatency60s90thPercentileLatency" : 0,
    "WriteLatency60s95thPercentileLatency" : 0,
    "WriteLatency60s99thPercentileLatency" : 0,
    "ReadBytes" : 128011956,
    "ReadOpCount" : 131,
    "ReadTimeNumOps" : 131,
    "ReadTimeAvgTime" : 1.3125,
    "WriteBytes" : 0,
    "WriteOpCount" : 0,
    "WriteTimeNumOps" : 0,
    "WriteTimeAvgTime" : 0.0
  }

Optimized the display of JVM information in the DataNode UI.

What is the link to the Apache JIRA

JIRA: HDDS-11462. Enhancing DataNode I/O Monitoring Capabilities.

How was this patch tested?

Junit Test.

errose28

Thanks for working on this @slfan1989. I only reviewed the metrics changes for now since the web UI changes are not something I'm too familiar with.

errose28 · 2024-09-17T17:03:43Z

hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java

      clientProtocolServer = new HddsDatanodeClientProtocolServer(
          datanodeDetails, conf, HddsVersionInfo.HDDS_VERSION_INFO,
          reconfigurationHandler);

+      serviceRuntimeInfo.setRpcPort(String.valueOf(clientProtocolServer.getClientRpcAddress().getPort()));


The datanode has multiple RPC ports for various purposes, see here. This one is specifically the one used by the client for config reload. Do other components have an RPC port metric, and do they add more information about what this port is used for?

Thank you very much for your question!

I hope to present the RPC port needed for client connections, which might be more appropriately named ClientRpcPort. I should add some comments to make the code clearer.

OM displays the RpcPort (9862) in JMX.

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMMXBean.java

Lines 28 to 33 in d3899d2

@InterfaceAudience.Private

public interface OMMXBean extends ServiceRuntimeInfo {

String getRpcPort();

List<List<String>> getRatisRoles();

SCM currently displays DatanodeRpcPort (9861) and ClientRpcPort (9860) in JMX.

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java

Lines 34 to 50 in d3899d2

/**

* Get the SCM RPC server port that used to listen to datanode requests.

* @return SCM datanode RPC server port

*/

String getDatanodeRpcPort();

/**

* Get the SCM RPC server port that used to listen to client requests.

* @return SCM client RPC server port

*/

String getClientRpcPort();

/**

* Returns safe mode status.

* @return boolean

*/

boolean isInSafeMode();

errose28 · 2024-09-17T17:06:45Z

hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java

@@ -303,15 +304,17 @@ public void start() {
          datanodeDetails.setPort(DatanodeDetails.newPort(HTTPS,
                  httpServer.getHttpsAddress().getPort()));
        }
+        serviceRuntimeInfo.setHttpPort(String.valueOf(httpServer.getHttpAddress().getPort()));


What about the HTTPS port and address? are other components publishing these as separate metrics?

You are correct. we should add the corresponding port metric for HTTPS. As for the hostname, I believe we can have HTTP and HTTPS share the same one, since they are theoretically consistent.

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java

errose28 · 2024-09-17T17:21:20Z

...service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeInfoMetrics.java

+  public void setContainers(long count) {
+    this.containers = count;
+  }


This class currently pulls most of its metrics from the volume field and does not depend on external setters except for dbCompactLatency. Can we do the same for this metric?

I came up with a solution. we will inject the ContainerController into the HddsVolume to make it easier to retrieve the container count on each disk. What do you think about this approach?

errose28 · 2024-09-17T17:32:28Z

...rc/main/java/org/apache/hadoop/ozone/container/ozoneimpl/BackgroundContainerDataScanner.java

@@ -110,9 +112,19 @@ public void scanContainer(Container<?> c)

  @Override
  public Iterator<Container<?>> getContainerIterator() {
+    recordContainersMetric();


This is only called at the beginning of a background scan, which on dense volumes may happen once every few weeks. Real time updates as containers are moved, created, deleted, should be plugged directly into the ContainerSet.

It was very helpful and helped me better understand this part of the code. I’ve improved the code. If you have time, please take a look.

slfan1989 · 2024-09-18T05:39:03Z

Thanks for working on this @slfan1989. I only reviewed the metrics changes for now since the web UI changes are not something I'm too familiar with.

@errose28 Thank you very much for reviewing the code! I will make improvements to it as soon as possible.

slfan1989 · 2024-10-04T13:20:10Z

@errose28 I have made some modifications to the PR code. Could you please take some time to review it again? Thank you very much!

slfan1989 · 2024-10-13T07:53:00Z

@ivandika3 This pr involves some changes to the display of DN. I'm wondering if you could help with the review?
I want to keep pushing this PR forward.

cc: @errose28

ivandika3

@slfan1989 Thanks for the patch. Overall LGTM. Left some comments.

hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn-overview.html

hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn.js

hadoop-hdds/framework/src/main/resources/webapps/static/templates/overview.html

hadoop-hdds/framework/src/main/resources/webapps/static/ozone.js

hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn.js

slfan1989 · 2024-10-17T06:24:28Z

@slfan1989 Thanks for the patch. Overall LGTM. Left some comments.

@ivandika3 Thank you very much for helping to review the code! I will improve it as soon as possible based on your suggestions.

ivandika3

@slfan1989 Thanks for the update. LGTM +1.

Please help to do final verification on the pages and update the screenshot in the description page.

hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn-overview.html

hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn.js

slfan1989 · 2024-11-07T23:50:28Z

@slfan1989 Thanks for the update. LGTM +1.

Please help to do final verification on the pages and update the screenshot in the description page.

@ivandika3 Thank you very much for reviewing this PR. I will prepare the relevant screenshots as soon as possible.

Overview

IOStatus

Data Scanner

ivandika3 · 2024-11-08T05:54:39Z

Thanks @slfan1989 for the patch and @errose28 for the review.

slfan1989 · 2024-11-09T09:48:55Z

@ivandika3 @errose28 Thank you very much for the review!

slfan1989 marked this pull request as ready for review September 16, 2024 11:22

errose28 requested review from kerneltime and errose28 September 16, 2024 16:38

errose28 reviewed Sep 17, 2024

View reviewed changes

slfan1989 requested a review from errose28 October 3, 2024 14:50

ivandika3 self-requested a review October 16, 2024 07:00

ivandika3 reviewed Oct 16, 2024

View reviewed changes

HDDS-11462. Enhancing DataNode I/O Monitoring Capabilities.

963f157

slfan1989 force-pushed the HDDS-11462 branch from 7ba9075 to 963f157 Compare October 23, 2024 09:05

Merge branch 'apache:master' into HDDS-11462

a96d77b

adoroszlai added the metrics label Nov 6, 2024

HDDS-11462. Improve Some Code.

c4100ed

ivandika3 approved these changes Nov 7, 2024

View reviewed changes

hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn-overview.html Show resolved Hide resolved

hadoop-hdds/container-service/src/main/resources/webapps/hddsDatanode/dn.js Show resolved Hide resolved

ivandika3 merged commit 4e603aa into apache:master Nov 8, 2024
40 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-11462. Enhancing DataNode I/O Monitoring Capabilities. #7206

HDDS-11462. Enhancing DataNode I/O Monitoring Capabilities. #7206

slfan1989 commented Sep 16, 2024 •

edited

Loading

errose28 left a comment

errose28 Sep 17, 2024

slfan1989 Sep 19, 2024

errose28 Sep 17, 2024

slfan1989 Sep 19, 2024

errose28 Sep 17, 2024

slfan1989 Sep 23, 2024

errose28 Sep 17, 2024

slfan1989 Sep 23, 2024

slfan1989 commented Sep 18, 2024

slfan1989 commented Oct 4, 2024

slfan1989 commented Oct 13, 2024

ivandika3 left a comment

slfan1989 commented Oct 17, 2024

ivandika3 left a comment

slfan1989 commented Nov 7, 2024 •

edited

Loading

ivandika3 commented Nov 8, 2024

slfan1989 commented Nov 9, 2024

	@InterfaceAudience.Private
	public interface OMMXBean extends ServiceRuntimeInfo {

	String getRpcPort();

	List<List<String>> getRatisRoles();

	/**
	* Get the SCM RPC server port that used to listen to datanode requests.
	* @return SCM datanode RPC server port
	*/
	String getDatanodeRpcPort();

	/**
	* Get the SCM RPC server port that used to listen to client requests.
	* @return SCM client RPC server port
	*/
	String getClientRpcPort();

	/**
	* Returns safe mode status.
	* @return boolean
	*/
	boolean isInSafeMode();

HDDS-11462. Enhancing DataNode I/O Monitoring Capabilities. #7206

HDDS-11462. Enhancing DataNode I/O Monitoring Capabilities. #7206

Conversation

slfan1989 commented Sep 16, 2024 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

errose28 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slfan1989 commented Sep 18, 2024

slfan1989 commented Oct 4, 2024

slfan1989 commented Oct 13, 2024

ivandika3 left a comment

Choose a reason for hiding this comment

slfan1989 commented Oct 17, 2024

ivandika3 left a comment

Choose a reason for hiding this comment

slfan1989 commented Nov 7, 2024 • edited Loading

ivandika3 commented Nov 8, 2024

slfan1989 commented Nov 9, 2024

slfan1989 commented Sep 16, 2024 •

edited

Loading

slfan1989 commented Nov 7, 2024 •

edited

Loading