Enable Top Clients/Files Metrics in Harvest

[rahulguptajss]

Top Clients collection is disabled by default. To enable Top Clients Tracking in Harvest, follow these steps. Note that Top Clients support is available only through the RestPerf Collector in ONTAP version 9.13.1 and later. The collection is enabled in the RestPerf Collector Volume template via a plugin VolumeTopClients, as detailed below.

Note: Available from Harvest 24.11.

Note: Top File Metrics are available in Harvest from 24.11.1 via same plugin. Refer here for more details.

NABox4 and Harvest 25.05

⚠️ Note:
Prerequisite: ssh to NABox via admin
Create relevant directories if they don't exist. You need to run these commands in sudo bash

1. Navigate to the restperf directory.

cd /etc/nabox/harvest/user/restperf

Create custom.yaml:

vi custom.yaml

Update the file with the following content:

objects:
  Volume: custom_volume.yaml

2. Create a custom_volume.yaml file in the /etc/nabox/harvest/user/restperf/9.12.0 directory. Update custom_volume.yaml as below.

name:                     Volume
query:                    api/cluster/counter/tables/volume
object:                   volume

counters:
  - ^^uuid
  - ^name                  => volume
  - ^node.name             => node
  - ^parent_aggregate      => aggr
  - ^svm.name              => svm
  - average_latency        => avg_latency
  - bytes_read             => read_data
  - bytes_written          => write_data
  - nfs.access_latency     => nfs_access_latency
  - nfs.access_ops         => nfs_access_ops
  - nfs.getattr_latency    => nfs_getattr_latency
  - nfs.getattr_ops        => nfs_getattr_ops
  - nfs.lookup_latency     => nfs_lookup_latency
  - nfs.lookup_ops         => nfs_lookup_ops
  - nfs.other_latency      => nfs_other_latency
  - nfs.other_ops          => nfs_other_ops
  - nfs.punch_hole_latency => nfs_punch_hole_latency
  - nfs.punch_hole_ops     => nfs_punch_hole_ops
  - nfs.read_latency       => nfs_read_latency
  - nfs.read_ops           => nfs_read_ops
  - nfs.setattr_latency    => nfs_setattr_latency
  - nfs.setattr_ops        => nfs_setattr_ops
  - nfs.total_ops          => nfs_total_ops
  - nfs.write_latency      => nfs_write_latency
  - nfs.write_ops          => nfs_write_ops
  - other_latency
  - read_latency
  - total_ops
  - total_other_ops        => other_ops
  - total_read_ops         => read_ops
  - total_write_ops        => write_ops
  - write_latency

plugins:
  - MetricAgent:
      compute_metric:
        - total_data ADD bytes_read bytes_written
  - Aggregator:
      # plugin will create summary/average for each object
      # any names after the object names will be treated as label names that will be added to instances
      - node
  - Volume:
      include_constituents: false

  - LabelAgent:
      # Ignore transient volumes, e.g. SnapProtect, SnapManager, SnapCenter, CommVault, Clone, and Metadata volumes
      exclude_regex:
        - volume `.+_CVclone`
        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
        - volume `sdw_cl_.+`
        - volume `MDV_CRS_.+`
        - volume `MDV_aud_.+`

  - VolumeTopClients:
      # `max_volumes` is the maximum number of volumes to consider for top client metrics. This value is capped at 50, even if a higher number is specified.
      # When enabled, this plugin will collect read/write operations and throughput metrics for the top clients of each volume.
      - schedule:
          - data: 1h # This value should be a multiple of the poll duration. By default, Harvest will check once an hour to see how many volumes have activity_tracking.state set to on.
      - max_volumes: 5
      - objects:
          - client  # collect read/write operations and throughput metrics for the top clients.
          - file    # collect read/write operations and throughput metrics for the top files.

export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume

3. Restart harvest

rpm/deb/native/container

Note:

Below steps are applicable for rpm/deb/native/container installation. In this example, let's assume that Harvest is installed in /opt/harvest

1. Navigate to the restperf directory:

cd /opt/harvest/conf/restperf

Create custom.yaml:

vi custom.yaml

Update the file with the following content:

objects:
  Volume: custom_volume.yaml

2. Create a custom_volume.yaml file in the /opt/harvest/conf/restperf/9.12.0/ directory. Update custom_volume.yaml as below.

name:                     Volume
query:                    api/cluster/counter/tables/volume
object:                   volume

counters:
  - ^^uuid
  - ^name                  => volume
  - ^node.name             => node
  - ^parent_aggregate      => aggr
  - ^svm.name              => svm
  - average_latency        => avg_latency
  - bytes_read             => read_data
  - bytes_written          => write_data
  - nfs.access_latency     => nfs_access_latency
  - nfs.access_ops         => nfs_access_ops
  - nfs.getattr_latency    => nfs_getattr_latency
  - nfs.getattr_ops        => nfs_getattr_ops
  - nfs.lookup_latency     => nfs_lookup_latency
  - nfs.lookup_ops         => nfs_lookup_ops
  - nfs.other_latency      => nfs_other_latency
  - nfs.other_ops          => nfs_other_ops
  - nfs.punch_hole_latency => nfs_punch_hole_latency
  - nfs.punch_hole_ops     => nfs_punch_hole_ops
  - nfs.read_latency       => nfs_read_latency
  - nfs.read_ops           => nfs_read_ops
  - nfs.setattr_latency    => nfs_setattr_latency
  - nfs.setattr_ops        => nfs_setattr_ops
  - nfs.total_ops          => nfs_total_ops
  - nfs.write_latency      => nfs_write_latency
  - nfs.write_ops          => nfs_write_ops
  - other_latency
  - read_latency
  - total_ops
  - total_other_ops        => other_ops
  - total_read_ops         => read_ops
  - total_write_ops        => write_ops
  - write_latency

plugins:
  - MetricAgent:
      compute_metric:
        - total_data ADD bytes_read bytes_written
  - Aggregator:
      # plugin will create summary/average for each object
      # any names after the object names will be treated as label names that will be added to instances
      - node
  - Volume:
      include_constituents: false

  - VolumeTopClients:
      # The maximum number of volumes to consider for top client metrics.
      # The actual maximum value is capped at 50, even if a higher number is specified.
      # When enabled, this plugin will collect read/write operations and throughput metrics for the top clients of each volume.
      - schedule:
          - data: 1h  # This value should be a multiple of the poll duration. By default, Harvest will check once an hour to see how many volumes have activity_tracking.state set to on.
      - max_volumes: 5


export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume

5. Restart harvest

NABox3

⚠️ Note:
Prerequisite: ssh to NABox3 via root
Create relevant directories if they don't exist.

Note: These steps are for NABox3.

Prerequisite: ssh to NABox3 via root

1. Create custom_volume.yaml

Navigate to the restperf directory:

cd /opt/harvest2-conf/conf/restperf

Create custom_volume.yaml:

vi custom_volume.yaml

Update the file with the following content:

objects:
  Volume:
    - custom_volume.yaml

2. Create custom_volume.yaml

Navigate to the 9.12.0 directory:

cd /opt/harvest2-conf/conf/restperf/9.12.0

Create custom_volume.yaml:

vi custom_volume.yaml

Update the file with the following content:

name:                     Volume
query:                    api/cluster/counter/tables/volume
object:                   volume

counters:
  - ^^uuid
  - ^name                  => volume
  - ^node.name             => node
  - ^parent_aggregate      => aggr
  - ^svm.name              => svm
  - average_latency        => avg_latency
  - bytes_read             => read_data
  - bytes_written          => write_data
  - nfs.access_latency     => nfs_access_latency
  - nfs.access_ops         => nfs_access_ops
  - nfs.getattr_latency    => nfs_getattr_latency
  - nfs.getattr_ops        => nfs_getattr_ops
  - nfs.lookup_latency     => nfs_lookup_latency
  - nfs.lookup_ops         => nfs_lookup_ops
  - nfs.other_latency      => nfs_other_latency
  - nfs.other_ops          => nfs_other_ops
  - nfs.punch_hole_latency => nfs_punch_hole_latency
  - nfs.punch_hole_ops     => nfs_punch_hole_ops
  - nfs.read_latency       => nfs_read_latency
  - nfs.read_ops           => nfs_read_ops
  - nfs.setattr_latency    => nfs_setattr_latency
  - nfs.setattr_ops        => nfs_setattr_ops
  - nfs.total_ops          => nfs_total_ops
  - nfs.write_latency      => nfs_write_latency
  - nfs.write_ops          => nfs_write_ops
  - other_latency
  - read_latency
  - total_ops
  - total_other_ops        => other_ops
  - total_read_ops         => read_ops
  - total_write_ops        => write_ops
  - write_latency

plugins:
  - MetricAgent:
      compute_metric:
        - total_data ADD bytes_read bytes_written
  - Aggregator:
      # plugin will create summary/average for each object
      # any names after the object names will be treated as label names that will be added to instances
      - node
  - Volume:
      include_constituents: false

  - LabelAgent:
      # Ignore transient volumes, e.g. SnapProtect, SnapManager, SnapCenter, CommVault, Clone, and Metadata volumes
      exclude_regex:
        - volume `.+_CVclone`
        - volume `.+(0[1-9]|[12][0-9]|3[01])(0[1-9]|1[012])\d\d[0-9]{6}`
        - volume `cl_.+_(19|20)\d\d(0[1-9]|1[012])( 0[1-9]|[12][0-9]|3[01])[0-9]{6}`
        - volume `sdw_cl_.+`
        - volume `MDV_CRS_.+`
        - volume `MDV_aud_.+`

  - VolumeTopClients:
     # `max_volumes` is the maximum number of volumes to consider for top client metrics. This value is capped at 50, even if a higher number is specified.
     # When enabled, this plugin will collect read/write operations and throughput metrics for the top clients of each volume.
     - schedule:
         - data: 1h # This value should be a multiple of the poll duration. By default, Harvest will check once an hour to see how many volumes have activity_tracking.state set to on.
     - max_volumes: 5
     - objects:
         - client  # collect read/write operations and throughput metrics for the top clients.
         - file    # collect read/write operations and throughput metrics for the top files.

export_options:
  instance_keys:
    - aggr
    - node
    - style
    - svm
    - volume

3. Restart NABox containers

dc restart

In the volume dashboard under the clients panel, you should see something similar to the following:

[mamoep]

How are the volumes chosen? Let's say I have 20 volumes and set a limit of 5. Which volumes will be reported?

[rahulguptajss]

For example, for read IOPS, we will select the top 5 volumes based on descending order of read IOPS in each performance poll. The same approach applies to write IOPS, as well as read and write throughput.

[WayneShen2]

I have enable it, but I can not see any data. Have any configure must enable from Filer end (ex: Activity Tracking)

[rahulguptajss]

I have enable it, but I can not see any data. Have any configure must enable from Filer end (ex: Activity Tracking)

Yes activity tracking must be enabled for relevant volumes.

[WayneShen2]

Thanks for your confirmed.

[wooyoungAhn]

I can not see any data..

time=2024-11-21T05:33:51.153Z level=ERROR source=collector.go:483 msg="" Poller=CVO collector=RestPerf:Volume error="failed to fetchAll href=api/storage/volumes/*/top-metrics/clients..., hrefLength=168 err=error making request StatusCode: 400, Message: Volume wildcard queries are not supported for activity tracking reports., Code: 124518417, API: /api/storage/volumes/*/top-metrics/clients?fields=client_ip%2Ciops.write%2Csvm%2Cvolume.name&max_records=500&return_records=true&svm=svm_CVO1&top_metric=iops.write&volume=vol2" plugin=VolumeTopClients

[rahulguptajss]

Could you share ONTAP version?

[rahulguptajss]

I have updated the instructions to specify that ONTAP version 9.13.1 or higher is required for TopClients to function. This is because the wildcard queries used in TopClients are only supported starting from version 9.13.1 onwards.

[kwinkelman-bw]

Is there any intent to expose this configuration in the NAbox UI? It seems like a "feature" and not necessarily a "customization" to me.

[faguayot]

Hello,

how can we enable this functionality only for some cluster which has a 9.13.1 version or onwards. Or the harvest only use this configuration for those clusters?

[rahulguptajss]

@faguayot If you want to apply the configuration to a specific cluster, you can follow the steps outlined above with a slight modification. Instead of creating a custom.yaml file, create a limited.yaml file with the same content in same location. Then, in the poller collectors configuration, use the following setup for the RestPerf collector. Define other collectors as needed.

dc-1:
  datacenter: u2
  addr: 1.1.1.1
  username: user
  password: pass
  collectors:
    - RestPerf:
        - default.yaml
        - limited.yaml

By doing this, only the specified cluster will use this modified template.

[leejshades]

flexvolume is work,flexgroup is not work