Add knob to disable slow io notifications

[oshogbo]

Motivation and Context

Adding a knob allows disabling the slow_io check on a single vdev. Some users have reported that it breaks their enterprise configuration when one or more vdevs are using fiber channels with redundancy. Another reason to disable the check might be when a vdev is damaged, but we still want to force some resilvering from it.

Description

Add a knob to disable slow I/O event generation for a single vdev.

How Has This Been Tested?

New test has been added.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

I wonder if this feature partially duplicates slow_io_n/slow_io_t? Though I may not understand their meaning.

[oshogbo]

I wonder if this feature partially duplicates slow_io_n/slow_io_t? Though I may not understand their meaning.

I see this new feature as a supplement to slow_io_n/slow_io_t. Those settings serve two purposes: they let zpool status report how many slow I/Os have occurred, and they generate events for ZED/zfsd so it can decide whether to downgrade the pool. However, sometimes you don’t want the pool downgraded, but you still might want some accounting on slow I/O.

For example, in multipath environments (as has been pointed out multiple times on X/Twitter), pools have unexpectedly been downgraded. From my point of view, this feature is useful when you know a device is failing but still want to extract as much throughput as possible during recovery.

Simply raising slow_io_n/slow_io_t to very high values isn’t a great solution, and seems like an bypass not solution — how high is “high enough”? A single knob to control this behavior makes more sense in these scenarios.

[behlendorf]

We'll need to make sure this change and #17227 work well together. Depending on what order they end up being merged in, it looks like that will mainly entail making sure the new property covers disabling slow IO events from sitout events.

[tonyhutter]

I don't know if it's related to this PR, but CI is reporting these failures for almalinux8 & almalinux10:

    FAIL events/slow_vdev_degraded_sit_out (expected PASS)

[behlendorf]

Those are almost certainly false positives which snuck in with the sitout changes. @pcd1193182 since the test isn't entirely deterministic perhaps we can do something like retry in the test case if we fail to get a sitout on the first attempt.

[behlendorf]

nit: it looks like you could put this back the way it was on a single line.

[behlendorf]

Let's add a properly formatted version of this (or similar), "Even when disabled, slow I/Os will be included in the zpool status -s output".