ON-14783: Use additional NIC flags for X3522

[ibeecraft-amd]

Running with Warnings enabled on X3 you get output warning that the NIC is not running low-latency firmware.

This commit simply suppresses the warning on X3 NICs.

Without commit on X3522:

ZF_ATTR="log_level=0x2;interface=ens1f0np0" ./zfsink 224.1.1.1:9001
  1000 | Interface ens1f0np0 is not in low latency mode.
  1000 | Low latency mode is recommended for best latency with TCPDirect.
  1000 | PIO not supported by efct interface but pio=3. Not attempting to allocate PIO buffer.
Polling reactor

With commit on X3522:

ZF_ATTR="log_level=0x2;interface=ens1f0np0" ./zfsink 224.1.1.1:9001
  1000 | PIO not supported by efct interface but pio=3. Not attempting to allocate PIO buffer.
Polling reactor

With Commit on X2522 running full-feature firmware:

ZF_ATTR="log_level=0x2;interface=enp131s0f0" ./zfsink 224.1.1.1:9001
  1000 | Interface enp131s0f0 is not in low latency mode.
  1000 | Low latency mode is recommended for best latency with TCPDirect.
Polling reactor

With commit on X2522 running ultra-low-latency firmware:

ZF_ATTR="log_level=0x2;interface=enp131s0f0" ./zfsink 224.1.1.1:^C01
sudo ip l set enp129s0f0 up
ZF_ATTR="log_level=0x2;interface=enp129s0f0" ./zfsink 224.1.1.1:9001
Polling reactor

[sianj-xilinx]

This should be based on EF_VI_CAP_RX_REF.

[ibeecraft-amd]

This was based against Onload-9 branch which does not have this capability I can update it and rebase to master.

[ibeecraft-amd]

If the following commit is merged to Onload-9 branch happy to update this: Xilinx-CNS/onload@7764b68

[ligallag-amd]

I'm confused in this case. You've set pio_available to 0, but we still try ef_pio_alloc if !(attr->pio >= PIO_MUST_USE). I think we may as well exit here if ef_pio_alloc will always fail for x3 - which it should right? That said, I see that you're trying to preserve previous behaviour so maybe you can keep it as is

[ibeecraft-amd]

Yes sorry, I pushed it but hadn't finished working on this. Either way I see two ways of proceeding with this:

1. Remove the setting of pio_available as it is not used
2. Change the code to use this new capability check rather than try allocating the pio buffer (hesitant to do so as although should provide the same end result uses different function calls)

[ligallag-amd]

I don't mind either option. If we always fail ef_pio_alloc when the PIO capability is not supported, I would rather we not make the additional call. But it doesn't matter that we do, it's just more work.

Slightly related, this error message only seems to refer to the -ENOSYS case - "the API does not know how to retrieve support for the supplied capability". I think this may be a tad misleading when reporting -EOPNOTSUPP. You have reported the rc which would dispel some confusion.