[Matrix] The vis does not allow to distinguish numeric values that are very close

[loichuder]

Is your feature request related to a problem?

I happened to find a dataset (link below) with only 2 values that are extremely close numerically but yet different. When looking at the Line vis (see screenshots below), I see that the two values are different (the line is not horizontal).

But the numeric precision of the tooltip and of the Matrix vis does not allow me to see this difference.

Requested solution or feature

Not sure. Perhaps a way of specifying the number of digits in fixed-point notation as suggested earlier in #1175 ?

Alternatives you've considered

Export as CSV and reading the values there but that is really convoluted.

Additional context

Dataset:

/1.1/instrument/positioners/obpitch in strain.hdf5

Screenshots

Line vis:

Matrix vis:

[axelboc]

I'd be okay with adding a "Precision" field in the toolbar. Only problem I see is that we would have to pick a default precision that works for all notations out of the box. Currently we use the following numeric formats:

• Fixed-point: .3f (or .2f for complex members)
• Scientific: .3e (or .3~e for complex members)
• Auto: .5~g (or .3~g for complex members)

[axelboc]

See also #1742, which I've just closed as duplicate.

Here is an overview of our current notations:

Raw number	Auto (.5~g)	Scientific (.3e)	Fixed-point (.3f)
11111111111	"1.1111e+10"	"1.111e+10"	"11111111111.000"
999999	"1e+6"	"1.000e+6"	"999999.000"
99999.9	"1e+5"	"1.000e+5"	"99999.900"
99999	"99999"	"1.000e+5"	"99999.000"
9999.99	"10000"	"1.000e+4"	"9999.990"
9999.9999	"10000"	"1.000e+4"	"10000.000"
9999.0009	"9999"	"9.999e+3"	"9999.001"
1.99999	"2"	"2.000e+0"	"2.000"
1.9999	"1.9999"	"2.000e+0"	"2.000"
1.999	"1.999"	"1.999e+0"	"1.999"
1.00009	"1.0001"	"1.000e+0"	"1.000"
1	"1"	"1.000e+0"	"1.000"
0.1	"0.1"	"1.000e-1"	"0.100"
0.0001	"0.0001"	"1.000e-4"	"0.000"
0.0000019999	"0.0000019999"	"2.000e-6"	"0.000"
0.000000199999	"2e-7"	"2.000e-7"	"0.000"
0.00000019999	"1.9999e-7"	"2.000e-7"	"0.000"
0.0000001	"1e-7"	"1.000e-7"	"0.000"

With #1745, when the underlying dtype of the data is IntegerType and either the Auto or Fixed-point notation is selected, numbers are simply cast to string. So 11111111111 is now displayed as "11111111111" in this situation.

• Auto (.5~g) is meant as a good default for readability (even though the number of significant digits is not respected with small numbers like 0.0000019999...)
• Scientific (.3e) guarantees an almost-fixed amount of characters (only the length of the exponent can vary), which can be convenient in some cases, or a bit unreadable in others (e.g. with numbers that could be displayed exactly with fewer characters and unnecessary zeros, like 1).
• Fixed-point (.3f) seems pretty pointless as per this issue and MatrixVis: customisable fixed-point precision #1742. Giving users the ability to increase the fixed-point precision would definitely make this notation more useful, but the user need stated in the latter issue ("We work with frequency metrology, at least 12 decimal digits should ideally be available.") makes me think that what might be needed is in fact an exact notation.

Would it make sense to replace the Fixed-point notation with an Exact notation that would just call toString() like in the Scalar vis, even for floats? I feel like it would provide a better UX than to have to switch to the Fixed-point notation and increase the "precision" to e.g. 12 using another toolbar control (which would also ad clutter).

Other (or complementary) idea: remove the Fixed-point notation altogether and instead, show exact numbers in a tooltip on hover (probably with the abbr tag).

[loichuder]

Would it make sense to replace the Fixed-point notation with an Exact notation that would just call toString() like in the Scalar vis, even for floats?

Worth a try. Let's settle this tomorrow.