[ENH] Warning when fh range does not start at 1

[mabuimo]

The use of a range for the fh argument of the splitter is problematic and not transparent to the user as it allows the range to begin at 0 without any warning. Starting at 0 will include the last point of the training set in the test. It is very easy for the user to forget to explicitly declare 1 as the beginning of the range.

Example:

benchmark2 = ForecastingBenchmark()

scorers = [MeanSquaredError(square_root=True)]
single_fold_splitter = SingleWindowSplitter(window_length = 52, fh=range(1,5))
 # Starting at 0 would include the last value of the in-sample data (training set) in the test set (!)


benchmark2.add_task(
        y_sample_sorted,
        single_fold_splitter,
        scorers
    )

benchmark2.add_estimator(
   ChronosForecaster("amazon/chronos-bolt-tiny"), estimator_id="chronos_bolt"
)
benchmark2.add_estimator(
   ChronosForecaster("amazon/chronos-bolt-base"), estimator_id="chronos_base"
)
benchmark2.add_estimator(
   TinyTimeMixerForecaster()
)
benchmark2.add_estimator(
   MOIRAIForecaster(checkpoint_path=f"sktime/moirai-1.0-R-small"), estimator_id="moirai_small"
)
benchmark2.add_estimator(
   MOIRAIForecaster(checkpoint_path=f"sktime/moirai-1.0-R-large"), estimator_id="moirai_large"
)
results2_df = benchmark2.run(local_output_path2)

The proposed enhancement is to display a warning as a minimum, or implement a hard restriction so that starting the range at 0 is not allowed to enforeced best practices in cross-validation.

This issue is related to:
#4716
sktime/tutorial_haicon_prologue_day#1

[jgyasu]

Hmm, I agree with the confusion that this might cause. I am pasting the suggestion by @fkiraly on Discord to continue the discussion here,

In the above conversation, I can detect two roots of the problem:

accidental use of range in the benchmarking tutorial in one place, instead of range(1, horizon + 1)
generally, the "+1" is clunky and confusing.

Suggested actions:

• fix the wrong use of range in the tutorial

• potentially: define a fh(h:int) function, that can be used as fh=fh(5) and returns the correct range

debatable: change coercion of int so it is same as"fh(5)" and not [5]

[fkiraly]

The important point to discuss is, what if the user intentionally passes "0"? This would also result in the warning.

[jgyasu]

What are the scenarios when a user would intentionally like to have in-sample prediction?

[fkiraly]

What are the scenarios when a user would intentionally like to have in-sample prediction?

• denoising a signal
• specific types of tuning
• imputation, or fitting imputers
• boosting, or fitting boosting algorithms

More generally, this is a valid use case, as indicated also by the capability capability:insample, or the fact that many algorithms can do this.

[fkiraly]

@mabuimo , @jgyasu, what do you think about this idea: we replace range(1, 37) by use of a custom shorthand upto(36).

#8531

[mabuimo]

How would selecting a custom range (e.g. 4 to 8), or a single value work? Thanks

[fkiraly]

How would selecting a custom range (e.g. 4 to 8), or a single value work? Thanks

List or iterable based fh would still work, for instance:

• custom range would be [4, 5, 6, 7, 8], or range(4, 7)
• single value would be [42] or 42