improving of position embedding

[pass-lin]

from #2192

Removed the submission about the export of the repair roformer

[pass-lin]

ruff.....................................................................Passed
ruff-format..............................................................Passed
Error: Process completed with exit code 1.

What is the cause of this error? I need help
@madhusshivakumar @divyashreepathihalli

[mattdangerw]

@pass-lin try just rebase/syncing to latest master. Some tooling changes just went out here. #2222

[mattdangerw]

Thanks! Just a question for now. What models use this?

[mattdangerw]

We'd need a docstring to add this. But before we do, is this used by more than one model architecture? If this is just used in one, we probably would prefer to keep the layer simple and just leave the customization per model arch.

[pass-lin]

Thanks! Just a question for now. What models use this?

The main function of this feature is to non-destructively expand the max sequence of models such as BERT and ViT.
For example, in BERT, this function will only be activated when the sequence length is greater than 512.

[pass-lin]

@mattdangerw
At present, the following Keras_hub models can probably benefit from this feature.
bert roberta albert gpt2
Also, in the future, I plan to add ESM2.
The current maximum length of esm2 is only 1024, which is far from enough in bioinformatics.

[mattdangerw]

@pass-lin how would you expect to use this feature with BERT currently? Seems tricky to actual use given that bert will be loaded form a config without this option.

[pass-lin]

@pass-lin how would you expect to use this feature with BERT currently? Seems tricky to actual use given that bert will be loaded form a config without this option.

I hope it can be used without feeling anything. It can be activated when the length exceeds the preset maximum length.

If you think this is unnecessary, you can close this PR.