Add dora support

[Ajinkya-25]

Description of the change

This PR adds DoRA (Weight-Decomposed Low-Rank Adaptation) support to Keras Hub by introducing new reusable layers:

DoRADense: Drop-in replacement for Dense with DoRA decomposition.

DoRAEmbedding: Drop-in replacement for Embedding with DoRA adaptation.

DoRAPositionEmbedding: DoRA-based positional embeddings.

Conversion utilities (convert_dense_to_dora, convert_embedding_to_dora) for upgrading existing layers.

To demonstrate usage, we integrated DoRA into the BertBackbone (enable_dora=True) as an example of how existing models can adopt DoRA with minimal changes.

Reference

Paper: https://arxiv.org/abs/2402.09353

Checklist

• Added comprehensive unit tests for DoRADense, DoRAEmbedding, and DoRAPositionEmbedding.
• Verified conversion utilities preserve pretrained weights
• Integrated DoRA into BERT backbone as an example; confirmed DoRA and regular BERT produce identical output shapes.
• Verified model saving/loading works with DoRA-enabled layers.
• Followed Keras Hub contribution guidelines.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Summary of Changes

Hello @Ajinkya-25, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces DoRA (Weight-Decomposed Low-Rank Adaptation) support to Keras Hub, providing a novel approach for efficient fine-tuning of large language models. It adds new Keras layers, DoRADense, DoRAEmbedding, and DoRAPositionEmbedding, which implement the DoRA mechanism by decomposing weight matrices into magnitude and direction components. The PR also includes convenient utility functions to convert existing standard Dense and Embedding layers to their DoRA-enabled versions. To showcase its practical application, DoRA has been integrated into the BertBackbone model, allowing users to easily leverage this technique for more efficient model adaptation.

Highlights

• New DoRADense Layer: Introduced DoRADense, a new Keras layer that implements Weight-Decomposed Low-Rank Adaptation for dense connections, allowing for efficient fine-tuning by decomposing weights into frozen pretrained weights, learnable low-rank matrices, and a learnable magnitude vector.
• New DoRA Embedding Layers: Added DoRAEmbedding and DoRAPositionEmbedding layers, which extend the DoRA concept to embedding and positional embedding matrices, enabling efficient adaptation of these crucial components in neural networks.
• Conversion Utilities for Existing Layers: Provided utility functions, convert_dense_to_dora and convert_embedding_to_dora, to seamlessly transform existing standard Keras Dense and Embedding layers into their DoRA-enabled counterparts, preserving pretrained weights.
• BertBackbone Integration: Integrated DoRA support into the BertBackbone model, allowing users to enable DoRA for its token embeddings, positional embeddings, and pooled dense layer via a new enable_dora flag, demonstrating a practical application of the new layers.
• Extensive Testing: Included comprehensive unit and integration tests for all new DoRA layers and their integration within the BertBackbone, ensuring correctness, functional equivalence with original layers (when applicable), and proper model saving/loading.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces DoRA (Weight-Decomposed Low-Rank Adaptation) support to Keras Hub by adding new reusable layers: DoRADense, DoRAEmbedding, and DoRAPositionEmbedding. The changes also include conversion utilities and an integration example in the BertBackbone. The implementation is well-structured and includes comprehensive tests. My main feedback is to address a style guide violation regarding the use of type hints in function signatures in the new layer files. Addressing this will make the contribution align perfectly with the project's standards.

[Ajinkya-25]

This PR is complete with all implementations, and the test cases are passing. Please let me know if anything else is required from my side.