December 2024 Modules 1,2,3

[HSILA]

December 2024 Student Submission

Module Completed

• Module 1: Instruction Tuning
• Module 2: Preference Alignment
• Module 3: Parameter-efficient Fine-tuning
• Module 4: Evaluation
• Module 5: Vision-language Models
• Module 6: Synthetic Datasets
• Module 7: Inference
• Module 8: Deployment

Changes Made

Describe what you've done in this PR:

1. What concepts did you learn?
   This course is a great resource for getting familiar with the next steps needed after having a pre-trained language model to adapt it to a specific domain or task. In Module 1, I learned how to adapt pre-trained models to specific tasks using supervised fine-tuning and chat templates. Module 2 focused on aligning models with human preferences using techniques like DPO and ORPO. Lastly, Module 3 introduced parameter-efficient fine-tuning methods like LoRA and prompt tuning, which are efficient ways to adapt large language models without retraining the entire model.
2. What changes or additions did you make? I didn't get the chance to add any new examples, I just tried to complete the modules as a skimmer.
3. Any challenges you faced? I was confused about the correct dataset formats and how the trainers handle them, but this was clarified by examining the trl source code as well as the trainer scripts and utilities used to format or prepare datasets. Additionally, in some examples, the arguments passed to the trainers should actually be fed to the config objects in newer versions.

Notebooks Added/Modified

List any notebooks you've added or modified:

• Added new example in module_name/student_examples/hsila/* for first three modules.
• Modified existing notebook with additional examples
• Added documentation or comments

Checklist

• I have read the module materials
• My code runs without errors
• I have pushed models and datasets to the huggingface hub
• My PR is based on the december-2024 branch

Questions or Discussion Points

Add any questions you have or points you'd like to discuss:
1.
3.

Additional Notes

Any other information that might be helpful for reviewers:

[burtenshaw]

Nice work @HSILA !

Would you like to take part in peer review? If so, mention me on a PR from another student, review it, and I'll get a student to review yours.

Maybe #161 ?

[HSILA]

Nice work @HSILA !

Would you like to take part in peer review? If so, mention me on a PR from another student, review it, and I'll get a student to review yours.

Maybe #161 ?

Thank you! Sure, I’d be happy to take part in peer review.