- https://github.com/axolotl-ai-cloud/axolotl/blob/main/docs/config.qmd
- https://www.youtube.com/watch?v=mmsa4wDsiy0
Model result is empty when using
modal run -q src.inference --prompt "<INSERT PROMPT>"The log references a request id. How to access it? May contain the model response
- https://github.com/axolotl-ai-cloud/axolotl?tab=readme-ov-file#tokenization-mismatch-bw-inference--training
- https://hamel.dev/notes/llm/finetuning/05_tokenizer_gotchas.html
- https://youtu.be/mmsa4wDsiy0?feature=shared&t=1927
Likely improvements to be made to formatting prompts (XML like separators). Formatting in prep_data.py.
Axolotl was used because it is what Modal pushed for finetuning as part of its tutorial. It is not required to use axolotl. It is worth seeing if the original training code can be adapted to run on Modal's serverless platform. Will likely require updating the VLLM image definition in src/common.py. Example of training without axolotl https://modal.com/docs/examples/slack-finetune