[sync #10544] llama/ggml: add LLM training support #13105
Conversation
github-actions
bot
added
testing
|
@JohannesGaessler This is a tentative sync - still need to wait for #12799 to get merged. The optimization code in In #12799, the batch management is delegated to the KV cache object, so I've updated |
ggerganov
force-pushed
the
gg/llama-kv-cache-v6
branch
5 times, most recently
from
780d6fb to
58115a2
Compare
|
this new feature is very helpful for AI beginners(such as me) to understand more details in hard-core AI tech. thanks too much! |
ggerganov
force-pushed
the
gg/llama-kv-cache-v6
branch
from
58115a2 to
7e79a42
Compare
more compact progress bar refactor: llama_prepare_sbatch/ubatch llama_save_model_to_file gqa_mode arg for repeat_back llama_opt_param_filter ggml_graph_dup force_grads refactor ggml_opt, fix test-opt
ggml-ci
|
@JohannesGaessler I've rebased this and should be good to update #10544 respectively and merge. Let me know if something does not work as expected. |
|
Thank you. I'll take a look when I get a chance. |
original #10544
This is a rebase of the #10544 PR by @JohannesGaessler on top of the upcoming #12799 (edit: merged now into
master). The purpose is only to highlight the necessary changes that need to be applied to #10544.Testing with:
make -j && ./bin/llama-finetune --file ./wikitext-2-raw/wiki.test.raw --model ../models/llama-3.2-3b/ggml-model-f32.gguf -c 512 -b 512 -ub 512TODOs:
test-backend-opsasserts becauseggml_set_paramassertstensor->op == GGML_OP_NONE, but does not take into account that the tensor could be a view.