Extras

LLaMA Pro

To address the forgetting problem in large language models, LLaMA Pro adds new modules to the original model to adapt to new tasks, enabling it to perform better than the original model on multiple new tasks. LLaMA-Factory supports the use of LLaMA Pro. You can run expand.sh to expand Meta-Llama-3-8B-Instruct to llama3-8b-instruct-pro.

When training LLaMA Pro models, you need to specify use_llama_pro as true.