Tuning algorithms¶

LLaMA-Factory supports multiple fine-tuning algorithms, including: Full Parameter Fine-tuning, Freeze, LoRA, Galore, BAdam.

Full Parameter Fine-tuning¶

Full parameter fine-tuning refers to updating all weights of the pre-trained model during training, but it requires a huge amount of memory.

If you need to perform full parameter fine-tuning, please set finetuning_type to full. Here is an example:

### examples/train_full/qwen3_full_sft.yaml
# ...
finetuning_type: full
# ...
#deepspeed:
deepspeed: examples/deepspeed/ds_z3_config.json

Freeze¶

Freeze (Freeze Fine-tuning) refers to updating only a small portion of the model’s weights during training, which can reduce memory requirements.

If you need to perform freeze fine-tuning, please set finetuning_type to freeze and set related parameters, such as the number of frozen layers freeze_trainable_layers, trainable module names freeze_trainable_modules, etc.

Here is an example:

...
### method
stage: sft
do_train: true
finetuning_type: freeze
freeze_trainable_layers: 8
freeze_trainable_modules: all
...

FreezeArguments¶
Parameter Name	Type	Description
freeze_trainable_layers	int	The number of trainable layers. A positive number indicates the last n layers are set as trainable, a negative number indicates the first n layers are set as trainable. Default value is `2`
freeze_trainable_modules	str	The names of trainable layers. Use `all` to specify all modules. Default value is `all`
freeze_extra_modules[Optional]	str	Names of modules that can be trained besides hidden layers; the specified modules will be set as trainable. Use commas to separate multiple modules. Default value is `None`

LoRA¶

If you need to perform LoRA fine-tuning, please set finetuning_type to lora and set related parameters. Here is an example:

...
### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all
lora_rank: 8
lora_alpha: 16
lora_dropout: 0.1
...

LoraArguments¶
Parameter Name	Type	Description
additional_target[Optional]	[str,]	Names of modules besides LoRA layers that are set as trainable and saved in the final checkpoint. Use commas to separate multiple modules. Default value is `None`
lora_alpha[Optional]	int	LoRA scaling factor. Generally lora_rank * 2, default value is `None`
lora_dropout	float	Dropout rate in LoRA fine-tuning. Default value is `0`
lora_rank	int	The intrinsic dimensionality `r` of LoRA fine-tuning. The larger `r`, the more trainable parameters. Default value is `8`
lora_target	str	Names of modules to apply LoRA method to. Use commas to separate multiple modules, use `all` to specify all modules. Default value is `all`
loraplus_lr_ratio[Optional]	float	LoRA+ learning rate ratio (`λ = ηB/ηA`). `ηA, ηB` are the learning rates of adapter matrices A and B respectively. The ideal value for LoRA+ depends on the model and task chosen. Default value is `None`
loraplus_lr_embedding[Optional]	float	Learning rate for LoRA+ embedding layer, default value is `1e-6`
use_rslora	bool	Whether to use Rank-Stabilized LoRA, default value is `False`.
use_dora	bool	Whether to use Weight-Decomposed LoRA, default value is `False`
pissa_init	bool	Whether to initialize PiSSA adapter, default value is `False`
pissa_iter	int	Number of iteration steps for FSVD execution in PiSSA. Use `-1` to disable it, default value is `16`
pissa_convert	bool	Whether to convert PiSSA adapter to normal LoRA adapter, default value is `False`
create_new_adapter	bool	Whether to create a new adapter with randomly initialized weights, default value is `False`

LoRA+¶

In LoRA, adapter matrices A and B have the same learning rate. You can adjust the learning rate ratio by setting loraplus_lr_ratio. In LoRA+, the learning rate ηA of adapter matrix A is the optimizer learning rate. The learning rate ηB of adapter matrix B is λ * ηA. Where λ is the value of loraplus_lr_ratio.

rsLoRA¶

LoRA fine-tunes by adding low-rank adapters, however increasing lora_rank often leads to gradient collapse, making training unstable. This makes it difficult to achieve satisfactory results when using larger lora_rank for LoRA fine-tuning. rsLoRA (Rank-Stabilized LoRA) makes model training more stable by modifying the scaling factor. When using rsLoRA, you only need to set use_rslora to True and set the desired lora_rank.

DoRA¶

DoRA (Weight-Decomposed Low-Rank Adaptation) proposes that although LoRA significantly reduces inference costs, there is still a gap between the performance achieved by this method and full fine-tuning.

DoRA decomposes the weight matrix into the product of a magnitude and a unit direction matrix, and further fine-tunes both (the direction matrix is further decomposed using LoRA), thereby achieving a balance between LoRA and Full Fine-tuning.

If you need to use DoRA, please set use_dora to True.

PiSSA¶

In LoRA, adapter matrix A is initialized by kaiming_uniform, while adapter matrix B is initialized to 0. This causes initial inputs not to change model outputs and results in smaller gradients and slower convergence. PiSSA initializes by directly decomposing the original weight matrix through singular value decomposition, with the advantage that it can converge faster and better.

If you need to use PiSSA, please set pissa_init to True.

Galore¶

When you need to use the GaLore (Gradient Low-Rank Projection) algorithm in training, you can configure it by setting parameters in GaloreArguments.

Here is an example:

...
### method
stage: sft
do_train: true
finetuning_type: full
use_galore: true
galore_layerwise: true
galore_target: mlp,self_attn
galore_rank: 128
galore_scale: 2.0
...

Warning

Do not use LoRA together with GaLore/BAdam.
When galore_layerwise is true, please do not set the gradient_accumulation parameter.

GaLoreArguments¶
Parameter Name	Type	Description
use_galore	bool	Whether to use the GaLore algorithm, default value is `False`.
galore_target	str	Names of modules to apply GaLore to. Use commas to separate multiple modules, use `all` to specify all linear modules. Default value is `all`.
galore_rank	int	Rank of GaLore gradient, default value is `16`.
galore_update_interval	int	Step interval for updating GaLore projection, default value is `200`.
galore_scale	float	Scaling factor for GaLore, default value is `0.25`.
galore_proj_type	Literal	Type of GaLore projection, available values are: `std`, `reverse_std`, `right`, `left`, `full`. Default value is `std`.
galore_layerwise	bool	Whether to enable layer-wise updates to further save memory, default value is `False`.

BAdam¶

BAdam is a memory-efficient full parameter optimization method. You can configure it in detail by setting parameters in BAdamArgument. Here is an example:

### model
...
### method
stage: sft
do_train: true
finetuning_type: full
use_badam: true
badam_mode: layer
badam_switch_mode: ascending
badam_switch_interval: 50
badam_verbose: 2
pure_bf16: true
...

Warning

Do not use LoRA together with GaLore/BAdam.
When using BAdam, please set finetuning_type to full and pure_bf16 to True.
When badam_mode = layer, only single-card or multi-card training using DeepSpeed ZeRO3 is supported.
When badam_mode = ratio, only single-card training is supported.

BAdamArgument¶
Parameter Name	Type	Description
use_badam	bool	Whether to use the BAdam optimizer, default value is `False`.
badam_mode	Literal	Mode of BAdam usage, available values are `layer` or `ratio`, default value is `layer`.
badam_start_block	Optional[int]	Starting block index for layer-wise BAdam, default value is `None`.
badam_switch_mode	Optional[Literal]	Block update strategy in layer-wise BAdam, available values are: `ascending`, `descending`, `random`, `fixed`. Default value is `ascending`.
badam_switch_interval	Optional[int]	Step interval for block updates in layer-wise BAdam. Use `-1` to disable block updates, default value is `50`.
badam_update_ratio	float	Update ratio in ratio-wise BAdam, default value is `0.05`.
badam_mask_mode	Literal	Mask mode for BAdam optimizer, available values are `adjacent` or `scatter`, default value is `adjacent`.
badam_verbose	int	Verbose output level for BAdam optimizer, 0 means no output, 1 means output block prefix, 2 means output trainable parameters. Default value is `0`.