Eval

General Capability Evaluation

After completing model training, you can evaluate the model’s performance using llamafactory-cli eval examples/train_lora/llama3_lora_eval.yaml.

The configuration example file examples/train_lora/llama3_lora_eval.yaml is as follows:

### examples/train_lora/llama3_lora_eval.yaml
### model
model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
adapter_name_or_path: saves/llama3-8b/lora/sft # 可选项

### method
finetuning_type: lora

### dataset
task: mmlu_test # mmlu_test, ceval_validation, cmmlu_test
template: fewshot
lang: en
n_shot: 5

### output
save_dir: saves/llama3-8b/lora/eval

### eval
batch_size: 4

NLG Evaluation

Additionally, you can obtain the model’s BLEU and ROUGE scores to evaluate the model’s generation quality using llamafactory-cli train examples/extras/nlg_eval/llama3_lora_predict.yaml.

The configuration example file examples/extras/nlg_eval/llama3_lora_predict.yaml is as follows:

### examples/extras/nlg_eval/llama3_lora_predict.yaml
### model
model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct
adapter_name_or_path: saves/llama3-8b/lora/sft

### method
stage: sft
do_predict: true
finetuning_type: lora

### dataset
eval_dataset: identity,alpaca_en_demo
template: llama3
cutoff_len: 2048
max_samples: 50
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/llama3-8b/lora/predict
overwrite_output_dir: true

### eval
per_device_eval_batch_size: 1
predict_with_generate: true
ddp_timeout: 180000000

Similarly, you can also use the vllm inference framework for faster inference speed by specifying the model and dataset in the command python scripts/vllm_infer.py --model_name_or_path path_to_merged_model --dataset alpaca_en_demo.

Evaluation-Related Arguments

EvalArguments

Parameter Name	Type	Description
task	str	Name of the evaluation task. Options: mmlu_test, ceval_validation, cmmlu_test
task_dir	str	Path to the folder containing the evaluation dataset. Default: evaluation.
batch_size	int	Batch size per GPU. Default: 4.
seed	int	Random seed for data loader. Default: 42.
lang	str	Language for evaluation. Options: en, zh. Default: en.
n_shot	int	Number of few-shot examples. Default: 5.
save_dir	str	Path to save evaluation results. Default: None. An error will be thrown if this path already exists.
download_mode	str	Download mode for the evaluation dataset. Default: DownloadMode.REUSE_DATASET_IF_EXISTS. Reuses the dataset if it already exists, otherwise downloads it.