Inference

LLaMA-Factory supports multiple inference methods.

You can use llamafactory-cli chat inference_config.yaml or llamafactory-cli webchat inference_config.yaml for inference and conversation with the model. During conversation, the configuration file only needs to specify the original model model_name_or_path and template, and specify adapter_name_or_path and finetuning_type depending on whether it is a fine-tuned model.

If you want to input a large dataset to the model and save the inference results, you can start the vllm inference engine for fast batch inference on large datasets. You can also perform batch inference through API calls by api deploying service.

By default, model inference will use the Huggingface engine. You can also specify infer_backend: vllm to use the vllm inference engine for faster inference speed.

Note

When using any inference method, the model model_name_or_path must exist and correspond to the template.

Original Model Inference Configuration

For original model inference, only the original model model_name_or_path and template need to be specified in inference_config.yaml.

### examples/inference/qwen3.yaml
model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
template: qwen3_nothink
infer_backend: huggingface #choices： [huggingface, vllm]
trust_remote_code: true

Fine-tuned Model Inference Configuration

For fine-tuned model inference, in addition to the original model and template, you also need to specify the adapter path adapter_name_or_path and the fine-tuning type finetuning_type.

### examples/inference/qwen3_lora_sft.yaml
model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
adapter_name_or_path: saves/qwen3-4b/lora/sft
template: qwen3_nothink
infer_backend: huggingface #choices： [huggingface, vllm]
trust_remote_code: true

Multimodal Model

For multimodal models, you can run the following command for inference.

llamafactory-cli webchat examples/inference/qwen3vl.yaml

The configuration example of examples/inference/qwen3vl.yaml is as follows:

model_name_or_path: Qwen/Qwen3-VL-4B-Instruct
template: qwen3_vl
infer_backend: huggingface #choices： [huggingface, vllm]
trust_remote_code: true

Batch Inference

Dataset

You can start the vllm inference engine and perform batch inference using a dataset with the following command:

python scripts/vllm_infer.py --model_name_or_path path_to_merged_model --dataset alpaca_en_demo

api

If you need to use API for batch inference, you only need to specify the model, adapter (optional), template, fine-tuning method, and other information.

Below is an example of a configuration file:

### examples/inference/qwen3_lora_sft.yaml
model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
adapter_name_or_path: saves/qwen3-4b/lora/sft
template: qwen3_nothink
trust_remote_code: true

Below is an example of starting and calling the API service:

You can start the API service using API_PORT=8000 CUDA_VISIBLE_DEVICES=0 llamafactory-cli api examples/inference/qwen3_lora_sft.yaml and run the following example program to make API calls:

# api_call_example.py
from openai import OpenAI
client = OpenAI(api_key="0",base_url="http://0.0.0.0:8000/v1")
messages = [{"role": "user", "content": "Who are you?"}]
result = client.chat.completions.create(messages=messages, model="Qwen/Qwen3-4B-Instruct-2507")
print(result.choices[0].message)