NPU Installation

This document describes how to prepare the LlamaFactory environment on Huawei Ascend NPUs. It currently focuses on Atlas A2/A3 training series devices. Before installation, confirm the hardware model and operating system compatibility, and then choose the appropriate deployment method.

Hardware Compatibility and Supported Operating Systems

Table 1 Hardware Support List

Product	Supported
Ascend 950 Series Products	√
Atlas A3 Training Series Products	√
Atlas A3 Inference Series Products	x
Atlas A2 Training Series Products	√
Atlas A2 Inference Series Products	x
Atlas 200I/500 A2 Inference Products	x
Atlas Inference Series Products	x
Atlas Training Series Products	x

Note

In this table, “√” indicates supported and “x” indicates not supported.

• For operating systems supported by each hardware product in physical-machine deployment scenarios, see the Compatibility Query Assistant.

• For operating systems supported by each hardware product in VM and container deployment scenarios, see the “Operating System Compatibility Description” section in CANN Software Installation (commercial edition) or the “Operating System Compatibility Description” section (community edition).

After confirming that the hardware and operating system meet the preceding requirements, choose one of the following three methods for environment setup and usage:

• Method 1: Manual Environment Installation

• Method 2: Docker Pre-installed Image

• Method 3: Docker Local Build

Core Dependencies

All installation methods depend on the following components:

• HDK: Firmware and drivers

• CANN: Heterogeneous Computing Architecture

• TorchNPU: Ascend adaptation plugin for PyTorch

Required steps vary depending on the installation method:

• Manual installation: Manually install HDK, CANN, and TorchNPU.

• Docker image/build: The host only needs to install HDK (driver/firmware). CANN and TorchNPU are integrated in the image.

Method 1: Manual Environment Installation

This method requires you to manually install HDK, CANN, and TorchNPU.

1. Versions and Download Links

This document lists the latest dependency versions and download links. Please choose according to your device model:

Device	Dependency	Link
A3	HDK	https://www.hiascend.com/hardware/firmware-drivers?ids=d803%2C89dda9ba9de741349efa03687a487678%2C16%2CAArch64%2Conline_Yum
	CANN	https://www.hiascend.com/developer/download/community/result?module=cann&cann=9.1.0
	TorchNPU	2.10.0.post2
A2	HDK	https://www.hiascend.com/hardware/firmware-drivers?ids=d802%2C26958bcc909e4cd48fa56d4c4a43ebec%2C17%2CAArch64%2Conline_Yum
	CANN	https://www.hiascend.com/developer/download/community/result?module=cann&cann=9.1.0
	TorchNPU	2.10.0.post2

2. Drivers and Firmware

Choose the HDK installation package in .run or .deb format as appropriate, noting that packages are differentiated for aarch64 and x86.

The following uses the A2 series as an example. For the A3 series, the firmware and driver package names differ; choose based on the linked page.

A3 internal package names are similar to Atlas-A3-hdk-npu-driver_25.0.rc1.3_linux-aarch64.run and Atlas-A3-hdk-npu-firmware_7.7.0.3.228.run. The installation method does not change.

1. Upload installation packages, log in as root and upload the driver and firmware packages to the server (e.g., /home).

2. Add execution permissions, enter the package directory and execute the following commands.

   chmod +x Ascend-hdk-<chip_type>-npu-driver_<version>_linux-<arch>.run
   chmod +x Ascend-hdk-<chip_type>-npu-firmware_<version>.run
   

3. Install drivers and firmware, The default installation path is /usr/local/Ascend.

   Install driver:

   ./Ascend-hdk-<chip_type>-npu-driver_<version>_linux-<arch>.run --full --install-for-all
   

   If Driver package installed successfully! appears, the installation succeeded.

   Install firmware:

   ./Ascend-hdk-<chip_type>-npu-firmware_<version>.run --full
   

   If Firmware package installed successfully! appears, the installation succeeded.

   Note

   If the default user HwHiAiUser has not been created, specify the user and group in the installation command: ./Ascend-hdk-*.run --full --install-username=<username> --install-usergroup=<usergroup>

4. Decide whether to reboot according to prompts. To reboot:

   reboot
   

5. Verify installation. Run the following command to check driver load status:

   npu-smi info
   

   

3. CANN

Choose the CANN installation package in .run or .deb format as appropriate, noting that packages are differentiated for aarch64 and x86.

The following uses the A2 series as an example. The only difference for the A3 series is the name of the ops package; choose according to the linked page. A3 package names are similar to Ascend-cann-A3-ops_<version>_linux-aarch64.run. The installation method does not change.

(1) Install Toolkit development kit

Toolkit is used for training, inference, and development.

Note

Ensure the installation directory has more than 10G of free space.

1. Authorization and installation: For root user: /usr/local/Ascend; For normal user: ${HOME}/Ascend.

   chmod +x Ascend-cann-toolkit_<version>_linux-aarch64.run
   ./Ascend-cann-toolkit_<version>_linux-aarch64.run --install
   

2. Configure environment variables: For root users, it is recommended to write to ~/.bashrc.

   source /usr/local/Ascend/ascend-toolkit/set_env.sh
   

(2) Install ops operator package

Execute after installing Toolkit. To install static libraries, replace --install with --devel.

chmod +x Ascend-cann-<chip_type>-ops_<version>_linux-aarch64.run
./Ascend-cann-<chip_type>-ops_<version>_linux-aarch64.run --install

(3) Install NNAL neural network acceleration library (optional)

Includes ATB and SiP acceleration libraries. Execute after installing Toolkit.

1. Authorization and installation:

   chmod +x Ascend-cann-nnal_<version>_linux-aarch64.run
   ./Ascend-cann-nnal_<version>_linux-aarch64.run --install
   

2. Configure environment variables:

   (Choose one of the two, do not configure both)

   # ATB
   source ${HOME}/Ascend/nnal/atb/set_env.sh
   
   # SiP
   source ${HOME}/Ascend/nnal/asdsip/set_env.sh
   

4. TorchNPU

It is recommended to install the TorchNPU plugin together with LlamaFactory. LlamaFactory dependencies will continue to track stable TorchNPU releases.

pip install -r requirements/npu.txt

You can also download and install the TorchNPU plugin manually, for example:

pip install torch_npu-2.10.0.post2-cp312-cp312-manylinux_2_28_aarch64.whl

Note the following when installing the TorchNPU plugin:

• TorchNPU packages are built for specific Python versions, so choose the package that matches your environment. Running pip install torch_npu also installs the corresponding version of torch.

• The TorchNPU and torch versions installed in the environment must be compatible. For example, TorchNPU 2.10.0.post2 requires torch 2.10.0. Dependency conflicts may sometimes cause torch to be upgraded during another package installation, resulting in errors.

5. Verify Installation

Run the following Python script:

import torch
import torch_npu
print(torch.npu.is_available())

Expected output: True

This indicates that HDK, CANN, and TorchNPU are installed correctly and working.

Method 2: Docker Pre-installed Image

Note

Ensure the host has installed firmware and drivers; refer to the previous section for installation.

LlamaFactory’s official images are hosted on Docker Hub and quay.io; the images are identical.

1. Pull Image

When downloading the latest image from the main branch, select the tag according to both the device and the container operating system. A2 images use the 910b identifier in the tag, while A3 images use a3:

Hardware Series	Container Operating System	Tag
A2	Ubuntu 22.04	latest-910b-ubuntu
A3	Ubuntu 22.04	latest-a3-ubuntu
A2	openEuler 24.03	latest-910b-openeuler
A3	openEuler 24.03	latest-a3-openeuler

After determining the tag from the table above, select one image registry, either Docker Hub or quay.io, and run the corresponding command to pull the image locally:

# Docker Hub
docker pull hiyouga/llamafactory:latest-910b-ubuntu
docker pull hiyouga/llamafactory:latest-a3-ubuntu
docker pull hiyouga/llamafactory:latest-910b-openeuler
docker pull hiyouga/llamafactory:latest-a3-openeuler

# quay.io
docker pull quay.io/ascend/llamafactory:latest-910b-ubuntu
docker pull quay.io/ascend/llamafactory:latest-a3-ubuntu
docker pull quay.io/ascend/llamafactory:latest-910b-openeuler
docker pull quay.io/ascend/llamafactory:latest-a3-openeuler

The latest images use the latest-<910b|a3>-<ubuntu|openeuler> format, and scheduled builds update these tags. Release images use tags containing the complete version information:

<LlamaFactory-version>-cann<CANN-version>-torch_npu<TorchNPU-version>-<910b|a3>-<OS-and-version>-py<Python-version>

2. Start Container

Start the container with the following command (modify DOCKER_IMAGE and device as appropriate):

CONTAINER_NAME=llamafactory-npu
DOCKER_IMAGE=hiyouga/llamafactory:latest-910b-ubuntu

docker run -itd \
    --net=host \
    --device=/dev/davinci0 \
    --device=/dev/davinci1 \
    --device=/dev/davinci2 \
    --device=/dev/davinci3 \
    --device=/dev/davinci4 \
    --device=/dev/davinci5 \
    --device=/dev/davinci6 \
    --device=/dev/davinci7 \
    --device=/dev/davinci_manager \
    --device=/dev/devmm_svm \
    --device=/dev/hisi_hdc \
    --shm-size=1200g \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /data:/data \
    --name "$CONTAINER_NAME" \
    "$DOCKER_IMAGE" \
    /bin/bash

Note

Setting --privileged=true enables privileged mode, granting the container full access to hardware management devices (such as /dev/davinci_manager). This resolves driver initialization failures in multi-container scenarios caused by permission restrictions, ensuring NPU resources can be reused across containers.

Note: Without this parameter, subsequent containers may fail to access devices due to insufficient permissions after the first container occupies them. Given the broad permissions of privileged mode, evaluate security risks carefully before using it in production.

3. Verify the Container Environment

After the container starts, enter it:

docker exec -it llamafactory-npu /bin/bash

Then load the Ascend environment and verify the software versions, NPU availability, and the LlamaFactory command:

source /usr/local/Ascend/ascend-toolkit/set_env.sh
npu-smi info
python -c "import torch, torch_npu; print(torch.__version__, torch_npu.__version__, torch.npu.is_available())"
llamafactory-cli help

Note

In some driver environments, npu-smi is located at /usr/local/sbin/npu-smi; in that case, adjust the source path of the mount. Use --device=/dev/davinci<N> to mount additional NPU devices. Device numbers are automatically remapped inside the container (for example, davinci6 on the host becomes device 0 in the container).

After the environment verification succeeds, you can start training directly with llamafactory-cli train.

Method 3: Docker Local Build

Note

Ensure the host has installed firmware and drivers.

LlamaFactory provides two build methods: 1. Build Using Docker Build and 2. Build Using Docker Compose.

1. Build Using Docker Build

Run the following command in the project root to build an A2 Ubuntu image:

docker build \
    -f ./docker/docker-npu/Dockerfile \
    --build-arg BASE_IMAGE=quay.io/ascend/cann:9.1.0-910b-ubuntu22.04-py3.12 \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:npu-910b-ubuntu \
    .

The Dockerfile uses an A2 Ubuntu base image by default. To build another combination, replace BASE_IMAGE and the local image tag:

Hardware Series	Container Operating System	BASE_IMAGE
A2	Ubuntu 22.04	quay.io/ascend/cann:9.1.0-910b-ubuntu22.04-py3.12
A3	Ubuntu 22.04	quay.io/ascend/cann:9.1.0-a3-ubuntu22.04-py3.12
A2	openEuler 24.03	quay.io/ascend/cann:9.1.0-910b-openeuler24.03-py3.12
A3	openEuler 24.03	quay.io/ascend/cann:9.1.0-a3-openeuler24.03-py3.12

The available Dockerfile build arguments are as follows:

Argument	Default	Purpose
BASE_IMAGE	quay.io/ascend/cann:9.1.0-910b-ubuntu22.04-py3.12	Select the corresponding base image for the device model and container operating system
PIP_INDEX	https://pypi.org/simple	Specify the Python package index
PYTORCH_INDEX	https://download.pytorch.org/whl/cpu	Specify the PyTorch wheel index used with TorchNPU
HTTP_PROXY	Empty	Configure the HTTP/HTTPS proxy used during the build

After the build completes, reuse the docker run command in Method 2: Docker Pre-installed Image and set DOCKER_IMAGE to the local image name specified with -t during the build.

2. Build Using Docker Compose

1. Enter directory

cd docker/docker-npu

2. Start the container. If the image is not available locally, Docker Compose builds it first. Select a profile according to the device model and container operating system:

# A2 + Ubuntu
docker compose --profile a2-ubuntu up -d

# A3 + Ubuntu
docker compose --profile a3-ubuntu up -d

# A2 + openEuler
docker compose --profile a2-openeuler up -d

# A3 + openEuler
docker compose --profile a3-openeuler up -d

3. Enter container

# A2 + Ubuntu
docker exec -it llamafactory-910b-ubuntu /bin/bash

# A3 + Ubuntu
docker exec -it llamafactory-a3-ubuntu /bin/bash

# A2 + openEuler
docker exec -it llamafactory-910b-openeuler /bin/bash

# A3 + openEuler
docker exec -it llamafactory-a3-openeuler /bin/bash

To build the image without starting a container, use docker compose --profile <profile> build.

Note

Before building, check the devices list in docker-compose.yml. By default, only device 0 is mounted; modify the list as needed.