AIAS/How to config engine.md at main

test/AIAS

mirror of https://gitee.com/mymagicpower/AIAS.git synced 2024-11-29 18:58:16 +08:00

2024-10-20 14:01:45 +08:00

16 KiB

Raw Permalink Blame History

Engine configuration

Includes CPU computing configurations for PaddlePaddle, Pytorch, MxNet, and Tensorflow (automatic configuration and manual configuration). During the first run, automatic configuration will download the native engine from Amazon servers, which may be slow. Manual configuration, on the other hand, will download from a maven server (which can point to a domestic server), which is much faster. Additionally, local libraries will be bundled together, so they will not be loaded during runtime after deployment.

Architecture:

Supported engines and operating systems:

MXNet - full support
PyTorch - full support
TensorFlow - supports inference and NDArray operations
ONNX Runtime - supports basic inference
PaddlePaddle - supports basic inference
TFLite - supports basic inference
TensorRT - supports basic inference
DLR - supports basic inference

Additional information: The following versions can be directly supported CentOS Linux release 7.9.2009 (Core) and above ubuntu 18.04 and above

Detailed configuration information (including CPU, GPU) reference links:

PaddlePaddle
Pytorch
MxNet
Tensorflow
ONNX
TensorFlow Lite
DLR
TensorRT TensorRT driver version can be found in the TRT_VERSION of the following Dockerfile: https://github.com/deepjavalibrary/djl/blob/master/docker/tensorrt/Dockerfile

Below are the configuration reference examples:

version 0.20.0 (versions may not be updated in time, please refer to the above documentation)

1. PaddlePaddle engine

1). Automatic configuration

<dependency>
    <groupId>ai.djl.paddlepaddle</groupId>
    <artifactId>paddlepaddle-engine</artifactId>
    <version>0.20.0</version>
    <scope>runtime</scope>
</dependency>

2). macOS - CPU

<dependency>
    <groupId>ai.djl.paddlepaddle</groupId>
    <artifactId>paddlepaddle-native-cpu</artifactId>
    <classifier>osx-x86_64</classifier>
    <version>2.2.2</version>
    <scope>runtime</scope>
</dependency>

3). Linux - CPU

<dependency>
    <groupId>ai.djl.paddlepaddle</groupId>
    <artifactId>paddlepaddle-native-cpu</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>2.2.2</version>
    <scope>runtime</scope>
</dependency>

4). Linux - GPU Need to set the environment variable LD_LIBRARY_PATH, such as setting it in /etc/profile.

LD_LIBRARY_PATH=$HOME/.djl.ai/paddle/x.x.x-<cuda-flavor>-linux-x86_64

<dependency>
    <groupId>ai.djl.paddlepaddle</groupId>
    <artifactId>paddlepaddle-native-cu102</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>2.2.2</version>
    <scope>runtime</scope>
</dependency>

<dependency>
    <groupId>ai.djl.paddlepaddle</groupId>
    <artifactId>paddlepaddle-native-cu112</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>2.2.2</version>
    <scope>runtime</scope>
</dependency>

CUDA supported versions

- <artifactId>paddlepaddle-native-cuXXX</artifactId>
  <version>X.X.X</version>

# Search for ai.djl.paddlepaddle: <https://mvnrepository.com/>
# or <https://repo1.maven.org/maven2/ai/djl/paddlepaddle/>
CUDA 11.2: paddlepaddle-native-cu112
CUDA 11.0: paddlepaddle-native-cu110
CUDA 10.2: pytorch-native-cu102
CUDA 10.1: pytorch-native-cu101

5). Windows - CPU

<dependency>
    <groupId>ai.djl.paddlepaddle</groupId>
    <artifactId>paddlepaddle-native-cpu</artifactId>
    <classifier>win-x86_64</classifier>
    <version>2.2.2</version>
    <scope>runtime</scope>
</dependency>

6). Windows GPU

<dependency>
    <groupId>ai.djl.paddlepaddle</groupId>
    <artifactId>paddlepaddle-native-cu112</artifactId>
    <classifier>win-x86_64</classifier>
    <version>2.2.2</version>
    <scope>runtime</scope>
</dependency>

CUDA supported versions

- <artifactId>paddlepaddle-native-cuXXX</artifactId>
  <version>X.X.X</version>

# Search for ai.djl.paddlepaddle: <https://mvnrepository.com/>
# or <https://repo1.maven.org/maven2/ai/djl/paddlepaddle/>
CUDA 11.2: paddlepaddle-native-cu112
CUDA 11.0: paddlepaddle-native-cu110
CUDA 10.2: pytorch-native-cu102
CUDA 10.1: pytorch-native-cu101

2. Pytorch engine

1). Automatic configuration

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-engine</artifactId>
    <version>0.20.0</version>
    <scope>runtime</scope>
</dependency>

#####If you need to specify the PyTorch version explicitly, you can use the following configuration method:

http://docs.djl.ai/engines/pytorch/pytorch-engine/index.html

PyTorch engine version	PyTorch native library version
pytorch-engine:0.20.0	1.11.0, 1.12.1, 1.13.0
pytorch-engine:0.19.0	1.10.0, 1.11.0, 1.12.1
pytorch-engine:0.18.0	1.9.1, 1.10.0, 1.11.0
pytorch-engine:0.17.0	1.9.1, 1.10.0, 1.11.0

......

2). macOS - CPU

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cpu</artifactId>
    <classifier>osx-x86_64</classifier>
    <version>1.13.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

3). macOS - M1

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cpu</artifactId>
    <classifier>osx-aarch64</classifier>
    <version>1.13.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

4). Linux - CPU

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cpu</artifactId>
    <classifier>linux-x86_64</classifier>
    <scope>runtime</scope>
    <version>1.13.0</version>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

5). For aarch64 build

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cpu-precxx11</artifactId>
    <classifier>linux-aarch64</classifier>
    <scope>runtime</scope>
    <version>1.13.0</version>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

6). For Pre-CXX11 build

# We also provide packages for the system like CentOS 7/Ubuntu 14.04 with GLIBC >= 2.17. 
# All the package were built with GCC 7, we provided a newer libstdc++.so.6.24 in the package that 
# contains CXXABI_1.3.9 to use the package successfully.
ai.djl.pytorch:pytorch-jni:1.11.0-0.17.0
ai.djl.pytorch:pytorch-native-cu113-precxx11:1.11.0:linux-x86_64 - CUDA 11.3
ai.djl.pytorch:pytorch-native-cpu-precxx11:1.11.0:linux-x86_64 - CPU

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cu117-precxx11</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>1.13.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

#---------------------------------------------------------------#

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cpu-precxx11</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>1.13.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

7). Linux - GPU

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cu117</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>1.13.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

CUDA Supported Versions

- <artifactId>pytorch-native-cuXXX</artifactId>
  <version>X.X.X</version>

# Search ai.djl.pytorch query: <https://mvnrepository.com/>
# Or <https://repo1.maven.org/maven2/ai/djl/pytorch/>
CUDA 11.7: pytorch-native-cu117
CUDA 11.6: pytorch-native-cu116
CUDA 11.3: pytorch-native-cu113
CUDA 11.1: pytorch-native-cu111
CUDA 10.2: pytorch-native-cu102
CUDA 10.1: pytorch-native-cu101
CUDA 10.0: pytorch-native-cu110
CUDA 9.2: pytorch-native-cu92

8). Windows - CPU

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cpu</artifactId>
    <classifier>win-x86_64</classifier>
    <scope>runtime</scope>
    <version>1.13.0</version>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

9). Windows - GPU

<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cu117</artifactId>
    <classifier>win-x86_64</classifier>
    <version>1.13.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-jni</artifactId>
    <version>1.13.0-0.20.0</version>
    <scope>runtime</scope>
</dependency>

CUDA Supported Versions

- <artifactId>pytorch-native-cuXXX</artifactId>
  <version>X.X.X</version>

# Search ai.djl.pytorch query: <https://mvnrepository.com/>
# Or <https://repo1.maven.org/maven2/ai/djl/pytorch/>
CUDA 11.7: pytorch-native-cu117
CUDA 11.6: pytorch-native-cu116
CUDA 11.3: pytorch-native-cu113
CUDA 11.1: pytorch-native-cu111
CUDA 10.2: pytorch-native-cu102
CUDA 10.1: pytorch-native-cu101
CUDA 10.0: pytorch-native-cu110
CUDA 9.2: pytorch-native-cu92

3. MxNet engine

1). Auto configuration

<dependency>
    <groupId>ai.djl.mxnet</groupId>
    <artifactId>mxnet-engine</artifactId>
    <version>0.20.0</version>
    <scope>runtime</scope>
</dependency>

2). macOS - CPU

<dependency>
    <groupId>ai.djl.mxnet</groupId>
    <artifactId>mxnet-native-mkl</artifactId>
    <classifier>osx-x86_64</classifier>
    <version>1.9.1</version>
    <scope>runtime</scope>
</dependency>

3). Linux - CPU

<dependency>
    <groupId>ai.djl.mxnet</groupId>
    <artifactId>mxnet-native-mkl</artifactId>
    <classifier>linux-x86_64</classifier>
    <scope>runtime</scope>
    <version>1.9.1</version>
</dependency>

4). Linux - GPU

<dependency>
    <groupId>ai.djl.mxnet</groupId>
    <artifactId>mxnet-native-cu112mkl</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>1.9.1</version>
    <scope>runtime</scope>
</dependency>

CUDA Supported Versions

  <artifactId>mxnet-native-cuXXXmkl</artifactId>
  <version>X.X.X</version>

# Search ai.djl.mxnet query: <https://mvnrepository.com/>
# Or <https://repo1.maven.org/maven2/ai/djl/mxnet/>
CUDA 11.2: mxnet-native-cu112mkl
CUDA 11.0: mxnet-native-cu101mkl
CUDA 10.2: mxnet-native-cu101mkl
CUDA 10.1: mxnet-native-cu102mkl
CUDA 9.2: mxnet-native-cu102mkl

5). Windows - CPU

<dependency>
    <groupId>ai.djl.mxnet</groupId>
    <artifactId>mxnet-native-mkl</artifactId>
    <classifier>win-x86_64</classifier>
    <scope>runtime</scope>
    <version>1.9.1</version>
</dependency>

6). Windows - GPU

<dependency>
    <groupId>ai.djl.mxnet</groupId>
    <artifactId>mxnet-native-cu112mkl</artifactId>
    <classifier>win-x86_64</classifier>
    <version>1.9.1</version>
    <scope>runtime</scope>
</dependency>

CUDA Supported Versions

  <artifactId>mxnet-native-cuXXXmkl</artifactId>
  <version>X.X.X</version>

# Search ai.djl.mxnet query: <https://mvnrepository.com/>
# Or <https://repo1.maven.org/maven2/ai/djl/mxnet/>
CUDA 11.2: mxnet-native-cu112mkl
CUDA 11.0: mxnet-native-cu101mkl
CUDA 10.2: mxnet-native-cu101mkl
CUDA 10.1: mxnet-native-cu102mkl
CUDA 9.2: mxnet-native-cu102mkl

4. Tensorflow engine

1). Auto configuration

<dependency>
    <groupId>ai.djl.tensorflow</groupId>
    <artifactId>tensorflow-engine</artifactId>
    <version>0.20.0</version>
    <scope>runtime</scope>
</dependency>

2). macOS - CPU

<dependency>
    <groupId>ai.djl.tensorflow</groupId>
    <artifactId>tensorflow-native-cpu</artifactId>
    <classifier>osx-x86_64</classifier>
    <version>2.7.0</version>
    <scope>runtime</scope>
</dependency>

3). Linux - CPU

<dependency>
    <groupId>ai.djl.tensorflow</groupId>
    <artifactId>tensorflow-native-cpu</artifactId>
    <classifier>linux-x86_64</classifier>
    <scope>runtime</scope>
    <version>2.7.0</version>
</dependency>

4). Linux - GPU

<dependency>
    <groupId>ai.djl.tensorflow</groupId>
    <artifactId>tensorflow-native-cu113</artifactId>
    <classifier>linux-x86_64</classifier>
    <version>2.7.0</version>
    <scope>runtime</scope>
</dependency>

CUDA Supported Versions

  <artifactId>tensorflow-native-cuXXX</artifactId>
  <version>X.X.X</version>

# Search ai.djl.tensorflow query: <https://mvnrepository.com/>
# Or <https://repo1.maven.org/maven2/ai/djl/tensorflow/>
CUDA 11.3:  tensorflow-native-cu113
CUDA 11.0:  tensorflow-native-cu110
CUDA 10.1:  tensorflow-native-cu101

5). Windows - CPU

<dependency>
    <groupId>ai.djl.tensorflow</groupId>
    <artifactId>tensorflow-native-cpu</artifactId>
    <classifier>win-x86_64</classifier>
    <scope>runtime</scope>
    <version>2.7.0</version>
</dependency>

6). Windows - GPU

<dependency>
    <groupId>ai.djl.tensorflow</groupId>
    <artifactId>tensorflow-native-cu113</artifactId>
    <classifier>win-x86_64</classifier>
    <version>2.7.0</version>
    <scope>runtime</scope>
</dependency>

CUDA Supported Versions

  <artifactId>tensorflow-native-cuXXX</artifactId>
  <version>X.X.X</version>

# Search ai.djl.tensorflow query: <https://mvnrepository.com/>
# Or <https://repo1.maven.org/maven2/ai/djl/tensorflow/>
CUDA 11.3:  tensorflow-native-cu113
CUDA 11.0:  tensorflow-native-cu110
CUDA 10.1:  tensorflow-native-cu101

16 KiB Raw Permalink Blame History

Engine configuration

Architecture:

Supported engines and operating systems:

Detailed configuration information (including CPU, GPU) reference links:

Below are the configuration reference examples:

version 0.20.0 (versions may not be updated in time, please refer to the above documentation)

1. PaddlePaddle engine

2. Pytorch engine

3. MxNet engine

4. Tensorflow engine

16 KiB

Raw Permalink Blame History