test/AIAS
1
0
Fork 0
mirror of https://gitee.com/mymagicpower/AIAS.git synced 2024-12-01 19:58:22 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

上传ocr web 应用。

Browse Source
This commit is contained in:
Calvin 2024-11-01 15:02:58 +08:00
parent 6426fde6a6
commit e79b8852b3
149 changed files with 12751 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -60,3 +60,5 @@ archive/4_video_sdks/mp4_facemask_sdk/models/face_mask.zip
archive/4_video_sdks/mp4_facemask_sdk/target/
1_image_sdks/smart_construction_sdk/target/
1_image_sdks/smart_construction_sdk/models/helmet_head_person_s.zip
6_web_app/ocr_web_app/ocr_backend/models/
6_web_app/ocr_web_app/ocr_ui/node_modules/

8
1_image_sdks/seg_sam2_sdk/pom_win_gpu.xml
View File

@ -93,15 +93,15 @@
<!-- </dependency>-->
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-native-cu118</artifactId>
<classifier>linux-x86_64</classifier>
<version>2.0.1</version>
<artifactId>pytorch-native-cu121</artifactId>
<classifier>win-x86_64</classifier>
<version>2.1.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-jni</artifactId>
<version>2.0.1-0.23.0</version>
<version>2.1.2-0.30.0</version>
<scope>runtime</scope>
</dependency>

145
6_web_app/ocr_web_app/README.md Normal file
View File

@ -0,0 +1,145 @@
### 目录：
https://www.aias.top/
### 模型下载：
- 链接：https://pan.baidu.com/s/1-OEOcYHjSeqbfu7XD3ASgw?pwd=f43t
### OCR Web应用
文字识别（OCR）目前在多个行业中得到了广泛应用，比如金融行业的单据识别输入，餐饮行业中的发票识别，
交通领域的车票识别，企业中各种表单识别，以及日常工作生活中常用的身份证，驾驶证，护照识别等等。
OCR（文字识别）是目前常用的一种AI能力。
一般OCR的识别结果是一种按行识别的结构化输出，能够给出一行文字的检测框坐标及文字内容。
但是我们更想要的是带有字段定义的结构化输出，由于表单还活着卡证的多样性，全都预定义好是不现实的。
所以，设计了自定义模板的功能，能够让人设置参照锚点（通过锚点匹配定位，图片透视变换对齐），以及内容识别区
来得到key-value形式的结构化数据。
#### 当前版本包含了下面功能：
1. 自由文本识别（支持旋转、倾斜的图片）
2. 文本图片转正 （一般情况下不需要，因为ocr 原生支持旋转、倾斜的图片 ）
3. 表格文本识别（图片需是剪切好的单表格图片）
4. 表格自动检测文本识别（支持表格文字混编，自动检测表格识别文字，支持多表格）
### 1. 前端部署
#### 1.1 直接运行：
```bash
npm run dev
```
#### 1.2 构建dist安装包：
```bash
npm run build:prod
```
#### 1.3 nginx部署运行(mac环境部署管理前端为例)：
```bash
cd /usr/local/etc/nginx/
vi /usr/local/etc/nginx/nginx.conf
# 编辑nginx.conf
server {
listen 8080;
server_name localhost;
location / {
root /Users/calvin/ocr_ui/dist/;
index index.html index.htm;
}
......
# 重新加载配置：
sudo nginx -s reload
# 部署应用后，重启：
cd /usr/local/Cellar/nginx/1.19.6/bin
# 快速停止
sudo nginx -s stop
# 启动
sudo nginx
```
### 2. 后端部署
#### 2.1 环境要求：
- 系统JDK 1.8+，建议11
#### 2.2 下载模型：
```bash
### 模型下载地址：
链接：https://pan.baidu.com/s/1-OEOcYHjSeqbfu7XD3ASgw?pwd=f43t
### 假设系统为linux,假设路径如下：
/home/models/ocr/
```
#### 2.3 更新模型地址：
```bash
### 配置文件路径：
ocr\ocr_backend\src\main\resources
### 选择系统配置文件，替换成实际的模型路径，以linux为例：
model:
......
table:
# 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测
layout: /home/models/iocr/picodet_lcnet_x1_0_fgd_layout_table_infer_onnx.zip
# 英文表格识别
rec: /home/models/iocr/en_ppstructure_mobile_v2.0_SLANet_infer.zip
# 中文表格识别
# rec: /home/models/iocr/ch_ppstructure_mobile_v2.0_SLANet_infer.zip
ocrv4:
# server detection model URI
det: /home/models/iocr/ch_PP-OCRv4_det_infer.zip
# server recognition model URI
rec: /home/models/iocr/ch_PP-OCRv4_rec_infer.zip
```
#### 2.4 其它模型加载方式，参考下面的文档：
- 模型加载方式（在线自动加载，及本地配置）:
- https://aias.top/AIAS/guides/load_model.html
### 3. 运行程序：
运行编译后的jar：
```bash
# 运行程序
# -Dfile.encoding=utf-8 参数可以解决操作系统默认编码导致的中文乱码问题
nohup java -Dfile.encoding=utf-8 -jar xxxxx.jar > log.txt 2>&1 &
```
### 4. 打开浏览器
- 输入地址： http://localhost:8089
#### 1. 通用文本识别
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/free.jpg)
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/ocrweb_free.jpg)
#### 2. 文本转正
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/ocrweb_mlsd.jpg)
#### 3. 中英文表格文字识别
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/table.jpg)
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/ocrweb_table.jpg)
#### 帮助文档：
- https://aias.top/guides.html
- 1.性能优化常见问题:
- https://aias.top/AIAS/guides/performance.html
- 2.引擎配置（包括CPU，GPU在线自动加载，及本地配置）:
- https://aias.top/AIAS/guides/engine_config.html
- 3.模型加载方式（在线自动加载，及本地配置）:
- https://aias.top/AIAS/guides/load_model.html
- 4.Windows环境常见问题:
- https://aias.top/AIAS/guides/windows.html

145
6_web_app/ocr_web_app/README_EN.md Normal file
View File

@ -0,0 +1,145 @@
### 目录：
https://www.aias.top/
### 模型下载：
- 链接：https://pan.baidu.com/s/1-OEOcYHjSeqbfu7XD3ASgw?pwd=f43t
### OCR Web应用
文字识别（OCR）目前在多个行业中得到了广泛应用，比如金融行业的单据识别输入，餐饮行业中的发票识别，
交通领域的车票识别，企业中各种表单识别，以及日常工作生活中常用的身份证，驾驶证，护照识别等等。
OCR（文字识别）是目前常用的一种AI能力。
一般OCR的识别结果是一种按行识别的结构化输出，能够给出一行文字的检测框坐标及文字内容。
但是我们更想要的是带有字段定义的结构化输出，由于表单还活着卡证的多样性，全都预定义好是不现实的。
所以，设计了自定义模板的功能，能够让人设置参照锚点（通过锚点匹配定位，图片透视变换对齐），以及内容识别区
来得到key-value形式的结构化数据。
#### 当前版本包含了下面功能：
1. 自由文本识别（支持旋转、倾斜的图片）
2. 文本图片转正 （一般情况下不需要，因为ocr 原生支持旋转、倾斜的图片 ）
3. 表格文本识别（图片需是剪切好的单表格图片）
4. 表格自动检测文本识别（支持表格文字混编，自动检测表格识别文字，支持多表格）
### 1. 前端部署
#### 1.1 直接运行：
```bash
npm run dev
```
#### 1.2 构建dist安装包：
```bash
npm run build:prod
```
#### 1.3 nginx部署运行(mac环境部署管理前端为例)：
```bash
cd /usr/local/etc/nginx/
vi /usr/local/etc/nginx/nginx.conf
# 编辑nginx.conf
server {
listen 8080;
server_name localhost;
location / {
root /Users/calvin/ocr_ui/dist/;
index index.html index.htm;
}
......
# 重新加载配置：
sudo nginx -s reload
# 部署应用后，重启：
cd /usr/local/Cellar/nginx/1.19.6/bin
# 快速停止
sudo nginx -s stop
# 启动
sudo nginx
```
### 2. 后端部署
#### 2.1 环境要求：
- 系统JDK 1.8+，建议11
#### 2.2 下载模型：
```bash
### 模型下载地址：
链接：https://pan.baidu.com/s/1-OEOcYHjSeqbfu7XD3ASgw?pwd=f43t
### 假设系统为linux,假设路径如下：
/home/models/ocr/
```
#### 2.3 更新模型地址：
```bash
### 配置文件路径：
ocr\ocr_backend\src\main\resources
### 选择系统配置文件，替换成实际的模型路径，以linux为例：
model:
......
table:
# 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测
layout: /home/models/iocr/picodet_lcnet_x1_0_fgd_layout_table_infer_onnx.zip
# 英文表格识别
rec: /home/models/iocr/en_ppstructure_mobile_v2.0_SLANet_infer.zip
# 中文表格识别
# rec: /home/models/iocr/ch_ppstructure_mobile_v2.0_SLANet_infer.zip
ocrv4:
# server detection model URI
det: /home/models/iocr/ch_PP-OCRv4_det_infer.zip
# server recognition model URI
rec: /home/models/iocr/ch_PP-OCRv4_rec_infer.zip
```
#### 2.4 其它模型加载方式，参考下面的文档：
- 模型加载方式（在线自动加载，及本地配置）:
- https://aias.top/AIAS/guides/load_model.html
### 3. 运行程序：
运行编译后的jar：
```bash
# 运行程序
# -Dfile.encoding=utf-8 参数可以解决操作系统默认编码导致的中文乱码问题
nohup java -Dfile.encoding=utf-8 -jar xxxxx.jar > log.txt 2>&1 &
```
### 4. 打开浏览器
- 输入地址： http://localhost:8089
#### 1. 通用文本识别
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/free.jpg)
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/ocrweb_free.jpg)
#### 2. 文本转正
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/ocrweb_mlsd.jpg)
#### 3. 中英文表格文字识别
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/table.jpg)
![Screenshot](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/OCR/images/ocrweb_table.jpg)
#### 帮助文档：
- https://aias.top/guides.html
- 1.性能优化常见问题:
- https://aias.top/AIAS/guides/performance.html
- 2.引擎配置（包括CPU，GPU在线自动加载，及本地配置）:
- https://aias.top/AIAS/guides/engine_config.html
- 3.模型加载方式（在线自动加载，及本地配置）:
- https://aias.top/AIAS/guides/load_model.html
- 4.Windows环境常见问题:
- https://aias.top/AIAS/guides/windows.html

BIN
6_web_app/ocr_web_app/Ubuntu配置java与nginx_V0.2.pdf Normal file
View File

Binary file not shown.

BIN
6_web_app/ocr_web_app/ocr_backend/build/opencv/perspective.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

BIN
6_web_app/ocr_web_app/ocr_backend/build/opencv/result.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

8
6_web_app/ocr_web_app/ocr_backend/ocr_backend.iml Normal file
View File

@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<module version="4">
<component name="FacetManager">
<facet type="Spring" name="Spring">
<configuration />
</facet>
</component>
</module>

BIN
6_web_app/ocr_web_app/ocr_backend/output/ocr/paddle_OCR_detect.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

304
6_web_app/ocr_web_app/ocr_backend/pom.xml Normal file
View File

@ -0,0 +1,304 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.9.RELEASE</version>
</parent>
<groupId>aias</groupId>
<artifactId>ocr_backend</artifactId>
<version>0.23.0</version>
<name>ocr_backend</name>
<description>AIAS OCR Project</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<jna.version>5.13.0</jna.version>
<fastjson.version>2.0.40</fastjson.version>
<swagger.version>2.9.2</swagger.version>
<djl.version>0.23.0</djl.version>
<javacv.version>1.5.8</javacv.version>
<javacv.ffmpeg.version>5.1.2-1.5.8</javacv.ffmpeg.version>
<javacpp.platform.macosx-x86_64>macosx-x86_64</javacpp.platform.macosx-x86_64>
<javacpp.platform.linux-x86>linux-x86</javacpp.platform.linux-x86>
<javacpp.platform.linux-x86_64>linux-x86_64</javacpp.platform.linux-x86_64>
<javacpp.platform.windows-x86>windows-x86</javacpp.platform.windows-x86>
<javacpp.platform.windows-x86_64>windows-x86_64</javacpp.platform.windows-x86_64>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
<!-- apache commons -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8.1</version>
</dependency>
<!-- Gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.15.0</version>
</dependency>
<!-- DJL -->
<dependency>
<groupId>ai.djl</groupId>
<artifactId>api</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>basicdataset</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version> <!-- overrides default spring boot version to comply with DJL -->
</dependency>
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- ONNX -->
<dependency>
<groupId>ai.djl.onnxruntime</groupId>
<artifactId>onnxruntime-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- PaddlePaddle 表格识别，不支持onnx转换，所以仍然使用 paddle格式-->
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.opencv</groupId>
<artifactId>opencv</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- java cv -->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>javacv-platform</artifactId>-->
<!-- <version>1.5.7</version>-->
<!-- </dependency>-->
<!--javacv截取视频帧-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv</artifactId>
<version>${javacv.version}</version>
</dependency>
<!--MacOS平台-->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>javacpp</artifactId>-->
<!-- <version>${javacv.version}</version>-->
<!-- <classifier>${javacpp.platform.macosx-x86_64}</classifier>-->
<!-- </dependency>-->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>ffmpeg</artifactId>-->
<!-- <version>${javacv.ffmpeg.version}</version>-->
<!-- <classifier>${javacpp.platform.macosx-x86_64}</classifier>-->
<!-- </dependency>-->
<!--Linux平台-->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>javacpp</artifactId>-->
<!-- <version>${javacv.version}</version>-->
<!-- <classifier>${javacpp.platform.linux-x86}</classifier>-->
<!-- </dependency>-->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>ffmpeg</artifactId>-->
<!-- <version>${javacv.ffmpeg.version}</version>-->
<!-- <classifier>${javacpp.platform.linux-x86}</classifier>-->
<!-- </dependency>-->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>javacpp</artifactId>-->
<!-- <version>${javacv.version}</version>-->
<!-- <classifier>${javacpp.platform.linux-x86_64}</classifier>-->
<!-- </dependency>-->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>ffmpeg</artifactId>-->
<!-- <version>${javacv.ffmpeg.version}</version>-->
<!-- <classifier>${javacpp.platform.linux-x86_64}</classifier>-->
<!-- </dependency>-->
<!--Windows平台-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>${javacv.version}</version>
<classifier>${javacpp.platform.windows-x86}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg</artifactId>
<version>${javacv.ffmpeg.version}</version>
<classifier>${javacpp.platform.windows-x86}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>${javacv.version}</version>
<classifier>${javacpp.platform.windows-x86_64}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg</artifactId>
<version>${javacv.ffmpeg.version}</version>
<classifier>${javacpp.platform.windows-x86_64}</classifier>
</dependency>
<!--lombok-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.24</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.19.0</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
<!-- fastjson -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson.version}</version>
</dependency>
<dependency>
<groupId>com.github.xiaoymin</groupId>
<artifactId>knife4j-spring-boot-starter</artifactId>
<version>2.0.2</version>
</dependency>
<!-- Swagger UI 相关 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>${swagger.version}</version>
<exclusions>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>${swagger.version}</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>4.0.0</version>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>top.aias.ocr.MainApplication</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>

236
6_web_app/ocr_web_app/ocr_backend/pom_auto.xml Normal file
View File

@ -0,0 +1,236 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.9.RELEASE</version>
</parent>
<groupId>aias</groupId>
<artifactId>ocr_backend</artifactId>
<version>0.23.0</version>
<name>ocr_backend</name>
<description>AIAS OCR Project</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<jna.version>5.13.0</jna.version>
<fastjson.version>2.0.40</fastjson.version>
<swagger.version>2.9.2</swagger.version>
<djl.version>0.23.0</djl.version>
<javacv.version>1.5.8</javacv.version>
<javacv.ffmpeg.version>5.1.2-1.5.8</javacv.ffmpeg.version>
<javacpp.platform.macosx-x86_64>macosx-x86_64</javacpp.platform.macosx-x86_64>
<javacpp.platform.linux-x86>linux-x86</javacpp.platform.linux-x86>
<javacpp.platform.linux-x86_64>linux-x86_64</javacpp.platform.linux-x86_64>
<javacpp.platform.windows-x86>windows-x86</javacpp.platform.windows-x86>
<javacpp.platform.windows-x86_64>windows-x86_64</javacpp.platform.windows-x86_64>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
<!-- apache commons -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8.1</version>
</dependency>
<!-- Gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.15.0</version>
</dependency>
<!-- DJL -->
<dependency>
<groupId>ai.djl</groupId>
<artifactId>api</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>basicdataset</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version> <!-- overrides default spring boot version to comply with DJL -->
</dependency>
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- ONNX -->
<dependency>
<groupId>ai.djl.onnxruntime</groupId>
<artifactId>onnxruntime-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- PaddlePaddle 表格识别，不支持onnx转换，所以仍然使用 paddle格式-->
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.opencv</groupId>
<artifactId>opencv</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- java cv -->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv-platform</artifactId>
<version>1.5.7</version>
</dependency>
<!--javacv截取视频帧-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv</artifactId>
<version>${javacv.version}</version>
</dependency>
<!--lombok-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.24</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.19.0</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
<!-- fastjson -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson.version}</version>
</dependency>
<dependency>
<groupId>com.github.xiaoymin</groupId>
<artifactId>knife4j-spring-boot-starter</artifactId>
<version>2.0.2</version>
</dependency>
<!-- Swagger UI 相关 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>${swagger.version}</version>
<exclusions>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>${swagger.version}</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>4.0.0</version>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>top.aias.ocr.MainApplication</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>

280
6_web_app/ocr_web_app/ocr_backend/pom_linux.xml Normal file
View File

@ -0,0 +1,280 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.9.RELEASE</version>
</parent>
<groupId>aias</groupId>
<artifactId>ocr_backend</artifactId>
<version>0.23.0</version>
<name>ocr_backend</name>
<description>AIAS OCR Project</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<jna.version>5.13.0</jna.version>
<fastjson.version>2.0.40</fastjson.version>
<swagger.version>2.9.2</swagger.version>
<djl.version>0.23.0</djl.version>
<javacv.version>1.5.8</javacv.version>
<javacv.ffmpeg.version>5.1.2-1.5.8</javacv.ffmpeg.version>
<javacpp.platform.macosx-x86_64>macosx-x86_64</javacpp.platform.macosx-x86_64>
<javacpp.platform.linux-x86>linux-x86</javacpp.platform.linux-x86>
<javacpp.platform.linux-x86_64>linux-x86_64</javacpp.platform.linux-x86_64>
<javacpp.platform.windows-x86>windows-x86</javacpp.platform.windows-x86>
<javacpp.platform.windows-x86_64>windows-x86_64</javacpp.platform.windows-x86_64>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
<!-- apache commons -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8.1</version>
</dependency>
<!-- Gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.15.0</version>
</dependency>
<!-- DJL -->
<dependency>
<groupId>ai.djl</groupId>
<artifactId>api</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>basicdataset</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version> <!-- overrides default spring boot version to comply with DJL -->
</dependency>
<!-- Pytorch linux -->
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-native-cpu</artifactId>
<classifier>linux-x86_64</classifier>
<scope>runtime</scope>
<version>1.13.0</version>
</dependency>
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-jni</artifactId>
<version>1.13.0-0.20.0</version>
<scope>runtime</scope>
</dependency>
<!-- ONNX -->
<dependency>
<groupId>ai.djl.onnxruntime</groupId>
<artifactId>onnxruntime-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- PaddlePaddle 表格识别，不支持onnx转换，所以仍然使用 paddle格式-->
<!-- <dependency>-->
<!-- <groupId>ai.djl.paddlepaddle</groupId>-->
<!-- <artifactId>paddlepaddle-engine</artifactId>-->
<!-- <version>${djl.version}</version>-->
<!-- </dependency>-->
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-native-cpu</artifactId>
<classifier>linux-x86_64</classifier>
<version>2.2.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.opencv</groupId>
<artifactId>opencv</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- java cv -->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>javacv-platform</artifactId>-->
<!-- <version>1.5.7</version>-->
<!-- </dependency>-->
<!--javacv截取视频帧-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv</artifactId>
<version>${javacv.version}</version>
</dependency>
<!--Linux平台-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>${javacv.version}</version>
<classifier>${javacpp.platform.linux-x86}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg</artifactId>
<version>${javacv.ffmpeg.version}</version>
<classifier>${javacpp.platform.linux-x86}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>${javacv.version}</version>
<classifier>${javacpp.platform.linux-x86_64}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg</artifactId>
<version>${javacv.ffmpeg.version}</version>
<classifier>${javacpp.platform.linux-x86_64}</classifier>
</dependency>
<!--lombok-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.24</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.19.0</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
<!-- fastjson -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson.version}</version>
</dependency>
<dependency>
<groupId>com.github.xiaoymin</groupId>
<artifactId>knife4j-spring-boot-starter</artifactId>
<version>2.0.2</version>
</dependency>
<!-- Swagger UI 相关 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>${swagger.version}</version>
<exclusions>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>${swagger.version}</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>4.0.0</version>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>top.aias.ocr.MainApplication</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>

261
6_web_app/ocr_web_app/ocr_backend/pom_mac.xml Normal file
View File

@ -0,0 +1,261 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.9.RELEASE</version>
</parent>
<groupId>aias</groupId>
<artifactId>ocr_backend</artifactId>
<version>0.23.0</version>
<name>ocr_backend</name>
<description>AIAS OCR Project</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<jna.version>5.13.0</jna.version>
<fastjson.version>2.0.40</fastjson.version>
<swagger.version>2.9.2</swagger.version>
<djl.version>0.23.0</djl.version>
<javacv.version>1.5.8</javacv.version>
<javacv.ffmpeg.version>5.1.2-1.5.8</javacv.ffmpeg.version>
<javacpp.platform.macosx-x86_64>macosx-x86_64</javacpp.platform.macosx-x86_64>
<javacpp.platform.linux-x86>linux-x86</javacpp.platform.linux-x86>
<javacpp.platform.linux-x86_64>linux-x86_64</javacpp.platform.linux-x86_64>
<javacpp.platform.windows-x86>windows-x86</javacpp.platform.windows-x86>
<javacpp.platform.windows-x86_64>windows-x86_64</javacpp.platform.windows-x86_64>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
<!-- apache commons -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8.1</version>
</dependency>
<!-- Gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.15.0</version>
</dependency>
<!-- DJL -->
<dependency>
<groupId>ai.djl</groupId>
<artifactId>api</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>basicdataset</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version> <!-- overrides default spring boot version to comply with DJL -->
</dependency>
<!-- Pytorch Mac x86 -->
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-native-cpu</artifactId>
<classifier>osx-x86_64</classifier>
<version>1.13.1</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-jni</artifactId>
<version>1.13.1-0.23.0</version>
<scope>runtime</scope>
</dependency>
<!-- ONNX -->
<dependency>
<groupId>ai.djl.onnxruntime</groupId>
<artifactId>onnxruntime-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- PaddlePaddle 表格识别，不支持onnx转换，所以仍然使用 paddle格式-->
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-native-cpu</artifactId>
<classifier>osx-x86_64</classifier>
<version>2.2.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.opencv</groupId>
<artifactId>opencv</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- java cv -->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>javacv-platform</artifactId>-->
<!-- <version>1.5.7</version>-->
<!-- </dependency>-->
<!--javacv截取视频帧-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv</artifactId>
<version>${javacv.version}</version>
</dependency>
<!--MacOS平台-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>${javacv.version}</version>
<classifier>${javacpp.platform.macosx-x86_64}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg</artifactId>
<version>${javacv.ffmpeg.version}</version>
<classifier>${javacpp.platform.macosx-x86_64}</classifier>
</dependency>
<!--lombok-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.24</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.19.0</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
<!-- fastjson -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson.version}</version>
</dependency>
<dependency>
<groupId>com.github.xiaoymin</groupId>
<artifactId>knife4j-spring-boot-starter</artifactId>
<version>2.0.2</version>
</dependency>
<!-- Swagger UI 相关 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>${swagger.version}</version>
<exclusions>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>${swagger.version}</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>4.0.0</version>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>top.aias.ocr.MainApplication</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>

274
6_web_app/ocr_web_app/ocr_backend/pom_win.xml Normal file
View File

@ -0,0 +1,274 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.9.RELEASE</version>
</parent>
<groupId>aias</groupId>
<artifactId>ocr_backend</artifactId>
<version>0.23.0</version>
<name>ocr_backend</name>
<description>AIAS OCR Project</description>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<jna.version>5.13.0</jna.version>
<fastjson.version>2.0.40</fastjson.version>
<swagger.version>2.9.2</swagger.version>
<djl.version>0.23.0</djl.version>
<javacv.version>1.5.8</javacv.version>
<javacv.ffmpeg.version>5.1.2-1.5.8</javacv.ffmpeg.version>
<javacpp.platform.macosx-x86_64>macosx-x86_64</javacpp.platform.macosx-x86_64>
<javacpp.platform.linux-x86>linux-x86</javacpp.platform.linux-x86>
<javacpp.platform.linux-x86_64>linux-x86_64</javacpp.platform.linux-x86_64>
<javacpp.platform.windows-x86>windows-x86</javacpp.platform.windows-x86>
<javacpp.platform.windows-x86_64>windows-x86_64</javacpp.platform.windows-x86_64>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<version>2.1.2.RELEASE</version>
</dependency>
<!-- apache commons -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8.1</version>
</dependency>
<!-- Gson -->
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.15.0</version>
</dependency>
<!-- DJL -->
<dependency>
<groupId>ai.djl</groupId>
<artifactId>api</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>basicdataset</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version> <!-- overrides default spring boot version to comply with DJL -->
</dependency>
<!-- Pytorch windows -->
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-native-cpu</artifactId>
<classifier>win-x86_64</classifier>
<scope>runtime</scope>
<version>1.13.1</version>
</dependency>
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-jni</artifactId>
<version>1.13.1-0.23.0</version>
<scope>runtime</scope>
</dependency>
<!-- ONNX -->
<dependency>
<groupId>ai.djl.onnxruntime</groupId>
<artifactId>onnxruntime-engine</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- PaddlePaddle 表格识别，不支持onnx转换，所以仍然使用 paddle格式-->
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-native-cpu</artifactId>
<classifier>win-x86_64</classifier>
<version>2.2.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>ai.djl.paddlepaddle</groupId>
<artifactId>paddlepaddle-model-zoo</artifactId>
<version>${djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.opencv</groupId>
<artifactId>opencv</artifactId>
<version>${djl.version}</version>
</dependency>
<!-- java cv -->
<!-- <dependency>-->
<!-- <groupId>org.bytedeco</groupId>-->
<!-- <artifactId>javacv-platform</artifactId>-->
<!-- <version>1.5.7</version>-->
<!-- </dependency>-->
<!--javacv截取视频帧-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv</artifactId>
<version>${javacv.version}</version>
</dependency>
<!--Windows平台-->
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>${javacv.version}</version>
<classifier>${javacpp.platform.windows-x86}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg</artifactId>
<version>${javacv.ffmpeg.version}</version>
<classifier>${javacpp.platform.windows-x86}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacpp</artifactId>
<version>${javacv.version}</version>
<classifier>${javacpp.platform.windows-x86_64}</classifier>
</dependency>
<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>ffmpeg</artifactId>
<version>${javacv.ffmpeg.version}</version>
<classifier>${javacpp.platform.windows-x86_64}</classifier>
</dependency>
<!--lombok-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.24</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.19.0</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>1.4</version>
</dependency>
<!-- fastjson -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>${fastjson.version}</version>
</dependency>
<dependency>
<groupId>com.github.xiaoymin</groupId>
<artifactId>knife4j-spring-boot-starter</artifactId>
<version>2.0.2</version>
</dependency>
<!-- Swagger UI 相关 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>${swagger.version}</version>
<exclusions>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>${swagger.version}</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-annotations</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>io.swagger</groupId>
<artifactId>swagger-models</artifactId>
<version>1.5.21</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>4.0.0</version>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>top.aias.ocr.MainApplication</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>

3
6_web_app/ocr_web_app/ocr_backend/src/main/java/META-INF/MANIFEST.MF Normal file
View File

@ -0,0 +1,3 @@
Manifest-Version: 1.0
Main-Class: top.aias.ocr.MainApplication

19
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/MainApplication.java Normal file
View File

@ -0,0 +1,19 @@
package top.aias.ocr;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
/**
* 入口主程序
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@SpringBootApplication
public class MainApplication {
public static void main(String[] args) {
SpringApplication.run(MainApplication.class, args);
}
}

43
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/CrossRangeCellMeta.java Normal file
View File

@ -0,0 +1,43 @@
package top.aias.ocr.bean;
/**
* excel 单元
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class CrossRangeCellMeta {
public CrossRangeCellMeta(int firstRowIndex, int firstColIndex, int rowSpan, int colSpan) {
super();
this.firstRowIndex = firstRowIndex;
this.firstColIndex = firstColIndex;
this.rowSpan = rowSpan;
this.colSpan = colSpan;
}
private int firstRowIndex;
private int firstColIndex;
private int rowSpan;// 跨越行数
private int colSpan;// 跨越列数
public int getFirstRow() {
return firstRowIndex;
}
public int getLastRow() {
return firstRowIndex + rowSpan - 1;
}
public int getFirstCol() {
return firstColIndex;
}
public int getLastCol() {
return firstColIndex + colSpan - 1;
}
public int getColSpan(){
return colSpan;
}
}

15
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/DataBean.java Normal file
View File

@ -0,0 +1,15 @@
package top.aias.ocr.bean;
import lombok.Data;
import java.util.List;
/**
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class DataBean {
private String value;
private List<Point> points;
}

20
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/LabelBean.java Normal file
View File

@ -0,0 +1,20 @@
package top.aias.ocr.bean;
import lombok.Data;
import java.util.List;
/**
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class LabelBean {
private int index;
private int active;
private String type;
private String value;
private String field;
private List<Point> points;
private ai.djl.modality.cv.output.Point centerPoint;
}

13
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/LabelDTO.java Normal file
View File

@ -0,0 +1,13 @@
package top.aias.ocr.bean;
import lombok.Data;
/**
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class LabelDTO {
private String uid;
private LabelBean labelData;
}

21
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/Point.java Normal file
View File

@ -0,0 +1,21 @@
package top.aias.ocr.bean;
import lombok.Data;
/**
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class Point {
private int x;
private int y;
public Point() {
}
public Point(int x, int y) {
this.x = x;
this.y = y;
}
}

16
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/ProjBean.java Normal file
View File

@ -0,0 +1,16 @@
package top.aias.ocr.bean;
import ai.djl.modality.cv.Image;
import lombok.Data;
/**
* 透视变换对象
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class ProjBean {
private Image image;
private org.opencv.core.Mat warpMat;
}

43
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/ResultBean.java Normal file
View File

@ -0,0 +1,43 @@
package top.aias.ocr.bean;
import lombok.Data;
import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;
/**
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class ResultBean<T> implements Serializable {
private static final long serialVersionUID = 1L;
private int code;
private String value;
private Map<String, Object> data = new HashMap<String, Object>();
public static ResultBean success() {
ResultBean rb = new ResultBean();
rb.setCode(0);
rb.setValue("Success");
return rb;
}
public static ResultBean failure() {
ResultBean msg = new ResultBean();
msg.setCode(-1);
msg.setValue("Failure");
return msg;
}
public ResultBean() {
}
public ResultBean add(String key, Object value) {
this.getData().put(key, value);
return this;
}
}

50
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/RotatedBox.java Normal file
View File

@ -0,0 +1,50 @@
package top.aias.ocr.bean;
import ai.djl.ndarray.NDArray;
/**
* 旋转检测框
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class RotatedBox implements Comparable<RotatedBox> {
private NDArray box;
private String text;
public RotatedBox(NDArray box, String text) {
this.box = box;
this.text = text;
}
/**
* 将左上角 Y 坐标升序排序
*
* @param o
* @return
*/
@Override
public int compareTo(RotatedBox o) {
NDArray lowBox = this.getBox();
NDArray highBox = o.getBox();
float lowY = lowBox.toFloatArray()[1];
float highY = highBox.toFloatArray()[1];
return (lowY < highY) ? -1 : 1;
}
public NDArray getBox() {
return box;
}
public void setBox(NDArray box) {
this.box = box;
}
public String getText() {
return text;
}
public void setText(String text) {
this.text = text;
}
}

23
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/TableResult.java Normal file
View File

@ -0,0 +1,23 @@
package top.aias.ocr.bean;
import ai.djl.modality.cv.output.BoundingBox;
import lombok.Data;
import java.util.List;
/**
* 表格检测结果
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class TableResult {
private List<String> structure_str_list;
private List<BoundingBox> boxes;
public TableResult(List<String> structure_str_list, List<BoundingBox> boxes) {
this.structure_str_list = structure_str_list;
this.boxes = boxes;
}
}

19
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/bean/TemplateBean.java Normal file
View File

@ -0,0 +1,19 @@
package top.aias.ocr.bean;
import lombok.Data;
import java.util.List;
/**
* 模板对象
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
public class TemplateBean {
private String uid;
private String name;
private String imageName;
List<LabelBean> labelData;
}

78
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/configuration/ConfigAdapter.java Normal file
View File

@ -0,0 +1,78 @@
package top.aias.ocr.configuration;
import com.alibaba.fastjson.serializer.SerializerFeature;
import com.alibaba.fastjson.support.config.FastJsonConfig;
import com.alibaba.fastjson.support.spring.FastJsonHttpMessageConverter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.MediaType;
import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.web.cors.CorsConfiguration;
import org.springframework.web.cors.UrlBasedCorsConfigurationSource;
import org.springframework.web.filter.CorsFilter;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;
import org.springframework.web.servlet.config.annotation.ResourceHandlerRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
/**
* 配置类
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Configuration
@EnableWebMvc
public class ConfigAdapter implements WebMvcConfigurer {
// file configuration
private final FileProperties properties;
public ConfigAdapter(FileProperties properties) {
this.properties = properties;
}
@Bean
public CorsFilter corsFilter() {
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration();
config.setAllowCredentials(true);
config.addAllowedOrigin("*");
config.addAllowedHeader("*");
config.addAllowedMethod("*");
source.registerCorsConfiguration("/**", config);
return new CorsFilter(source);
}
@Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
FileProperties.ElPath path = properties.getPath();
String pathUtl = "file:" + path.getPath().replace("\\", "/");
registry.addResourceHandler("/file/**").addResourceLocations(pathUtl).setCachePeriod(0);
registry.addResourceHandler("/file/tables/**").addResourceLocations(pathUtl + "tables/").setCachePeriod(0);
registry.addResourceHandler("/file/images/**").addResourceLocations(pathUtl + "images/").setCachePeriod(0);
registry.addResourceHandler("/**").addResourceLocations("classpath:/META-INF/resources/").setCachePeriod(0);
}
@Override
public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
// 使用 fastjson 序列化，会导致 @JsonIgnore 失效，可以使用 @JSONField(serialize = false) 替换
// Use fastjson serialization, which will cause @JsonIgnore to be invalid, can be replaced with @JSONField (serialize = false)
FastJsonHttpMessageConverter fastJsonHttpMessageConverter = new FastJsonHttpMessageConverter();
List<MediaType> supportMediaTypeList = new ArrayList<>();
supportMediaTypeList.add(MediaType.APPLICATION_JSON_UTF8);
supportMediaTypeList.add(MediaType.TEXT_HTML);
FastJsonConfig config = new FastJsonConfig();
config.setDateFormat("yyyy-MM-dd HH:mm:ss");
config.setSerializerFeatures(SerializerFeature.DisableCircularReferenceDetect);
fastJsonHttpMessageConverter.setFastJsonConfig(config);
fastJsonHttpMessageConverter.setSupportedMediaTypes(supportMediaTypeList);
fastJsonHttpMessageConverter.setDefaultCharset(StandardCharsets.UTF_8);
converters.add(fastJsonHttpMessageConverter);
}
}

46
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/configuration/FileProperties.java Normal file
View File

@ -0,0 +1,46 @@
package top.aias.ocr.configuration;
import lombok.Data;
import top.aias.ocr.utils.Constants;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
/**
* 文件配置
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Data
@Configuration
@ConfigurationProperties(prefix = "file")
public class FileProperties {
/**
* File size limitation
*/
private Long maxSize;
private ElPath mac;
private ElPath linux;
private ElPath windows;
public ElPath getPath() {
String os = System.getProperty("os.name");
if (os.toLowerCase().startsWith(Constants.WIN)) {
return windows;
} else if (os.toLowerCase().startsWith(Constants.MAC)) {
return mac;
}
return linux;
}
@Data
public static class ElPath {
private String path;
}
}

66
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/configuration/ModelConfiguration.java Normal file
View File

@ -0,0 +1,66 @@
package top.aias.ocr.configuration;
import ai.djl.MalformedModelException;
import ai.djl.repository.zoo.ModelNotFoundException;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import top.aias.ocr.model.LayoutDetectionModel;
import top.aias.ocr.model.MlsdSquareModel;
import top.aias.ocr.model.RecognitionModel;
import top.aias.ocr.model.TableRecognitionModel;
import java.io.IOException;
/**
* 模型配置
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Configuration
public class ModelConfiguration {
@Value("${model.table.layout}")
private String tableLayout;
@Value("${model.table.rec}")
private String table;
// ocr model
@Value("${model.ocrv4.det}")
private String ocrDet;
@Value("${model.ocrv4.rec}")
private String ocrRec;
@Value("${model.mlsd.model}")
private String mlsd;
@Value("${model.poolSize}")
private int poolSize;
@Bean
public RecognitionModel recognitionModel() throws IOException, ModelNotFoundException, MalformedModelException {
RecognitionModel recognitionModel = new RecognitionModel();
recognitionModel.init(ocrDet, ocrRec, poolSize);
return recognitionModel;
}
@Bean
public TableRecognitionModel tableDetectionModel() throws IOException, ModelNotFoundException, MalformedModelException {
TableRecognitionModel tableRecognitionModel = new TableRecognitionModel();
tableRecognitionModel.init(table, poolSize);
return tableRecognitionModel;
}
@Bean
public LayoutDetectionModel layoutDetectionModel() throws IOException, ModelNotFoundException, MalformedModelException {
LayoutDetectionModel layoutDetectionModel = new LayoutDetectionModel();
layoutDetectionModel.init(tableLayout, poolSize);
return layoutDetectionModel;
}
@Bean
public MlsdSquareModel mlsdSquareModel() throws IOException, ModelNotFoundException, MalformedModelException {
MlsdSquareModel mlsdSquareModel = new MlsdSquareModel();
mlsdSquareModel.init(mlsd, poolSize);
return mlsdSquareModel;
}
}

112
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/configuration/SwaggerConfig.java Normal file
View File

@ -0,0 +1,112 @@
/*
* Copyright 2019-2020 Zheng Jie
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package top.aias.ocr.configuration;
import com.fasterxml.classmate.TypeResolver;
import com.google.common.base.Predicates;
import io.swagger.annotations.ApiModel;
import io.swagger.annotations.ApiModelProperty;
import lombok.Data;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.Ordered;
import org.springframework.data.domain.Pageable;
import springfox.documentation.builders.ApiInfoBuilder;
import springfox.documentation.builders.ParameterBuilder;
import springfox.documentation.builders.PathSelectors;
import springfox.documentation.schema.AlternateTypeRule;
import springfox.documentation.schema.AlternateTypeRuleConvention;
import springfox.documentation.service.ApiInfo;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;
import java.util.List;
import static com.google.common.collect.Lists.newArrayList;
import static springfox.documentation.schema.AlternateTypeRules.newRule;
/**
* api doc.html
* @author Calvin
* @date Oct 19, 2021
*/
@Configuration
@EnableSwagger2
public class SwaggerConfig {
@Value("${swagger.enabled}")
private Boolean enabled;
@Bean
@SuppressWarnings("all")
public Docket createRestApi() {
ParameterBuilder ticketPar = new ParameterBuilder();
return new Docket(DocumentationType.SWAGGER_2)
.enable(enabled)
.apiInfo(apiInfo())
.select()
.paths(Predicates.not(PathSelectors.regex("/error.*")))
.build();
}
private ApiInfo apiInfo() {
return new ApiInfoBuilder()
.description("")
.title("API Doc")
.version("1.0")
.build();
}
}
/**
* 将Pageable转换展示在swagger中
* Convert Pageable for display in Swagger
*/
@Configuration
class SwaggerDataConfig {
@Bean
public AlternateTypeRuleConvention pageableConvention(final TypeResolver resolver) {
return new AlternateTypeRuleConvention() {
@Override
public int getOrder() {
return Ordered.HIGHEST_PRECEDENCE;
}
@Override
public List<AlternateTypeRule> rules() {
return newArrayList(newRule(resolver.resolve(Pageable.class), resolver.resolve(Page.class)));
}
};
}
@ApiModel
@Data
private static class Page {
@ApiModelProperty("Page number (0..N)")
private Integer page;
@ApiModelProperty("Number of items per page")
private Integer size;
@ApiModelProperty("Sort criteria in the following format: property[,asc|desc]. Default sort order is ascending. Multiple sort conditions are supported, such as id,asc")
private List<String> sort;
}
}

153
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/controller/InferController.java Normal file
View File

@ -0,0 +1,153 @@
package top.aias.ocr.controller;
import ai.djl.Device;
import ai.djl.modality.cv.Image;
import ai.djl.ndarray.NDManager;
import ai.djl.opencv.OpenCVImageFactory;
import com.google.api.client.util.Base64;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import top.aias.ocr.bean.DataBean;
import top.aias.ocr.bean.Point;
import top.aias.ocr.bean.ResultBean;
import top.aias.ocr.bean.RotatedBox;
import top.aias.ocr.service.InferService;
import top.aias.ocr.utils.ImageUtils;
import top.aias.ocr.utils.OpenCVUtils;
import org.opencv.core.Mat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
/**
* 文字识别
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Api(tags = "通用文字识别 -General Text Recognition")
@RestController
@RequestMapping("/inference")
public class InferController {
private Logger logger = LoggerFactory.getLogger(InferController.class);
@Autowired
private InferService inferService;
@Value("${server.baseUri}")
private String baseUri;
@ApiOperation(value = "通用文字识别-URL -General Text Recognition-URL")
@GetMapping(value = "/generalInfoForImageUrl", produces = "application/json;charset=utf-8")
public ResultBean generalInfoForImageUrl(@RequestParam(value = "url") String url) {
try(NDManager manager = NDManager.newBaseManager(Device.cpu(), "PyTorch")) {
Image image = OpenCVImageFactory.getInstance().fromUrl(url);
List<RotatedBox> detections = inferService.getGeneralInfo(manager, image);
List<DataBean> dataList = this.getDataList(detections);
// 转 BufferedImage 解决 Imgproc.putText 中文乱码问题
Mat wrappedImage = (Mat) image.getWrappedImage();
BufferedImage bufferedImage = OpenCVUtils.mat2Image(wrappedImage);
for (RotatedBox result : detections) {
ImageUtils.drawImageRectWithText(bufferedImage, result.getBox(), result.getText());
}
String base64Img = ImageUtils.toBase64(bufferedImage, "jpg");
return ResultBean.success().add("result", dataList).add("base64Img", "data:imageName/jpeg;base64," + base64Img);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
@ApiOperation(value = "通用文字识别-图片 -General Text Recognition-Image")
@PostMapping(value = "/generalInfoForImageFile", produces = "application/json;charset=utf-8")
public ResultBean generalInfoForImageFile(@RequestParam(value = "imageFile") MultipartFile imageFile) {
try (InputStream inputStream = imageFile.getInputStream();
NDManager manager = NDManager.newBaseManager(Device.cpu(), "PyTorch")) {
// String base64Img = Base64.encodeBase64String(imageFile.getBytes());
Image image = OpenCVImageFactory.getInstance().fromInputStream(inputStream);
List<RotatedBox> detections = inferService.getGeneralInfo(manager, image);
List<DataBean> dataList = this.getDataList(detections);
// 转 BufferedImage 解决 Imgproc.putText 中文乱码问题
Mat wrappedImage = (Mat) image.getWrappedImage();
BufferedImage bufferedImage = OpenCVUtils.mat2Image(wrappedImage);
for (RotatedBox result : detections) {
ImageUtils.drawImageRectWithText(bufferedImage, result.getBox(), result.getText());
}
String base64Img = ImageUtils.toBase64(bufferedImage, "jpg");
return ResultBean.success().add("result", dataList).add("base64Img", "data:imageName/jpeg;base64," + base64Img);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
@ApiOperation(value = "转正对齐-URL")
@GetMapping(value = "/mlsdForImageUrl", produces = "application/json;charset=utf-8")
public ResultBean mlsdForImageUrl(@RequestParam(value = "url") String url) throws IOException {
try {
Image image = OpenCVImageFactory.getInstance().fromUrl(url);
Image warpImg = inferService.getWarpImg(image);
BufferedImage buffImage = OpenCVUtils.mat2Image((Mat) warpImg.getWrappedImage());
String base64Img = ImageUtils.toBase64(buffImage,"jpg");
return ResultBean.success().add("base64Img", "data:imageName/jpeg;base64," + base64Img);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
@ApiOperation(value = "转正对齐-图片")
@PostMapping(value = "/mlsdForImageFile", produces = "application/json;charset=utf-8")
public ResultBean mlsdForImageFile(@RequestParam(value = "imageFile") MultipartFile imageFile) {
try (InputStream inputStream = imageFile.getInputStream()) {
Image image = OpenCVImageFactory.getInstance().fromInputStream(inputStream);
Image warpImg = inferService.getWarpImg(image);
BufferedImage buffImage = OpenCVUtils.mat2Image((Mat) warpImg.getWrappedImage());
String orgBase64Img = Base64.encodeBase64String(imageFile.getBytes());
String base64Img = ImageUtils.toBase64(buffImage,"jpg");
return ResultBean.success().add("orgBase64Img", "data:imageName/jpeg;base64," + orgBase64Img).add("base64Img", "data:imageName/jpeg;base64," + base64Img);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
private List<DataBean> getDataList(List<RotatedBox> detections){
List<DataBean> dataList = new ArrayList<>();
for (RotatedBox rotatedBox : detections) {
DataBean dataBean = new DataBean();
List<Point> points = new ArrayList<>();
dataBean.setValue(rotatedBox.getText());
float[] pointsArr = rotatedBox.getBox().toFloatArray();
for (int i = 0; i < 4; i++) {
Point point = new Point((int) pointsArr[2 * i], (int) pointsArr[2 * i + 1]);
points.add(point);
}
dataBean.setPoints(points);
dataList.add(dataBean);
}
return dataList;
}
}

254
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/controller/TableController.java Normal file
View File

@ -0,0 +1,254 @@
package top.aias.ocr.controller;
import ai.djl.modality.cv.Image;
import ai.djl.opencv.OpenCVImageFactory;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import top.aias.ocr.bean.ResultBean;
import top.aias.ocr.configuration.FileProperties;
import top.aias.ocr.service.TableInferService;
import top.aias.ocr.utils.ConvertHtml2Excel;
import top.aias.ocr.utils.UUIDUtils;
import org.apache.commons.codec.binary.Base64;
import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStreamWriter;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/**
* 表格识别
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Api(tags = "表格文字识别 - Table Text Recognition")
@RestController
@RequestMapping("/table")
public class TableController {
private Logger logger = LoggerFactory.getLogger(TableController.class);
@Autowired
private TableInferService tableInferService;
@Value("${server.baseUri}")
private String baseUri;
/**
* file configuration
*/
@Autowired
private FileProperties properties;
@ApiOperation(value = "单表格文字识别-URL - Single Table Text Recognition - URL")
@GetMapping(value = "/tableInfoForImageUrl")
public ResultBean tableInfoForImageUrl(@RequestParam(value = "url") String url) {
try {
Image image = OpenCVImageFactory.getInstance().fromUrl(url);
String tableHtml = tableInferService.getTableHtml(image);
// 创建一个Excel文件
// Create an Excel file
tableHtml = tableHtml.replace("<html><body>", "");
tableHtml = tableHtml.replace("</body></html>", "");
HSSFWorkbook workbook = ConvertHtml2Excel.table2Excel(tableHtml);
FileProperties.ElPath path = properties.getPath();
String fileRelativePath = path.getPath().replace("\\", "/") + "tables/";
//Check & create file path
Path filePath = Paths.get(fileRelativePath);
File file = filePath.toFile();
if (!file.exists() && !file.isDirectory()) {
file.mkdir();
}
String fileId = UUIDUtils.getUUID();
workbook.write(new File(fileRelativePath + fileId + ".xls"));
try (OutputStreamWriter out = new OutputStreamWriter(
new FileOutputStream(fileRelativePath + fileId + ".html"), "UTF-8")) {
out.write(tableHtml);
}
String excelUri = baseUri + File.separator + fileRelativePath + fileId + ".xls";
String htmlUri = baseUri + File.separator + fileRelativePath + fileId + ".html";
Map<String, String> map = new ConcurrentHashMap<>();
map.put("excelUri", excelUri);
map.put("htmlUri", htmlUri);
map.put("html", tableHtml);
return ResultBean.success().add("result", map);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
@ApiOperation(value = "单表格文字识别-图片 -Single Table Text Recognition - Image")
@PostMapping("/tableInfoForImageFile")
public ResultBean tableInfoForImageFile(@RequestParam(value = "imageFile") MultipartFile imageFile) {
try (InputStream inputStream = imageFile.getInputStream()) {
String base64Img = Base64.encodeBase64String(imageFile.getBytes());
Image image = OpenCVImageFactory.getInstance().fromInputStream(inputStream);
String tableHtml = tableInferService.getTableHtml(image);
// 创建一个Excel文件
// Create an Excel file
tableHtml = tableHtml.replace("<html><body>", "");
tableHtml = tableHtml.replace("</body></html>", "");
HSSFWorkbook workbook = ConvertHtml2Excel.table2Excel(tableHtml);
FileProperties.ElPath path = properties.getPath();
String fileRelativePath = path.getPath().replace("\\", "/") + "tables/";
//Check & create file path
Path filePath = Paths.get(fileRelativePath);
File file = filePath.toFile();
if (!file.exists() && !file.isDirectory()) {
file.mkdir();
}
String fileId = UUIDUtils.getUUID();
workbook.write(new File(fileRelativePath + fileId + ".xls"));
try (OutputStreamWriter out = new OutputStreamWriter(
new FileOutputStream(fileRelativePath + fileId + ".html"), "UTF-8")) {
out.write(tableHtml);
}
String excelUri = baseUri + File.separator + fileRelativePath + fileId + ".xls";
String htmlUri = baseUri + File.separator + fileRelativePath + fileId + ".html";
Map<String, String> map = new ConcurrentHashMap<>();
map.put("excelUri", excelUri);
map.put("htmlUri", htmlUri);
map.put("html", tableHtml);
return ResultBean.success().add("result", map)
.add("base64Img", "data:imageName/jpeg;base64," + base64Img);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
@ApiOperation(value = "表单表格自动检测文字识别-URL -Auto Table Text Detection and Recognition - URL")
@GetMapping(value = "/autoTableInfoForImageUrl")
public ResultBean autoTableInfoForImageUrl(@RequestParam(value = "url") String url) {
try {
Image image = OpenCVImageFactory.getInstance().fromUrl(url);
List<String> tableHtmlList = tableInferService.getTableHtmlList(image);
List<HSSFWorkbook> workbookList = new ArrayList<>();
for (String tableHtml : tableHtmlList) {
tableHtml = tableHtml.replace("<html><body>", "");
tableHtml = tableHtml.replace("</body></html>", "");
// Create workbook for each table
HSSFWorkbook workbook = ConvertHtml2Excel.table2Excel(tableHtml);
workbookList.add(workbook);
}
FileProperties.ElPath path = properties.getPath();
String fileRelativePath = path.getPath().replace("\\", "/") + "tables/";
//Check & create file path
Path filePath = Paths.get(fileRelativePath);
File file = filePath.toFile();
if (!file.exists() && !file.isDirectory()) {
file.mkdir();
}
List<Map<String, String>> uriList = new ArrayList<>();
for (int i = 0; i < tableHtmlList.size(); i++) {
String fileId = UUIDUtils.getUUID();
HSSFWorkbook workbook = workbookList.get(i);
workbook.write(new File(fileRelativePath + fileId + ".xls"));
String tableHtml = tableHtmlList.get(i);
try (OutputStreamWriter out = new OutputStreamWriter(
new FileOutputStream(fileRelativePath + fileId + ".html"), "UTF-8")) {
out.write(tableHtml);
}
String excelUri = baseUri + File.separator + fileRelativePath + fileId + ".xls";
String htmlUri = baseUri + File.separator + fileRelativePath + fileId + ".html";
Map<String, String> map = new ConcurrentHashMap<>();
map.put("excelUri", excelUri);
map.put("htmlUri", htmlUri);
map.put("html", tableHtml);
uriList.add(map);
}
return ResultBean.success().add("result", uriList);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
@ApiOperation(value = "表单表格自动检测文字识别-URL -Auto Table Text Detection and Recognition - Image")
@PostMapping("/autoTableInfoForImageFile")
public ResultBean autoTableInfoForImageFile(@RequestParam(value = "imageFile") MultipartFile imageFile) {
try (InputStream inputStream = imageFile.getInputStream()) {
String base64Img = Base64.encodeBase64String(imageFile.getBytes());
Image image = OpenCVImageFactory.getInstance().fromInputStream(inputStream);
List<String> tableHtmlList = tableInferService.getTableHtmlList(image);
List<HSSFWorkbook> workbookList = new ArrayList<>();
for (String tableHtml : tableHtmlList) {
tableHtml = tableHtml.replace("<html><body>", "");
tableHtml = tableHtml.replace("</body></html>", "");
// Create workbook for each table
HSSFWorkbook workbook = ConvertHtml2Excel.table2Excel(tableHtml);
workbookList.add(workbook);
}
FileProperties.ElPath path = properties.getPath();
String fileRelativePath = path.getPath().replace("\\", "/") + "tables/";
//Check & create file path
Path filePath = Paths.get(fileRelativePath);
File file = filePath.toFile();
if (!file.exists() && !file.isDirectory()) {
file.mkdir();
}
List<Map<String, String>> uriList = new ArrayList<>();
for (int i = 0; i < tableHtmlList.size(); i++) {
String fileId = UUIDUtils.getUUID();
HSSFWorkbook workbook = workbookList.get(i);
workbook.write(new File(fileRelativePath + fileId + ".xls"));
String tableHtml = tableHtmlList.get(i);
try (OutputStreamWriter out = new OutputStreamWriter(
new FileOutputStream(fileRelativePath + fileId + ".html"), "UTF-8")) {
out.write(tableHtml);
}
String excelUri = baseUri + File.separator + fileRelativePath + fileId + ".xls";
String htmlUri = baseUri + File.separator + fileRelativePath + fileId + ".html";
Map<String, String> map = new ConcurrentHashMap<>();
map.put("excelUri", excelUri);
map.put("htmlUri", htmlUri);
map.put("html", tableHtml);
uriList.add(map);
}
return ResultBean.success().add("result", uriList).add("base64Img", "data:imageName/jpeg;base64," + base64Img);
} catch (Exception e) {
logger.error(e.getMessage());
e.printStackTrace();
return ResultBean.failure().add("message", e.getMessage());
}
}
}

63
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/LayoutDetectionModel.java Normal file
View File

@ -0,0 +1,63 @@
package top.aias.ocr.model;
import ai.djl.MalformedModelException;
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.repository.zoo.Criteria;
import ai.djl.repository.zoo.ModelNotFoundException;
import ai.djl.repository.zoo.ModelZoo;
import ai.djl.repository.zoo.ZooModel;
import ai.djl.training.util.ProgressBar;
import ai.djl.translate.TranslateException;
import top.aias.ocr.model.pool.LayoutPool;
import top.aias.ocr.translator.PicoDetLayoutDetectionTranslator;
import java.io.IOException;
import java.nio.file.Paths;
/**
* 布局检测模型
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public final class LayoutDetectionModel implements AutoCloseable {
private ZooModel<Image, DetectedObjects> model;
private LayoutPool layoutPool;
public void init(String layoutUri, int poolSize) throws MalformedModelException, ModelNotFoundException, IOException {
this.model = ModelZoo.loadModel(criteria(layoutUri));
this.layoutPool = new LayoutPool(poolSize, model);
}
public void close() {
this.model.close();
this.layoutPool.close();
}
public DetectedObjects predict(Image image) throws TranslateException {
Predictor<Image, DetectedObjects> predictor = layoutPool.getPredictor();
DetectedObjects detectedObjects = predictor.predict(image);
layoutPool.releasePredictor(predictor);
return detectedObjects;
}
private Criteria<Image, DetectedObjects> criteria(String model) {
Criteria<Image, DetectedObjects> criteria =
Criteria.builder()
.optEngine("OnnxRuntime")
.optModelName("inference")
.setTypes(Image.class, DetectedObjects.class)
.optModelPath(Paths.get(model))
// .optModelUrls(model)
.optTranslator(new PicoDetLayoutDetectionTranslator())
.optProgress(new ProgressBar())
.build();
return criteria;
}
}

815
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/MlsdSquareModel.java Normal file
View File

@ -0,0 +1,815 @@
package top.aias.ocr.model;
import ai.djl.Device;
import ai.djl.MalformedModelException;
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.ImageFactory;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDArrays;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.NDManager;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.repository.zoo.Criteria;
import ai.djl.repository.zoo.ModelNotFoundException;
import ai.djl.repository.zoo.ModelZoo;
import ai.djl.repository.zoo.ZooModel;
import ai.djl.training.util.ProgressBar;
import ai.djl.translate.Batchifier;
import ai.djl.translate.TranslateException;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import top.aias.ocr.model.pool.MlsdPool;
import top.aias.ocr.utils.NDArrayUtils;
import top.aias.ocr.utils.OpenCVUtils;
import org.opencv.core.Mat;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.ArrayList;
/**
* 图像转正模型
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public final class MlsdSquareModel implements AutoCloseable {
private ZooModel<Image, Image> model;
private MlsdPool mlsdPool;
private float thr_v = 0.1f;
private float thr_d = 0.1f;
private int detect_resolution = 512;
public void init(String modelUri, int poolSize) throws MalformedModelException, ModelNotFoundException, IOException {
this.model = ModelZoo.loadModel(onnxCriteria(modelUri));
this.mlsdPool = new MlsdPool(poolSize, model);
}
public void close() {
this.model.close();
this.mlsdPool.close();
}
// 多线程环境，每个线程一个predictor，共享一个model, 资源池（CPU Core 核心数）达到上限则等待
public Image predict(Image image) throws TranslateException {
Predictor<Image, Image> predictor = mlsdPool.getPredictor();
Image cropImg = predictor.predict(image);
// 释放资源
mlsdPool.releasePredictor(predictor);
return cropImg;
}
private Criteria<Image, Image> onnxCriteria(String modelUri) {
Criteria<Image, Image> criteria =
Criteria.builder()
.optEngine("OnnxRuntime")
.setTypes(Image.class, Image.class)
.optModelName("mlsd_traced_model")
.optModelPath(Paths.get(modelUri))
.optDevice(Device.cpu())
// .optDevice(Device.gpu())
.optTranslator(new FeatureTranslator())
.optProgress(new ProgressBar())
.build();
return criteria;
}
private final class FeatureTranslator implements Translator<Image, Image> {
protected Batchifier batchifier = Batchifier.STACK;
private int topk_n = 200;
private int ksize = 3;
private float score = 0.06f;
private float outside_ratio = 0.28f;
private float inside_ratio = 0.45f;
private float w_overlap = 0.0f;
private float w_degree = 1.95f;
private float w_length = 0.0f;
private float w_area = 1.86f;
private float w_center = 0.1f;
private NDArray imgArray;
private int original_shape[] = new int[2];
private int input_shape[] = new int[2];
FeatureTranslator() {
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
try (NDManager manager = NDManager.newBaseManager(ctx.getNDManager().getDevice(), "PyTorch")) {
original_shape[1] = input.getWidth(); // w - input_shape[1]
original_shape[0] = input.getHeight(); // h - input_shape[0]
NDArray array = input.toNDArray(ctx.getNDManager(), Image.Flag.COLOR);
array = array.toType(DataType.UINT8, false);
imgArray = array;
// NDArray padding_im = ctx.getNDManager().zeros(new Shape(array.getShape().get(0) + 200, array.getShape().get(1) + 200, array.getShape().get(2)), DataType.FLOAT32);
// padding_im.set(new NDIndex("100:" + (original_shape[0] + 100) + ",100:"+ (original_shape[1]+ 100) + ",:" ), imgArray);
// h : input_shape[0], w : input_shape[1]
input_shape = resize64(original_shape[0], original_shape[1], detect_resolution);
array = NDImageUtils.resize(array, input_shape[1], input_shape[0], Image.Interpolation.AREA);
NDArray ones = manager.ones(new Shape(array.getShape().get(0), array.getShape().get(1), 1), DataType.UINT8);
array = array.concat(ones, -1);
array = array.transpose(2, 0, 1); // HWC -> CHW RGB
array = array.toType(DataType.FLOAT32, false);
array = array.div(127.5f).sub(1.0f);
array = array.flip(0);
return new NDList(array);
}
}
@Override
public Image processOutput(TranslatorContext ctx, NDList list) {
try (NDManager manager = NDManager.newBaseManager(ctx.getNDManager().getDevice(), "PyTorch")) {
NDArray tpMap = list.singletonOrThrow();
// deccode_output_score_and_ptss(tpMap, topk_n = 200, ksize = 3) start
int width = (int) (tpMap.getShape().get(2));
NDArray displacement = tpMap.get("1:5, :, :");
NDArray center = tpMap.get("0, :, :");
// Sigmoid 函数，即f(x)=1/(1+e-x)
NDArray heat = NDArrayUtils.Sigmoid(center);
NDArray hmax = NDArrayUtils.maxPool(manager, heat, ksize, 1, (ksize - 1) / 2);
NDArray keep = hmax.eq(heat);
keep = keep.toType(DataType.FLOAT32, false);
heat = heat.mul(keep);
heat = heat.reshape(-1);
NDArray indices = heat.argSort(-1, false).get("0:200");
NDArray pts_score = heat.get(indices);
indices = indices.toType(DataType.FLOAT32, true);
NDArray yy = indices.div(width).floor().expandDims(-1);
NDArray xx = indices.mod(width).expandDims(-1);
NDArray pts = yy.concat(xx, -1);
NDArray vmap = displacement.transpose(1, 2, 0);
// deccode_output_score_and_ptss end
NDArray start = vmap.get(":, :, :2");
NDArray end = vmap.get(":, :, 2:");
NDArray dist_map = start.sub(end).pow(2).sum(new int[]{-1}).sqrt();
ArrayList<float[]> junc_list = new ArrayList<>();
ArrayList<float[]> segments_list = new ArrayList<>();
for (int i = 0; i < pts_score.size(); i++) {
center = pts.get(i);
int y = (int) center.getFloat(0);
int x = (int) center.getFloat(1);
float score = pts_score.getFloat(i);
float distance = dist_map.getFloat(y, x);
if (score > this.score && distance > 20.0f) {
float[] junc = new float[2];
junc[0] = x;
junc[1] = y;
junc_list.add(junc);
NDArray array = vmap.get(y + "," + x + ",:"); // y, x, :
float disp_x_start = array.getFloat(0);
float disp_y_start = array.getFloat(1);
float disp_x_end = array.getFloat(2);
float disp_y_end = array.getFloat(3);
float x_start = x + disp_x_start;
float y_start = y + disp_y_start;
float x_end = x + disp_x_end;
float y_end = y + disp_y_end;
float[] segment = new float[4];
segment[0] = x_start;
segment[1] = y_start;
segment[2] = x_end;
segment[3] = y_end;
segments_list.add(segment);
}
}
float[][] segmentsArr = new float[segments_list.size()][4];
for (int i = 0; i < segments_list.size(); i++) {
float[] item = segments_list.get(i);
segmentsArr[i][0] = item[0];
segmentsArr[i][1] = item[1];
segmentsArr[i][2] = item[2];
segmentsArr[i][3] = item[3];
}
NDArray segments = manager.create(segmentsArr).toType(DataType.FLOAT32, false);
// ####### post processing for squares
// 1. get unique lines
start = segments.get(":, :2");
end = segments.get(":, 2:");
NDArray diff = start.sub(end);
NDArray a = diff.get(":, 1");
NDArray b = diff.get(":, 0").neg();
NDArray c = a.mul(start.get(":, 0")).add(b.mul(start.get(":, 1")));
NDArray d = c.abs().div(a.square().add(b.square().add(Math.exp(-10))).sqrt());
NDArray theta = NDArrayUtils.arctan2(diff.get(":, 0"), diff.get(":, 1"));
NDArray index = theta.lt(0.0f);
index = index.toType(DataType.INT32, false).mul(180);
theta = theta.add(index);
NDArray hough = d.expandDims(1).concat(theta.expandDims(1), -1);
int d_quant = 1;
int theta_quant = 2;
hough.get(":, 0").divi(d_quant);
hough.get(":, 1").divi(theta_quant);
hough = hough.floor();
float[][] houghArr = NDArrayUtils.floatNDArrayToArray(hough);
NDList ndList = hough.unique(0, true, false, true);
// 唯一的元素列表
NDArray yx_indices = ndList.get(0).toType(DataType.INT32, false);
int[][] yx_indicesArr = NDArrayUtils.intNDArrayToArray(yx_indices);
int[] inds = new int[yx_indicesArr.length];
// 唯一的元素,对应的数量
NDArray counts = ndList.get(2);
long[] countsArr = counts.toLongArray();
for (int i = 0; i < yx_indicesArr.length; i++) {
for (int j = 0; j < houghArr.length; j++) {
if (yx_indicesArr[i][0] == houghArr[j][0] && yx_indicesArr[i][1] == houghArr[j][1]) {
inds[i] = j;
break;
}
}
}
NDArray acc_map = manager.zeros(new Shape(512 / d_quant + 1, 360 / theta_quant + 1), DataType.FLOAT32);
NDArray idx_map = manager.zeros(new Shape(512 / d_quant + 1, 360 / theta_quant + 1), DataType.INT32).sub(1);
for (int i = 0; i < yx_indicesArr.length; i++) {
acc_map.set(new NDIndex(yx_indicesArr[i][0], yx_indicesArr[i][1]), countsArr[i]);
idx_map.set(new NDIndex(yx_indicesArr[i][0], yx_indicesArr[i][1]), inds[i]);
}
float[][] acc_map_np = NDArrayUtils.floatNDArrayToArray(acc_map);
NDArray max_acc_map = NDArrayUtils.maxPool(manager, acc_map, 5, 1, 2);
keep = acc_map.eq(max_acc_map);
keep = keep.toType(DataType.FLOAT32, false);
acc_map = acc_map.mul(keep);
NDArray flatten_acc_map = acc_map.flatten();
indices = flatten_acc_map.argSort(-1, false).get("0:200");
NDArray scores = flatten_acc_map.get(indices);
int cols = (int) (acc_map.getShape().get(1));
yy = indices.div(cols).floor().expandDims(-1);
xx = indices.mod(cols).expandDims(-1);
NDArray yx = yy.concat(xx, -1);
float[][] yx_arr = NDArrayUtils.floatNDArrayToArray(yx);
float[] topk_values = scores.toFloatArray();
int[][] idx_map_arr = NDArrayUtils.intNDArrayToArray(idx_map);
int[] indices_arr = new int[yx_arr.length];
for (int i = 0; i < yx_arr.length; i++) {
indices_arr[i] = idx_map_arr[(int) yx_arr[i][0]][(int) yx_arr[i][1]];
}
int basis = 5 / 2;
NDArray merged_segments = manager.zeros(new Shape(0, 4), DataType.FLOAT32);
for (int i = 0; i < yx_arr.length; i++) {
float[] yx_pt = yx_arr[i];
float y = yx_pt[0];
float x = yx_pt[1];
int max_indice = indices_arr[i];
float value = topk_values[i];
if (max_indice == -1 || value == 0) {
continue;
}
NDList segment_list = new NDList();
for (int y_offset = -basis; y_offset < basis + 1; y_offset++) {
for (int x_offset = -basis; x_offset < basis + 1; x_offset++) {
if (y + y_offset < 0 || x + x_offset < 0) {
continue;
}
int indice = idx_map_arr[(int) (y + y_offset)][(int) (x + x_offset)];
int cnt = (int) acc_map_np[(int) (y + y_offset)][(int) (x + x_offset)];
if (indice != -1) {
segment_list.add(segments.get(indice));
}
if (cnt > 1) {
int check_cnt = 1;
NDArray current_hough = hough.get(indice);
for (int new_indice = 0; new_indice < hough.size(0); new_indice++) {
NDArray new_hough = hough.get(new_indice);
if (current_hough.eq(new_hough).all().toBooleanArray()[0] && indice != new_indice) {
segment_list.add(segments.get(new_indice));
check_cnt += 1;
if (check_cnt == cnt)
break;
}
}
}
}
}
NDArray group_segments = NDArrays.concat(segment_list).reshape(-1, 2);
NDArray sorted_group_segments = group_segments.sort(0);
float[] min = sorted_group_segments.get("0, :").toFloatArray();
float[] max = sorted_group_segments.get("-1, :").toFloatArray();
float x_min = min[0];
float y_min = min[1];
float x_max = max[0];
float y_max = max[1];
float deg = theta.get(max_indice).toFloatArray()[0];
if (deg >= 90) {
merged_segments = merged_segments.concat(manager.create(new float[]{x_min, y_max, x_max, y_min}).reshape(1, 4));
} else {
merged_segments = merged_segments.concat(manager.create(new float[]{x_min, y_min, x_max, y_max}).reshape(1, 4));
}
}
// 2. get intersections
NDArray new_segments = merged_segments;
start = new_segments.get(":, :2"); // (x1, y1)
end = new_segments.get(":, 2:"); // (x2, y2)
NDArray new_centers = start.add(end).div(2.0f);
diff = start.sub(end);
NDArray dist_segments = diff.square().sum(new int[]{-1}).sqrt();
// ax + by = c
a = diff.get(":, 1");
b = diff.get(":, 0").neg();
c = a.mul(start.get(":, 0")).add(b.mul(start.get(":, 1")));
NDArray pre_det = a.expandDims(1).mul(b.expandDims(0));
NDArray det = pre_det.sub(pre_det.transpose());
NDArray pre_inter_y = a.expandDims(1).mul(c.expandDims(0));
NDArray inter_y = pre_inter_y.sub(pre_inter_y.transpose()).div(det.add(Math.exp(-10)));
NDArray pre_inter_x = c.expandDims(1).mul(b.expandDims(0));
NDArray inter_x = pre_inter_x.sub(pre_inter_x.transpose()).div(det.add(Math.exp(-10)));
NDArray inter_pts = inter_x.expandDims(2).concat(inter_y.expandDims(2), -1).toType(DataType.INT32, false);
// 3. get corner information
// 3.1 get distance
NDArray dist_inter_to_segment1_start = inter_pts.sub(start.expandDims(1)).square().sum(new int[]{-1}, true).sqrt();
NDArray dist_inter_to_segment1_end = inter_pts.sub(end.expandDims(1)).square().sum(new int[]{-1}, true).sqrt();
NDArray dist_inter_to_segment2_start = inter_pts.sub(start.expandDims(0)).square().sum(new int[]{-1}, true).sqrt();
NDArray dist_inter_to_segment2_end = inter_pts.sub(end.expandDims(0)).square().sum(new int[]{-1}, true).sqrt();
// sort ascending
NDArray dist_inter_to_segment1 = dist_inter_to_segment1_start.concat(dist_inter_to_segment1_end, -1).sort(-1);
NDArray dist_inter_to_segment2 = dist_inter_to_segment2_start.concat(dist_inter_to_segment2_end, -1).sort(-1);
// 3.2 get degree
NDArray inter_to_start = new_centers.expandDims(1).sub(inter_pts);
NDArray deg_inter_to_start = NDArrayUtils.arctan2(inter_to_start.get(":, :, 1"), inter_to_start.get(":, :, 0"));
index = deg_inter_to_start.lt(0.0f);
index = index.toType(DataType.INT32, false).mul(360);
deg_inter_to_start = deg_inter_to_start.add(index);
NDArray inter_to_end = new_centers.expandDims(0).sub(inter_pts);
// np.arctan2和np.arctan都是计算反正切值的NumPy函数，但它们的参数和返回值不同。一般来说，np.arctan2的参数为(y, x)，
NDArray deg_inter_to_end = NDArrayUtils.arctan2(inter_to_end.get(":, :, 1"), inter_to_end.get(":, :, 0"));
index = deg_inter_to_end.lt(0.0f);
index = index.toType(DataType.INT32, false).mul(360);
deg_inter_to_end = deg_inter_to_end.add(index);
// rename variables
NDArray deg1_map = deg_inter_to_start;
NDArray deg2_map = deg_inter_to_end;
// sort deg ascending
NDArray deg_sort = deg1_map.expandDims(2).concat(deg2_map.expandDims(2), -1).sort(-1);
NDArray deg_diff_map = deg1_map.sub(deg2_map).abs();
// we only consider the smallest degree of intersect
// deg_diff_map[deg_diff_map > 180] = 360 - deg_diff_map[deg_diff_map > 180]
// x -> 360- x => x + 360 - 2x = 360 - x
index = deg_diff_map.gt(180);
NDArray val1 = index.toType(DataType.INT32, false).mul(360);
NDArray val2 = index.toType(DataType.INT32, false).mul(deg_diff_map).neg().mul(2);
deg_diff_map = deg_diff_map.add(val1).add(val2);
// define available degree range
int[] deg_range = new int[]{60, 120};
ArrayList<ArrayList<int[]>> corner_dict = new ArrayList<>();
ArrayList<int[]> blueList = new ArrayList<>();
ArrayList<int[]> greenList = new ArrayList<>();
ArrayList<int[]> blackList = new ArrayList<>();
ArrayList<int[]> cyanList = new ArrayList<>();
ArrayList<int[]> redList = new ArrayList<>();
corner_dict.add(blueList);
corner_dict.add(greenList);
corner_dict.add(blackList);
corner_dict.add(cyanList);
corner_dict.add(redList);
NDArray inter_points = manager.zeros(new Shape(0, 2));
float[] dist_segments_arr = dist_segments.toFloatArray();
for (int i = 0; i < inter_pts.getShape().get(0); i++) {
for (int j = i + 1; j < inter_pts.getShape().get(1); j++) {
// i, j > line index, always i < j
int[] point1 = inter_pts.get(i + "," + j + ",:").toIntArray();
int x = point1[0];
int y = point1[1];
float[] point2 = deg_sort.get(i + "," + j + ",:").toFloatArray();
float deg1 = point2[0];
float deg2 = point2[1];
float deg_diff = deg_diff_map.getFloat(i, j);
boolean check_degree = false;
if (deg_diff > deg_range[0] && deg_diff < deg_range[1]) {
check_degree = true;
}
boolean check_distance = false;
if (((dist_inter_to_segment1.getFloat(i, j, 1) >= dist_segments_arr[i] &&
dist_inter_to_segment1.getFloat(i, j, 0) <= dist_segments_arr[i] * this.outside_ratio) ||
(dist_inter_to_segment1.getFloat(i, j, 1) <= dist_segments_arr[i] &&
dist_inter_to_segment1.getFloat(i, j, 0) <= dist_segments_arr[i] * this.inside_ratio)) &&
((dist_inter_to_segment2.getFloat(i, j, 1) >= dist_segments_arr[j] &&
dist_inter_to_segment2.getFloat(i, j, 0) <= dist_segments_arr[j] * this.outside_ratio) ||
(dist_inter_to_segment2.getFloat(i, j, 1) <= dist_segments_arr[j] &&
dist_inter_to_segment2.getFloat(i, j, 0) <= dist_segments_arr[j] * this.inside_ratio))) {
check_distance = true;
}
if (check_degree && check_distance) {
int corner_info = 0;
if ((deg1 >= 0 && deg1 <= 45 && deg2 >= 45 && deg2 <= 120) ||
(deg2 >= 315 && deg1 >= 45 && deg1 <= 120)) {
corner_info = 0; // blue
} else if (deg1 >= 45 && deg1 <= 125 && deg2 >= 125 && deg2 <= 225) {
corner_info = 1; // green
} else if (deg1 >= 125 && deg1 <= 225 && deg2 >= 225 && deg2 <= 315) {
corner_info = 2; // black
} else if ((deg1 >= 0 && deg1 <= 45 && deg2 >= 225 && deg2 <= 315) ||
(deg2 >= 315 && deg1 >= 225 && deg1 <= 315)) {
corner_info = 3; // cyan
} else {
corner_info = 4; // red - we don't use it
continue;
}
corner_dict.get(corner_info).add(new int[]{x, y, i, j});
inter_points = inter_points.concat(manager.create(new int[]{x, y}).reshape(1, 2));
}
}
}
NDArray square_list = manager.zeros(new Shape(0, 8));
NDArray connect_list = manager.zeros(new Shape(0, 4));
NDArray segment_list = manager.zeros(new Shape(0, 8));
int corner0_line = 0;
int corner1_line = 0;
int corner2_line = 0;
int corner3_line = 0;
for (int[] corner0 : corner_dict.get(0)) {
for (int[] corner1 : corner_dict.get(1)) {
boolean connect01 = false;
for (int i = 0; i < 2; i++) {
corner0_line = corner0[2 + i];
for (int j = 0; j < 2; j++) {
if (corner0_line == corner1[2 + j]) {
connect01 = true;
break;
}
}
}
if (connect01) {
for (int[] corner2 : corner_dict.get(2)) {
boolean connect12 = false;
for (int i = 0; i < 2; i++) {
corner1_line = corner1[2 + i];
for (int j = 0; j < 2; j++) {
if (corner1_line == corner2[2 + j]) {
connect12 = true;
break;
}
}
}
if (connect12) {
for (int[] corner3 : corner_dict.get(3)) {
boolean connect23 = false;
for (int i = 0; i < 2; i++) {
corner2_line = corner1[2 + i];
for (int j = 0; j < 2; j++) {
if (corner2_line == corner2[2 + j]) {
connect23 = true;
break;
}
}
}
if (connect23) {
for (int i = 0; i < 2; i++) {
corner3_line = corner3[2 + i];
for (int j = 0; j < 2; j++) {
if (corner3_line == corner0[2 + j]) {
square_list = square_list.concat(manager.create(new int[]{corner0[0], corner0[1], corner1[0], corner1[1], corner2[0], corner2[1], corner3[0], corner3[1]}).reshape(1, 8));
connect_list = connect_list.concat(manager.create(new int[]{corner0_line, corner1_line, corner2_line, corner3_line}).reshape(1, 4));
segment_list = segment_list.concat(manager.create(new int[]{corner0[2], corner0[3], corner1[2], corner1[3], corner2[2], corner2[3], corner3[2], corner3[3]}).reshape(1, 8));
}
}
}
}
}
}
}
}
}
}
float map_size = (int) imgArray.getShape().get(0) / 2;
NDArray squares = square_list.reshape(-1, 4, 2);
NDArray score_array = null;
NDArray connect_array = connect_list;
NDArray segments_array = segment_list.reshape(-1, 4, 2);
//get degree of corners:
NDArray squares_rollup = squares.duplicate();
NDArray last = squares.get(":," + (squares.size(1) - 1) + ",:");
for (int i = ((int) squares.size(1) - 1); i > 0; i--) {
squares_rollup.set(new NDIndex(":," + i + ",:"), squares.get(":," + (i - 1) + ",:"));
}
squares_rollup.set(new NDIndex(":,0,:"), last);
NDArray squares_rolldown = manager.zeros(squares.getShape());
NDArray first = squares.get(":,0,:");
for (int i = 0; i < squares.size(1) - 1; i++) {
squares_rolldown.set(new NDIndex(":," + i + ",:"), squares.get(":," + (i + 1) + ",:"));
}
squares_rolldown.set(new NDIndex(":," + (squares.size(1) - 1) + ",:"), first);
NDArray vec1 = squares_rollup.sub(squares);
NDArray normalized_vec1 = vec1.div(vec1.norm(new int[]{-1}, true).add(Math.exp(-10)));
NDArray vec2 = squares_rolldown.sub(squares);
NDArray normalized_vec2 = vec2.div(vec2.norm(new int[]{-1}, true).add(Math.exp(-10)));
NDArray inner_products = normalized_vec1.mul(normalized_vec2).sum(new int[]{-1});
NDArray squares_degree = inner_products.acos().mul(180).div(Math.PI);
NDArray overlap_scores = null;
NDArray degree_scores = null;
NDArray length_scores = null;
for (int i = 0; i < connect_array.size(0); i++) {
NDArray connects = connect_array.get(i);
segments = segments_array.get(i);
NDArray square = squares.get(i);
NDArray degree = squares_degree.get(i);
// ###################################### OVERLAP SCORES
float cover = 0;
float perimeter = 0;
// check 0 > 1 > 2 > 3
float[] square_length = new float[4];
for (int start_idx = 0; start_idx < 4; start_idx++) {
int end_idx = (start_idx + 1) % 4;
int connect_idx = (int) connects.get(start_idx).toFloatArray()[0];
NDArray start_segments = segments.get(start_idx);
NDArray end_segments = segments.get(end_idx);
// check whether outside or inside
int idx_i = (int) start_segments.toFloatArray()[0];
int idx_j = (int) start_segments.toFloatArray()[1];
NDArray check_dist_mat;
if (connect_idx == idx_i) {
check_dist_mat = dist_inter_to_segment1;
} else {
check_dist_mat = dist_inter_to_segment2;
}
float[] range = check_dist_mat.get(idx_i + "," + idx_j + ",:").toFloatArray();
float min_dist = range[0];
float max_dist = range[1];
float connect_dist = dist_segments.get(connect_idx).toFloatArray()[0];
String start_position;
float start_min;
int start_cover_param;
int start_peri_param;
if (max_dist > connect_dist) {
start_position = "outside";
start_min = min_dist;
start_cover_param = 0;
start_peri_param = 1;
} else {
start_position = "inside";
start_min = min_dist;
start_cover_param = -1;
start_peri_param = -1;
}
// check whether outside or inside
idx_i = (int) end_segments.toFloatArray()[0];
idx_j = (int) end_segments.toFloatArray()[1];
if (connect_idx == idx_i) {
check_dist_mat = dist_inter_to_segment1;
} else {
check_dist_mat = dist_inter_to_segment2;
}
range = check_dist_mat.get(idx_i + "," + idx_j + ",:").toFloatArray();
min_dist = range[0];
max_dist = range[1];
connect_dist = dist_segments.get(connect_idx).toFloatArray()[0];
String end_position;
float end_min;
int end_cover_param;
int end_peri_param;
if (max_dist > connect_dist) {
end_position = "outside";
end_min = min_dist;
end_cover_param = 0;
end_peri_param = 1;
} else {
end_position = "inside";
end_min = min_dist;
end_cover_param = -1;
end_peri_param = -1;
}
cover += connect_dist + start_cover_param * start_min + end_cover_param * end_min;
perimeter += connect_dist + start_peri_param * start_min + end_peri_param * end_min;
square_length[start_idx] = connect_dist + start_peri_param * start_min + end_peri_param * end_min;
}
if (overlap_scores == null) {
overlap_scores = manager.create(cover / perimeter).reshape(1);
} else {
overlap_scores = overlap_scores.concat(manager.create(cover / perimeter).reshape(1));
}
// ######################################
// ###################################### DEGREE SCORES
float[] degreeArr = degree.toFloatArray();
float deg0 = degreeArr[0];
float deg1 = degreeArr[1];
float deg2 = degreeArr[2];
float deg3 = degreeArr[3];
float deg_ratio1 = deg0 / deg2;
if (deg_ratio1 > 1.0) {
deg_ratio1 = 1 / deg_ratio1;
}
float deg_ratio2 = deg1 / deg3;
if (deg_ratio2 > 1.0) {
deg_ratio2 = 1 / deg_ratio2;
}
if (degree_scores == null) {
degree_scores = manager.create((deg_ratio1 + deg_ratio2) / 2).reshape(1);
} else {
degree_scores = degree_scores.concat(manager.create((deg_ratio1 + deg_ratio2) / 2).reshape(1));
}
// ######################################
// ###################################### LENGTH SCORES
float len0 = square_length[0];
float len1 = square_length[1];
float len2 = square_length[2];
float len3 = square_length[3];
float len_ratio1 = 0;
if (len2 > len0) {
len_ratio1 = len0 / len2;
} else {
len_ratio1 = len2 / len0;
}
float len_ratio2 = 0;
if (len3 > len1) {
len_ratio2 = len1 / len3;
} else {
len_ratio2 = len3 / len1;
}
if (length_scores == null) {
length_scores = manager.create((len_ratio1 + len_ratio2) / 2).reshape(1);
} else {
length_scores = length_scores.concat(manager.create((len_ratio1 + len_ratio2) / 2).reshape(1));
}
}
if (overlap_scores != null)
overlap_scores = overlap_scores.div(overlap_scores.max().toFloatArray()[0]);
// ###################################### AREA SCORES
NDArray area_scores = squares.reshape(new Shape(-1, 4, 2));
NDArray area_x = area_scores.get(":, :, 0");
NDArray area_y = area_scores.get(":, :, 1");
NDArray correction = area_x.get(":, -1").mul(area_y.get(":, 0")).sub(area_y.get(":, -1").mul(area_x.get(":, 0")));
NDArray area_scores1 = area_x.get(":, :-1").mul(area_y.get(":, 1:")).sum(new int[]{-1});
NDArray area_scores2 = area_y.get(":, :-1").mul(area_x.get(":, 1:")).sum(new int[]{-1});
area_scores = area_scores1.sub(area_scores2);
area_scores = area_scores.add(correction).abs().mul(0.5);
area_scores = area_scores.div(map_size * map_size);
// ###################################### CENTER SCORES
NDArray centers = manager.create(new float[]{256 / 2, 256 / 2});
NDArray square_centers = squares.mean(new int[]{1});
NDArray center2center = centers.sub(square_centers).square().sum().sqrt();
NDArray center_scores = center2center.div(map_size / Math.sqrt(2.0));
if (overlap_scores != null) {
score_array = overlap_scores.mul(this.w_overlap).add(degree_scores.mul(this.w_degree)).add(area_scores.mul(this.w_area)).add(center_scores.mul(this.w_center)).add(length_scores.mul(this.w_length));
NDArray sorted_idx = score_array.argSort(0, false);
score_array = score_array.get(sorted_idx);
squares = squares.get(sorted_idx);
}
try {
new_segments.get(":, 0").muli(2);
new_segments.get(":, 1").muli(2);
new_segments.get(":, 2").muli(2);
new_segments.get(":, 3").muli(2);
} catch (Exception e) {
new_segments = null;
}
try {
squares.get(":, :, 0").muli(2).divi(input_shape[1]).muli(original_shape[1]);
squares.get(":, :, 1").muli(2).divi(input_shape[0]).muli(original_shape[0]);
} catch (Exception e) {
squares = null;
score_array = null;
}
try {
inter_points.get(":, 0").muli(2);
inter_points.get(":, 1").muli(2);
} catch (Exception e) {
inter_points = null;
}
Image img = ImageFactory.getInstance().fromNDArray(imgArray);
Mat mat = (Mat) img.getWrappedImage();
if(squares.getShape().get(0) == 0)
return null;
NDArray maxSquare = squares.get(0);
float[] points = maxSquare.toFloatArray();
int[] wh = OpenCVUtils.imgCrop(points);
Mat dst = OpenCVUtils.perspectiveTransform(mat, points);
img = ImageFactory.getInstance().fromImage(dst);
// return img;
return img.getSubImage(0,0,wh[0],wh[1]);
}
}
private int[] resize64(double h, double w, double resolution) {
double k = resolution / Math.min(h, w);
h *= k;
w *= k;
int height = (int) (Math.round(h / 64.0)) * 64;
int width = (int) (Math.round(w / 64.0)) * 64;
return new int[]{height, width};
}
@Override
public Batchifier getBatchifier() {
return batchifier;
}
}
}

292
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/RecognitionModel.java Normal file
View File

@ -0,0 +1,292 @@
package top.aias.ocr.model;
import ai.djl.MalformedModelException;
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.modality.cv.output.Rectangle;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.NDManager;
import ai.djl.opencv.OpenCVImageFactory;
import ai.djl.repository.zoo.Criteria;
import ai.djl.repository.zoo.ModelNotFoundException;
import ai.djl.repository.zoo.ModelZoo;
import ai.djl.repository.zoo.ZooModel;
import ai.djl.training.util.ProgressBar;
import ai.djl.translate.TranslateException;
import top.aias.ocr.bean.Point;
import top.aias.ocr.bean.RotatedBox;
import top.aias.ocr.model.pool.DetectorPool;
import top.aias.ocr.model.pool.HorizontalDetectorPool;
import top.aias.ocr.model.pool.RecognizerPool;
import top.aias.ocr.translator.OCRDetectionTranslator;
import top.aias.ocr.translator.PpWordDetectionTranslator;
import top.aias.ocr.translator.PpWordRecognitionTranslator;
import top.aias.ocr.utils.OpenCVUtils;
import org.opencv.core.Mat;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
/**
* 文字识别模型
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public final class RecognitionModel implements AutoCloseable {
private ZooModel<Image, DetectedObjects> horizontalDetectionModel;
private ZooModel<Image, NDList> detectionModel;
private ZooModel<Image, String> recognitionModel;
private DetectorPool detectorPool;
private HorizontalDetectorPool horizontalDetectorPool;
private RecognizerPool recognizerPool;
public void init(String detModel, String recModel, int poolSize) throws MalformedModelException, ModelNotFoundException, IOException {
this.recognitionModel = ModelZoo.loadModel(recognizeCriteria(recModel));
this.detectionModel = ModelZoo.loadModel(detectCriteria(detModel));
this.horizontalDetectionModel = ModelZoo.loadModel(horizontalCriteria(detModel));
this.detectorPool = new DetectorPool(poolSize, detectionModel);
this.horizontalDetectorPool = new HorizontalDetectorPool(poolSize, horizontalDetectionModel);
this.recognizerPool = new RecognizerPool(poolSize, recognitionModel);
}
/**
* 释放资源
*/
public void close() {
this.recognitionModel.close();
this.detectionModel.close();
this.horizontalDetectionModel.close();
this.detectorPool.close();
this.horizontalDetectorPool.close();
this.recognizerPool.close();
}
/**
* 文本检测（支持有倾斜角的文本）
*
* @return
*/
private Criteria<Image, NDList> detectCriteria(String detUri) {
Criteria<Image, NDList> criteria =
Criteria.builder()
.optEngine("OnnxRuntime")
.optModelName("inference")
.setTypes(Image.class, NDList.class)
.optModelPath(Paths.get(detUri))
// .optModelUrls(detUri)
.optTranslator(new OCRDetectionTranslator(new ConcurrentHashMap<String, String>()))
.optProgress(new ProgressBar())
.build();
return criteria;
}
/**
* 水平文本检测
*
* @return
*/
private Criteria<Image, DetectedObjects> horizontalCriteria(String detUri) {
Criteria<Image, DetectedObjects> criteria =
Criteria.builder()
.optEngine("OnnxRuntime")
.optModelName("inference")
.setTypes(Image.class, DetectedObjects.class)
.optModelPath(Paths.get(detUri))
// .optModelUrls(detUri)
.optTranslator(new PpWordDetectionTranslator(new ConcurrentHashMap<String, String>()))
.optProgress(new ProgressBar())
.build();
return criteria;
}
/**
* 文本识别
*
* @return
*/
private Criteria<Image, String> recognizeCriteria(String recUri) {
Criteria<Image, String> criteria =
Criteria.builder()
.optEngine("OnnxRuntime")
.optModelName("inference")
.setTypes(Image.class, String.class)
.optModelPath(Paths.get(recUri))
// .optModelUrls(recUri)
.optProgress(new ProgressBar())
.optTranslator(new PpWordRecognitionTranslator((new ConcurrentHashMap<String, String>())))
.build();
return criteria;
}
// 多线程环境，每个线程一个predictor，共享一个model, 资源池（CPU Core 核心数）达到上限则等待
public String predictSingleLineText(Image image)
throws TranslateException {
Predictor<Image, String> recognizer = recognizerPool.getRecognizer();
String text = recognizer.predict(image);
// 释放资源
recognizerPool.releaseRecognizer(recognizer);
return text;
}
// 多线程环境，每个线程一个predictor，共享一个model, 资源池（CPU Core 核心数）达到上限则等待
public DetectedObjects predict(Image image)
throws TranslateException {
Predictor<Image, DetectedObjects> horizontalDetector = horizontalDetectorPool.getDetector();
DetectedObjects detections = horizontalDetector.predict(image);
horizontalDetectorPool.releaseDetector(horizontalDetector);
List<DetectedObjects.DetectedObject> boxes = detections.items();
List<String> names = new ArrayList<>();
List<Double> prob = new ArrayList<>();
List<BoundingBox> rect = new ArrayList<>();
Predictor<Image, String> recognizer = recognizerPool.getRecognizer();
for (int i = 0; i < boxes.size(); i++) {
Image subImg = getSubImage(image, boxes.get(i).getBoundingBox());
if (subImg.getHeight() * 1.0 / subImg.getWidth() > 1.5) {
subImg = rotateImg(subImg);
}
String name = recognizer.predict(subImg);
System.out.println(name);
names.add(name);
prob.add(-1.0);
rect.add(boxes.get(i).getBoundingBox());
}
// 释放资源
recognizerPool.releaseRecognizer(recognizer);
DetectedObjects detectedObjects = new DetectedObjects(names, prob, rect);
return detectedObjects;
}
// 多线程环境，每个线程一个predictor，共享一个model, 资源池（CPU Core 核心数）达到上限则等待
public List<RotatedBox> predict(NDManager manager, Image image)
throws TranslateException {
Predictor<Image, NDList> detector = detectorPool.getDetector();
NDList boxes = detector.predict(image);
// 释放资源
detectorPool.releaseDetector(detector);
// 交给 NDManager自动管理内存
// attach to manager for automatic memory management
boxes.attach(manager);
List<RotatedBox> result = new ArrayList<>();
Mat mat = (Mat) image.getWrappedImage();
Predictor<Image, String> recognizer = recognizerPool.getRecognizer();
for (int i = 0; i < boxes.size(); i++) {
NDArray box = boxes.get(i);
float[] pointsArr = box.toFloatArray();
float[] lt = java.util.Arrays.copyOfRange(pointsArr, 0, 2);
float[] rt = java.util.Arrays.copyOfRange(pointsArr, 2, 4);
float[] rb = java.util.Arrays.copyOfRange(pointsArr, 4, 6);
float[] lb = java.util.Arrays.copyOfRange(pointsArr, 6, 8);
int img_crop_width = (int) Math.max(distance(lt, rt), distance(rb, lb));
int img_crop_height = (int) Math.max(distance(lt, lb), distance(rt, rb));
List<Point> srcPoints = new ArrayList<>();
srcPoints.add(new Point((int) lt[0], (int) lt[1]));
srcPoints.add(new Point((int) rt[0], (int) rt[1]));
srcPoints.add(new Point((int) rb[0], (int) rb[1]));
srcPoints.add(new Point((int) lb[0], (int) lb[1]));
List<Point> dstPoints = new ArrayList<>();
dstPoints.add(new Point(0, 0));
dstPoints.add(new Point(img_crop_width, 0));
dstPoints.add(new Point(img_crop_width, img_crop_height));
dstPoints.add(new Point(0, img_crop_height));
Mat srcPoint2f = OpenCVUtils.toMat(srcPoints);
Mat dstPoint2f = OpenCVUtils.toMat(dstPoints);
Mat cvMat = OpenCVUtils.perspectiveTransform(mat, srcPoint2f, dstPoint2f);
Image subImg = OpenCVImageFactory.getInstance().fromImage(cvMat);
// ImageUtils.saveImage(subImg, i + ".png", "build/output");
subImg = subImg.getSubImage(0, 0, img_crop_width, img_crop_height);
if (subImg.getHeight() * 1.0 / subImg.getWidth() > 1.5) {
subImg = rotateImg(manager, subImg);
}
String name = recognizer.predict(subImg);
RotatedBox rotatedBox = new RotatedBox(box, name);
result.add(rotatedBox);
cvMat.release();
srcPoint2f.release();
dstPoint2f.release();
}
// 释放资源
recognizerPool.releaseRecognizer(recognizer);
return result;
}
private Image getSubImage(Image img, BoundingBox box) {
Rectangle rect = box.getBounds();
double[] extended = extendRect(rect.getX(), rect.getY(), rect.getWidth(), rect.getHeight());
int width = img.getWidth();
int height = img.getHeight();
int[] recovered = {
(int) (extended[0] * width),
(int) (extended[1] * height),
(int) (extended[2] * width),
(int) (extended[3] * height)
};
return img.getSubImage(recovered[0], recovered[1], recovered[2], recovered[3]);
}
private double[] extendRect(double xmin, double ymin, double width, double height) {
double centerx = xmin + width / 2;
double centery = ymin + height / 2;
if (width > height) {
width += height * 2.0;
height *= 3.0;
} else {
height += width * 2.0;
width *= 3.0;
}
double newX = centerx - width / 2 < 0 ? 0 : centerx - width / 2;
double newY = centery - height / 2 < 0 ? 0 : centery - height / 2;
double newWidth = newX + width > 1 ? 1 - newX : width;
double newHeight = newY + height > 1 ? 1 - newY : height;
return new double[]{newX, newY, newWidth, newHeight};
}
private float distance(float[] point1, float[] point2) {
float disX = point1[0] - point2[0];
float disY = point1[1] - point2[1];
float dis = (float) Math.sqrt(disX * disX + disY * disY);
return dis;
}
private Image rotateImg(Image image) {
try (NDManager manager = NDManager.newBaseManager()) {
NDArray rotated = NDImageUtils.rotate90(image.toNDArray(manager), 1);
return OpenCVImageFactory.getInstance().fromNDArray(rotated);
}
}
private Image rotateImg(NDManager manager, Image image) {
NDArray rotated = NDImageUtils.rotate90(image.toNDArray(manager), 1);
return OpenCVImageFactory.getInstance().fromNDArray(rotated);
}
}

282
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/TableRecognitionModel.java Normal file
View File

@ -0,0 +1,282 @@
package top.aias.ocr.model;
import ai.djl.MalformedModelException;
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.modality.cv.output.Rectangle;
import ai.djl.repository.zoo.Criteria;
import ai.djl.repository.zoo.ModelNotFoundException;
import ai.djl.repository.zoo.ModelZoo;
import ai.djl.repository.zoo.ZooModel;
import ai.djl.training.util.ProgressBar;
import ai.djl.translate.TranslateException;
import top.aias.ocr.bean.TableResult;
import top.aias.ocr.model.pool.TableRecPool;
import top.aias.ocr.translator.V2ENTableStructTranslator;
import top.aias.ocr.translator.V2TableStructTranslator;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.tuple.Pair;
import java.io.IOException;
import java.nio.file.Paths;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
/**
* 表格识别模型
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public final class TableRecognitionModel implements AutoCloseable{
private ZooModel<Image, TableResult> model;
private TableRecPool tableRecPool;
public void init(String tableUri, int poolSize) throws MalformedModelException, ModelNotFoundException, IOException {
this.model = ModelZoo.loadModel(enCriteria(tableUri));
this.tableRecPool = new TableRecPool(poolSize, model);
}
public void close() {
this.model.close();
this.tableRecPool.close();
}
private Criteria<Image, TableResult> enCriteria(String tableUri) {
/**
* 英文表格识别, paddle 部分算子不支持 paddle2onnx 的转换
*/
Criteria<Image, TableResult> criteria =
Criteria.builder()
.optEngine("PaddlePaddle")
.setTypes(Image.class, TableResult.class)
.optModelPath(Paths.get(tableUri))
// .optModelUrls(tableUri)
.optOption("removePass", "repeated_fc_relu_fuse_pass")
.optTranslator(new V2ENTableStructTranslator(new ConcurrentHashMap<String, String>()))
.optProgress(new ProgressBar())
.build();
return criteria;
}
private Criteria<Image, TableResult> chCriteria(String tableUri) {
/**
* 中文表格识别, paddle 部分算子不支持 paddle2onnx 的转换
*/
Criteria<Image, TableResult> criteria =
Criteria.builder()
.optEngine("PaddlePaddle")
.setTypes(Image.class, TableResult.class)
.optModelPath(Paths.get(tableUri))
.optOption("removePass", "repeated_fc_relu_fuse_pass")
.optTranslator(new V2TableStructTranslator(new ConcurrentHashMap<String, String>()))
.optProgress(new ProgressBar())
.build();
return criteria;
}
public String getTableHtml(Image image, DetectedObjects textDetections) throws TranslateException {
// Table cell detection
Predictor<Image, TableResult> predictor = tableRecPool.getPredictor();
TableResult tableResult = predictor.predict(image);
// 释放资源
tableRecPool.releasePredictor(predictor);
List<BoundingBox> cells = tableResult.getBoxes();
List<DetectedObjects.DetectedObject> dt_boxes = textDetections.items();
// 获取 Cell 与 文本检测框 的对应关系(1:N)。
Map<Integer, List<Integer>> matched = new ConcurrentHashMap<>();
for (int i = 0; i < dt_boxes.size(); i++) {
DetectedObjects.DetectedObject item = dt_boxes.get(i);
Rectangle textBounds = item.getBoundingBox().getBounds();
int[] box_1 = rectXYXY(textBounds, image.getWidth(), image.getHeight());
// 获取两两cell之间的L1距离和 1- IOU
List<Pair<Float, Float>> distances = new ArrayList<>();
for (BoundingBox cell : cells) {
Rectangle cellBounds = cell.getBounds();
int[] box_2 = rectXYXY(cellBounds, image.getWidth(), image.getHeight());
float distance = distance(box_1, box_2);
float iou = 1 - compute_iou(box_1, box_2);
distances.add(Pair.of(distance, iou));
}
// 根据距离和IOU挑选最"近"的cell
Pair<Float, Float> nearest = sorted(distances);
// 获取最小距离对应的下标id，也等价于cell的下标id （distances列表是根据遍历cells生成的）
int id = 0;
for (int idx = 0; idx < distances.size(); idx++) {
Pair<Float, Float> current = distances.get(idx);
if (current.getLeft().floatValue() == nearest.getLeft().floatValue()
&& current.getRight().floatValue() == nearest.getRight().floatValue()) {
id = idx;
break;
}
}
if (!matched.containsKey(id)) {
List<Integer> textIds = new ArrayList<>();
textIds.add(i);
// cell id, text id list (dt_boxes index list)
matched.put(id, textIds);
} else {
matched.get(id).add(i);
}
}
List<String> cell_contents = new ArrayList<>();
List<Double> probs = new ArrayList<>();
for (int i = 0; i < cells.size(); i++) {
List<Integer> textIds = matched.get(i);
List<String> contents = new ArrayList<>();
String content = "";
if (textIds != null) {
for (Integer id : textIds) {
DetectedObjects.DetectedObject item = dt_boxes.get(id);
contents.add(item.getClassName());
}
content = StringUtils.join(contents, " ");
}
cell_contents.add(content);
probs.add(-1.0);
}
List<String> pred_structures = tableResult.getStructure_str_list();
return getPredHtml(pred_structures, cell_contents);
}
/**
* Generate table html
*
* @param pred_structures
* @param cell_contents
* @return
*/
private String getPredHtml(List<String> pred_structures, List<String> cell_contents) {
StringBuffer html = new StringBuffer();
int td_index = 0;
for (String tag : pred_structures) {
if (tag.contains("<td></td>")) {
String content = cell_contents.get(td_index);
html.append("<td>");
html.append(content);
html.append("</td>");
td_index++;
continue;
}
html.append(tag);
}
return html.toString();
}
/**
* Calculate L1 distance
*
* @param box_1
* @param box_2
* @return
*/
private int distance(int[] box_1, int[] box_2) {
int x1 = box_1[0];
int y1 = box_1[1];
int x2 = box_1[2];
int y2 = box_1[3];
int x3 = box_2[0];
int y3 = box_2[1];
int x4 = box_2[2];
int y4 = box_2[3];
int dis = Math.abs(x3 - x1) + Math.abs(y3 - y1) + Math.abs(x4 - x2) + Math.abs(y4 - y2);
int dis_2 = Math.abs(x3 - x1) + Math.abs(y3 - y1);
int dis_3 = Math.abs(x4 - x2) + Math.abs(y4 - y2);
return dis + Math.min(dis_2, dis_3);
}
/**
* Get absolute coordinations
*
* @param rect
* @param width
* @param height
* @return
*/
private int[] rectXYXY(Rectangle rect, int width, int height) {
int left = Math.max((int) (width * rect.getX()), 0);
int top = Math.max((int) (height * rect.getY()), 0);
int right = Math.min((int) (width * (rect.getX() + rect.getWidth())), width - 1);
int bottom = Math.min((int) (height * (rect.getY() + rect.getHeight())), height - 1);
return new int[]{left, top, right, bottom};
}
/**
* computing IoU
*
* @param rec1: (y0, x0, y1, x1), which reflects (top, left, bottom, right)
* @param rec2: (y0, x0, y1, x1)
* @return scala value of IoU
*/
private float compute_iou(int[] rec1, int[] rec2) {
// computing area of each rectangles
int S_rec1 = (rec1[2] - rec1[0]) * (rec1[3] - rec1[1]);
int S_rec2 = (rec2[2] - rec2[0]) * (rec2[3] - rec2[1]);
// computing the sum_area
int sum_area = S_rec1 + S_rec2;
// find the each edge of intersect rectangle
int left_line = Math.max(rec1[1], rec2[1]);
int right_line = Math.min(rec1[3], rec2[3]);
int top_line = Math.max(rec1[0], rec2[0]);
int bottom_line = Math.min(rec1[2], rec2[2]);
// judge if there is an intersect
if (left_line >= right_line || top_line >= bottom_line) {
return 0.0f;
} else {
float intersect = (right_line - left_line) * (bottom_line - top_line);
return (intersect / (sum_area - intersect)) * 1.0f;
}
}
/**
* Distance sorted
*
* @param distances
* @return
*/
private Pair<Float, Float> sorted(List<Pair<Float, Float>> distances) {
Comparator<Pair<Float, Float>> comparator =
new Comparator<Pair<Float, Float>>() {
@Override
public int compare(Pair<Float, Float> a1, Pair<Float, Float> a2) {
// order by IoU first
if (a1.getRight().floatValue() > a2.getRight().floatValue()) {
return 1;
} else if (a1.getRight().floatValue() == a2.getRight().floatValue()) {
// Then order by L1 distance
if (a1.getLeft().floatValue() > a2.getLeft().floatValue()) {
return 1;
}
return -1;
}
return -1;
}
};
// Sort distances
List<Pair<Float, Float>> newDistances = new ArrayList<>();
CollectionUtils.addAll(newDistances, new Object[distances.size()]);
Collections.copy(newDistances, distances);
Collections.sort(newDistances, comparator);
return newDistances.get(0);
}
}

55
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/pool/DetectorPool.java Normal file
View File

@ -0,0 +1,55 @@
package top.aias.ocr.model.pool;// 导入需要的包
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.ndarray.NDList;
import ai.djl.repository.zoo.ZooModel;
import java.util.ArrayList;
/**
* 文字检测连接池
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class DetectorPool {
private int poolSize;
private ZooModel<Image, NDList> detectionModel;
private ArrayList<Predictor<Image, NDList>> detectorList = new ArrayList<>();
public DetectorPool(int poolSize, ZooModel<Image, NDList> detectionModel) {
this.poolSize = poolSize;
this.detectionModel = detectionModel;
for (int i = 0; i < poolSize; i++) {
Predictor<Image, NDList> detector = detectionModel.newPredictor();
detectorList.add(detector);
}
}
public synchronized Predictor<Image, NDList> getDetector() {
while (detectorList.isEmpty()) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Predictor<Image, NDList> detector = detectorList.remove(0);
return detector;
}
public synchronized void releaseDetector(Predictor<Image, NDList> detector) {
detectorList.add(detector);
notifyAll();
}
public void close() {
for (Predictor<Image, NDList> detector : detectorList) {
detector.close();
}
}
}

55
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/pool/HorizontalDetectorPool.java Normal file
View File

@ -0,0 +1,55 @@
package top.aias.ocr.model.pool;// 导入需要的包
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.repository.zoo.ZooModel;
import java.util.ArrayList;
/**
* 水平文字检测连接池
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class HorizontalDetectorPool {
private int poolSize;
private ZooModel<Image, DetectedObjects> detectionModel;
private ArrayList<Predictor<Image, DetectedObjects>> detectorList = new ArrayList<>();
public HorizontalDetectorPool(int poolSize, ZooModel<Image, DetectedObjects> detectionModel) {
this.poolSize = poolSize;
this.detectionModel = detectionModel;
for (int i = 0; i < poolSize; i++) {
Predictor<Image, DetectedObjects> detector = detectionModel.newPredictor();
detectorList.add(detector);
}
}
public synchronized Predictor<Image, DetectedObjects> getDetector(){
while (detectorList.isEmpty()) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Predictor<Image, DetectedObjects> detector = detectorList.remove(0);
return detector;
}
public synchronized void releaseDetector(Predictor<Image, DetectedObjects> detector) {
detectorList.add(detector);
notifyAll();
}
public void close() {
for (Predictor<Image, DetectedObjects> detector : detectorList) {
detector.close();
}
}
}

55
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/pool/LayoutPool.java Normal file
View File

@ -0,0 +1,55 @@
package top.aias.ocr.model.pool;// 导入需要的包
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.repository.zoo.ZooModel;
import java.util.ArrayList;
/**
* 布局检测连接池
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class LayoutPool {
private int poolSize;
private ZooModel<Image, DetectedObjects> model;
private ArrayList<Predictor<Image, DetectedObjects>> predictorList = new ArrayList<>();
public LayoutPool(int poolSize, ZooModel<Image, DetectedObjects> model) {
this.poolSize = poolSize;
this.model = model;
for (int i = 0; i < poolSize; i++) {
Predictor<Image, DetectedObjects> predictor = model.newPredictor();
predictorList.add(predictor);
}
}
public synchronized Predictor<Image, DetectedObjects> getPredictor() {
while (predictorList.isEmpty()) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Predictor<Image, DetectedObjects> predictor = predictorList.remove(0);
return predictor;
}
public synchronized void releasePredictor(Predictor<Image, DetectedObjects> predictor) {
predictorList.add(predictor);
notifyAll();
}
public void close() {
for (Predictor<Image, DetectedObjects> predictor : predictorList) {
predictor.close();
}
}
}

54
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/pool/MlsdPool.java Normal file
View File

@ -0,0 +1,54 @@
package top.aias.ocr.model.pool;// 导入需要的包
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.repository.zoo.ZooModel;
import java.util.ArrayList;
/**
* 文本转正连接池
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class MlsdPool {
private int poolSize;
private ZooModel<Image, Image> model;
private ArrayList<Predictor<Image, Image>> predictorList = new ArrayList<>();
public MlsdPool(int poolSize, ZooModel<Image, Image> model) {
this.poolSize = poolSize;
this.model = model;
for (int i = 0; i < poolSize; i++) {
Predictor<Image, Image> predictor = model.newPredictor();
predictorList.add(predictor);
}
}
public synchronized Predictor<Image, Image> getPredictor() {
while (predictorList.isEmpty()) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Predictor<Image, Image> predictor = predictorList.remove(0);
return predictor;
}
public synchronized void releasePredictor(Predictor<Image, Image> predictor) {
predictorList.add(predictor);
notifyAll();
}
public void close() {
for (Predictor<Image, Image> predictor : predictorList) {
predictor.close();
}
}
}

54
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/pool/RecognizerPool.java Normal file
View File

@ -0,0 +1,54 @@
package top.aias.ocr.model.pool;// 导入需要的包
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.repository.zoo.ZooModel;
import java.util.ArrayList;
/**
* 文字识别连接池
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class RecognizerPool {
private int poolSize;
private ZooModel<Image, String> recognitionModel;
private ArrayList<Predictor<Image, String>> recognizerList = new ArrayList<>();
public RecognizerPool(int poolSize, ZooModel<Image, String> detectionModel) {
this.poolSize = poolSize;
this.recognitionModel = detectionModel;
for (int i = 0; i < poolSize; i++) {
Predictor<Image, String> detector = detectionModel.newPredictor();
recognizerList.add(detector);
}
}
public synchronized Predictor<Image, String> getRecognizer(){
while (recognizerList.isEmpty()) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Predictor<Image, String> recognizer = recognizerList.remove(0);
return recognizer;
}
public synchronized void releaseRecognizer(Predictor<Image, String> recognizer) {
recognizerList.add(recognizer);
notifyAll();
}
public void close() {
for (Predictor<Image, String> detector : recognizerList) {
detector.close();
}
}
}

55
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/model/pool/TableRecPool.java Normal file
View File

@ -0,0 +1,55 @@
package top.aias.ocr.model.pool;// 导入需要的包
import ai.djl.inference.Predictor;
import ai.djl.modality.cv.Image;
import ai.djl.repository.zoo.ZooModel;
import top.aias.ocr.bean.TableResult;
import java.util.ArrayList;
/**
* 表格识别连接池
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class TableRecPool {
private int poolSize;
private ZooModel<Image, TableResult> model;
private ArrayList<Predictor<Image, TableResult>> predictorList = new ArrayList<>();
public TableRecPool(int poolSize, ZooModel<Image, TableResult> model) {
this.poolSize = poolSize;
this.model = model;
for (int i = 0; i < poolSize; i++) {
Predictor<Image, TableResult> predictor = model.newPredictor();
predictorList.add(predictor);
}
}
public synchronized Predictor<Image, TableResult> getPredictor() {
while (predictorList.isEmpty()) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Predictor<Image, TableResult> predictor = predictorList.remove(0);
return predictor;
}
public synchronized void releasePredictor(Predictor<Image, TableResult> predictor) {
predictorList.add(predictor);
notifyAll();
}
public void close() {
for (Predictor<Image, TableResult> predictor : predictorList) {
predictor.close();
}
}
}

20
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/service/InferService.java Normal file
View File

@ -0,0 +1,20 @@
package top.aias.ocr.service;
import ai.djl.modality.cv.Image;
import ai.djl.ndarray.NDManager;
import ai.djl.translate.TranslateException;
import top.aias.ocr.bean.RotatedBox;
import java.util.List;
/**
* 文字识别接口
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public interface InferService {
List<RotatedBox> getGeneralInfo(NDManager manager, Image image) throws TranslateException;
Image getWarpImg(Image image) throws TranslateException;
}

18
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/service/TableInferService.java Normal file
View File

@ -0,0 +1,18 @@
package top.aias.ocr.service;
import ai.djl.modality.cv.Image;
import ai.djl.translate.TranslateException;
import java.util.List;
/**
* 表格识别接口
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public interface TableInferService {
String getTableHtml(Image image) throws TranslateException;
List<String> getTableHtmlList(Image image) throws TranslateException;
}

43
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/service/impl/InferServiceImpl.java Normal file
View File

@ -0,0 +1,43 @@
package top.aias.ocr.service.impl;
import ai.djl.modality.cv.Image;
import ai.djl.ndarray.NDManager;
import ai.djl.translate.TranslateException;
import top.aias.ocr.bean.RotatedBox;
import top.aias.ocr.model.MlsdSquareModel;
import top.aias.ocr.model.RecognitionModel;
import top.aias.ocr.service.InferService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.List;
/**
* 文字识别服务
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Service
public class InferServiceImpl implements InferService {
private Logger logger = LoggerFactory.getLogger(InferServiceImpl.class);
@Autowired
private RecognitionModel recognitionModel;
@Autowired
private MlsdSquareModel mlsdSquareModel;
public List<RotatedBox> getGeneralInfo(NDManager manager, Image image) throws TranslateException {
List<RotatedBox> detectedObjects = recognitionModel.predict(manager, image);
return detectedObjects;
}
public Image getWarpImg(Image image) throws TranslateException {
Image cropImg = mlsdSquareModel.predict(image);
return cropImg;
}
}

76
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/service/impl/TableInferServiceImpl.java Normal file
View File

@ -0,0 +1,76 @@
package top.aias.ocr.service.impl;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.modality.cv.output.Rectangle;
import ai.djl.translate.TranslateException;
import top.aias.ocr.model.LayoutDetectionModel;
import top.aias.ocr.model.RecognitionModel;
import top.aias.ocr.model.TableRecognitionModel;
import top.aias.ocr.service.TableInferService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
/**
* 表格识别服务
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
@Service
public class TableInferServiceImpl implements TableInferService {
private Logger logger = LoggerFactory.getLogger(TableInferServiceImpl.class);
@Autowired
private RecognitionModel recognitionModel;
@Autowired
private TableRecognitionModel tableDetectionModel;
@Autowired
private LayoutDetectionModel layoutDetectionModel;
public String getTableHtml(Image image) throws TranslateException {
DetectedObjects textDetections = recognitionModel.predict(image);
String tableHtml = tableDetectionModel.getTableHtml(image, textDetections);
return tableHtml;
}
public List<String> getTableHtmlList(Image image) throws TranslateException {
List<String> tableHtmlList = new ArrayList<>();
DetectedObjects layoutDetections = layoutDetectionModel.predict(image);
List<DetectedObjects.DetectedObject> boxes = layoutDetections.items();
for (int i = 0; i < boxes.size(); i++) {
if (boxes.get(i).getClassName().equalsIgnoreCase("table")) {
Image subImage = getSubImage(image, boxes.get(i).getBoundingBox());
// 表格单元检测
// Table cell detection
DetectedObjects textDetections = recognitionModel.predict(subImage);
String tableHtml = tableDetectionModel.getTableHtml(subImage, textDetections);
tableHtmlList.add(tableHtml);
}
}
return tableHtmlList;
}
private Image getSubImage(Image img, BoundingBox box) {
Rectangle rect = box.getBounds();
int width = img.getWidth();
int height = img.getHeight();
int[] recovered = {
(int) (rect.getX() * width),
(int) (rect.getY() * height),
(int) (rect.getWidth() * width),
(int) (rect.getHeight() * height)
};
return img.getSubImage(recovered[0], recovered[1], recovered[2], recovered[3]);
}
}

520
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/translator/OCRDetectionTranslator.java Normal file
View File

@ -0,0 +1,520 @@
package top.aias.ocr.translator;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDArrays;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.NDManager;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import top.aias.ocr.utils.NDArrayUtils;
import org.opencv.core.*;
import org.opencv.imgproc.Imgproc;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
/**
* 文字识别前后处理（支持倾斜文本）
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class OCRDetectionTranslator implements Translator<Image, NDList> {
// det_algorithm == "DB"
private final float thresh = 0.3f;
private final boolean use_dilation = false;
private final String score_mode = "fast";
private final String box_type = "quad";
private final int limit_side_len;
private final int max_candidates;
private final int min_size;
private final float box_thresh;
private final float unclip_ratio;
private float ratio_h;
private float ratio_w;
private int img_height;
private int img_width;
public OCRDetectionTranslator(Map<String, ?> arguments) {
limit_side_len =
arguments.containsKey("limit_side_len")
? Integer.parseInt(arguments.get("limit_side_len").toString())
: 960;
max_candidates =
arguments.containsKey("max_candidates")
? Integer.parseInt(arguments.get("max_candidates").toString())
: 1000;
min_size =
arguments.containsKey("min_size")
? Integer.parseInt(arguments.get("min_size").toString())
: 3;
box_thresh =
arguments.containsKey("box_thresh")
? Float.parseFloat(arguments.get("box_thresh").toString())
: 0.6f; // 0.5f
unclip_ratio =
arguments.containsKey("unclip_ratio")
? Float.parseFloat(arguments.get("unclip_ratio").toString())
: 1.6f;
}
@Override
public NDList processOutput(TranslatorContext ctx, NDList list) {
NDManager manager = ctx.getNDManager();
NDArray pred = list.get(0);
pred = pred.squeeze();
NDArray segmentation = pred.gt(thresh); // thresh=0.3 .mul(255f)
segmentation = segmentation.toType(DataType.UINT8, true);
Shape shape = segmentation.getShape();
int rows = (int) shape.get(0);
int cols = (int) shape.get(1);
Mat newMask = new Mat();
if (this.use_dilation) {
Mat mask = new Mat();
//convert from NDArray to Mat
Mat srcMat = NDArrayUtils.uint8NDArrayToMat(segmentation);
// size 越小，腐蚀的单位越小，图片越接近原图
// Mat dilation_kernel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(2, 2));
Mat dilation_kernel = NDArrayUtils.uint8ArrayToMat(new byte[][]{{1, 1}, {1, 1}});
/**
* 膨胀说明： 图像的一部分区域与指定的核进行卷积， 求核的最`大`值并赋值给指定区域。 膨胀可以理解为图像中`高亮区域`的'领域扩大'。
* 意思是高亮部分会侵蚀不是高亮的部分，使高亮部分越来越多。
*/
Imgproc.dilate(srcMat, mask, dilation_kernel);
//destination Matrix
Scalar scalar = new Scalar(255);
Core.multiply(mask, scalar, newMask);
// release Mat
mask.release();
srcMat.release();
dilation_kernel.release();
} else {
Mat srcMat = NDArrayUtils.uint8NDArrayToMat(segmentation);
//destination Matrix
Scalar scalar = new Scalar(255);
Core.multiply(srcMat, scalar, newMask);
// release Mat
srcMat.release();
}
NDArray boxes = boxes_from_bitmap(manager, pred, newMask);
//boxes[:, :, 0] = boxes[:, :, 0] / ratio_w
NDArray boxes1 = boxes.get(":, :, 0").div(ratio_w);
boxes.set(new NDIndex(":, :, 0"), boxes1);
//boxes[:, :, 1] = boxes[:, :, 1] / ratio_h
NDArray boxes2 = boxes.get(":, :, 1").div(ratio_h);
boxes.set(new NDIndex(":, :, 1"), boxes2);
NDList dt_boxes = this.filter_tag_det_res(boxes);
dt_boxes.detach();
// release Mat
newMask.release();
return dt_boxes;
}
private NDList filter_tag_det_res(NDArray dt_boxes) {
NDList boxesList = new NDList();
int num = (int) dt_boxes.getShape().get(0);
for (int i = 0; i < num; i++) {
NDArray box = dt_boxes.get(i);
box = order_points_clockwise(box);
box = clip_det_res(box);
float[] box0 = box.get(0).toFloatArray();
float[] box1 = box.get(1).toFloatArray();
float[] box3 = box.get(3).toFloatArray();
int rect_width = (int) Math.sqrt(Math.pow(box1[0] - box0[0], 2) + Math.pow(box1[1] - box0[1], 2));
int rect_height = (int) Math.sqrt(Math.pow(box3[0] - box0[0], 2) + Math.pow(box3[1] - box0[1], 2));
if (rect_width <= 3 || rect_height <= 3)
continue;
boxesList.add(box);
}
return boxesList;
}
private NDArray clip_det_res(NDArray points) {
for (int i = 0; i < points.getShape().get(0); i++) {
int value = Math.max((int) points.get(i, 0).toFloatArray()[0], 0);
value = Math.min(value, img_width - 1);
points.set(new NDIndex(i + ",0"), value);
value = Math.max((int) points.get(i, 1).toFloatArray()[0], 0);
value = Math.min(value, img_height - 1);
points.set(new NDIndex(i + ",1"), value);
}
return points;
}
/**
* sort the points based on their x-coordinates
* 顺时针
*
* @param pts
* @return
*/
private NDArray order_points_clockwise(NDArray pts) {
NDList list = new NDList();
long[] indexes = pts.get(":, 0").argSort().toLongArray();
// grab the left-most and right-most points from the sorted
// x-roodinate points
Shape s1 = pts.getShape();
NDArray leftMost1 = pts.get(indexes[0] + ",:");
NDArray leftMost2 = pts.get(indexes[1] + ",:");
NDArray leftMost = leftMost1.concat(leftMost2).reshape(2, 2);
NDArray rightMost1 = pts.get(indexes[2] + ",:");
NDArray rightMost2 = pts.get(indexes[3] + ",:");
NDArray rightMost = rightMost1.concat(rightMost2).reshape(2, 2);
// now, sort the left-most coordinates according to their
// y-coordinates so we can grab the top-left and bottom-left
// points, respectively
indexes = leftMost.get(":, 1").argSort().toLongArray();
NDArray lt = leftMost.get(indexes[0] + ",:");
NDArray lb = leftMost.get(indexes[1] + ",:");
indexes = rightMost.get(":, 1").argSort().toLongArray();
NDArray rt = rightMost.get(indexes[0] + ",:");
NDArray rb = rightMost.get(indexes[1] + ",:");
list.add(lt);
list.add(rt);
list.add(rb);
list.add(lb);
NDArray rect = NDArrays.concat(list).reshape(4, 2);
return rect;
}
/**
* Get boxes from the binarized image predicted by DB
*
* @param manager
* @param pred the binarized image predicted by DB.
* @param bitmap new 'pred' after threshold filtering.
*/
private NDArray boxes_from_bitmap(NDManager manager, NDArray pred, Mat bitmap) {
int dest_height = (int) pred.getShape().get(0);
int dest_width = (int) pred.getShape().get(1);
int height = bitmap.rows();
int width = bitmap.cols();
List<MatOfPoint> contours = new ArrayList<>();
Mat hierarchy = new Mat();
// 寻找轮廓
Imgproc.findContours(
bitmap,
contours,
hierarchy,
Imgproc.RETR_LIST,
Imgproc.CHAIN_APPROX_SIMPLE);
int num_contours = Math.min(contours.size(), max_candidates);
NDList boxList = new NDList();
float[] scores = new float[num_contours];
for (int index = 0; index < num_contours; index++) {
MatOfPoint contour = contours.get(index);
MatOfPoint2f newContour = new MatOfPoint2f(contour.toArray());
float[][] pointsArr = new float[4][2];
int sside = get_mini_boxes(newContour, pointsArr);
if (sside < this.min_size)
continue;
NDArray points = manager.create(pointsArr);
float score = box_score_fast(manager, pred, points);
if (score < this.box_thresh)
continue;
NDArray box = unclip(manager, points); // TODO get_mini_boxes(box)
// box[:, 0] = np.clip(np.round(box[:, 0] / width * dest_width), 0, dest_width)
NDArray boxes1 = box.get(":,0").div(width).mul(dest_width).round().clip(0, dest_width);
box.set(new NDIndex(":, 0"), boxes1);
// box[:, 1] = np.clip(np.round(box[:, 1] / height * dest_height), 0, dest_height)
NDArray boxes2 = box.get(":,1").div(height).mul(dest_height).round().clip(0, dest_height);
box.set(new NDIndex(":, 1"), boxes2);
boxList.add(box);
scores[index] = score;
// release memory
contour.release();
newContour.release();
}
NDArray boxes = NDArrays.stack(boxList);
// release
hierarchy.release();
return boxes;
}
/**
* Shrink or expand the boxaccording to 'unclip_ratio'
*
* @param points The predicted box.
* @return uncliped box
*/
private NDArray unclip(NDManager manager, NDArray points) {
points = order_points_clockwise(points);
float[] pointsArr = points.toFloatArray();
float[] lt = java.util.Arrays.copyOfRange(pointsArr, 0, 2);
float[] lb = java.util.Arrays.copyOfRange(pointsArr, 6, 8);
float[] rt = java.util.Arrays.copyOfRange(pointsArr, 2, 4);
float[] rb = java.util.Arrays.copyOfRange(pointsArr, 4, 6);
float width = distance(lt, rt);
float height = distance(lt, lb);
if (width > height) {
float k = (lt[1] - rt[1]) / (lt[0] - rt[0]); // y = k * x + b
float delta_dis = height;
float delta_x = (float) Math.sqrt((delta_dis * delta_dis) / (k * k + 1));
float delta_y = Math.abs(k * delta_x);
if (k > 0) {
pointsArr[0] = lt[0] - delta_x + delta_y;
pointsArr[1] = lt[1] - delta_y - delta_x;
pointsArr[2] = rt[0] + delta_x + delta_y;
pointsArr[3] = rt[1] + delta_y - delta_x;
pointsArr[4] = rb[0] + delta_x - delta_y;
pointsArr[5] = rb[1] + delta_y + delta_x;
pointsArr[6] = lb[0] - delta_x - delta_y;
pointsArr[7] = lb[1] - delta_y + delta_x;
} else {
pointsArr[0] = lt[0] - delta_x - delta_y;
pointsArr[1] = lt[1] + delta_y - delta_x;
pointsArr[2] = rt[0] + delta_x - delta_y;
pointsArr[3] = rt[1] - delta_y - delta_x;
pointsArr[4] = rb[0] + delta_x + delta_y;
pointsArr[5] = rb[1] - delta_y + delta_x;
pointsArr[6] = lb[0] - delta_x + delta_y;
pointsArr[7] = lb[1] + delta_y + delta_x;
}
} else {
float k = (lt[1] - rt[1]) / (lt[0] - rt[0]); // y = k * x + b
float delta_dis = width;
float delta_y = (float) Math.sqrt((delta_dis * delta_dis) / (k * k + 1));
float delta_x = Math.abs(k * delta_y);
if (k > 0) {
pointsArr[0] = lt[0] + delta_x - delta_y;
pointsArr[1] = lt[1] - delta_y - delta_x;
pointsArr[2] = rt[0] + delta_x + delta_y;
pointsArr[3] = rt[1] - delta_y + delta_x;
pointsArr[4] = rb[0] - delta_x + delta_y;
pointsArr[5] = rb[1] + delta_y + delta_x;
pointsArr[6] = lb[0] - delta_x - delta_y;
pointsArr[7] = lb[1] + delta_y - delta_x;
} else {
pointsArr[0] = lt[0] - delta_x - delta_y;
pointsArr[1] = lt[1] - delta_y + delta_x;
pointsArr[2] = rt[0] - delta_x + delta_y;
pointsArr[3] = rt[1] - delta_y - delta_x;
pointsArr[4] = rb[0] + delta_x + delta_y;
pointsArr[5] = rb[1] + delta_y - delta_x;
pointsArr[6] = lb[0] + delta_x - delta_y;
pointsArr[7] = lb[1] + delta_y + delta_x;
}
}
points = manager.create(pointsArr).reshape(4, 2);
return points;
}
private float distance(float[] point1, float[] point2) {
float disX = point1[0] - point2[0];
float disY = point1[1] - point2[1];
float dis = (float) Math.sqrt(disX * disX + disY * disY);
return dis;
}
/**
* Get boxes from the contour or box.
*
* @param contour The predicted contour.
* @param pointsArr The predicted box.
* @return smaller side of box
*/
private int get_mini_boxes(MatOfPoint2f contour, float[][] pointsArr) {
// https://blog.csdn.net/qq_37385726/article/details/82313558
// bounding_box[1] - rect 返回矩形的长和宽
RotatedRect rect = Imgproc.minAreaRect(contour);
Mat points = new Mat();
Imgproc.boxPoints(rect, points);
float[][] fourPoints = new float[4][2];
for (int row = 0; row < 4; row++) {
fourPoints[row][0] = (float) points.get(row, 0)[0];
fourPoints[row][1] = (float) points.get(row, 1)[0];
}
float[] tmpPoint = new float[2];
for (int i = 0; i < 4; i++) {
for (int j = i + 1; j < 4; j++) {
if (fourPoints[j][0] < fourPoints[i][0]) {
tmpPoint[0] = fourPoints[i][0];
tmpPoint[1] = fourPoints[i][1];
fourPoints[i][0] = fourPoints[j][0];
fourPoints[i][1] = fourPoints[j][1];
fourPoints[j][0] = tmpPoint[0];
fourPoints[j][1] = tmpPoint[1];
}
}
}
int index_1 = 0;
int index_2 = 1;
int index_3 = 2;
int index_4 = 3;
if (fourPoints[1][1] > fourPoints[0][1]) {
index_1 = 0;
index_4 = 1;
} else {
index_1 = 1;
index_4 = 0;
}
if (fourPoints[3][1] > fourPoints[2][1]) {
index_2 = 2;
index_3 = 3;
} else {
index_2 = 3;
index_3 = 2;
}
pointsArr[0] = fourPoints[index_1];
pointsArr[1] = fourPoints[index_2];
pointsArr[2] = fourPoints[index_3];
pointsArr[3] = fourPoints[index_4];
int height = rect.boundingRect().height;
int width = rect.boundingRect().width;
int sside = Math.min(height, width);
// release
points.release();
return sside;
}
/**
* Calculate the score of box.
*
* @param bitmap The binarized image predicted by DB.
* @param points The predicted box
* @return
*/
private float box_score_fast(NDManager manager, NDArray bitmap, NDArray points) {
NDArray box = points.get(":");
long h = bitmap.getShape().get(0);
long w = bitmap.getShape().get(1);
// xmin = np.clip(np.floor(box[:, 0].min()).astype(np.int), 0, w - 1)
int xmin = box.get(":, 0").min().floor().clip(0, w - 1).toType(DataType.INT32, true).toIntArray()[0];
int xmax = box.get(":, 0").max().ceil().clip(0, w - 1).toType(DataType.INT32, true).toIntArray()[0];
int ymin = box.get(":, 1").min().floor().clip(0, h - 1).toType(DataType.INT32, true).toIntArray()[0];
int ymax = box.get(":, 1").max().ceil().clip(0, h - 1).toType(DataType.INT32, true).toIntArray()[0];
NDArray mask = manager.zeros(new Shape(ymax - ymin + 1, xmax - xmin + 1), DataType.UINT8);
box.set(new NDIndex(":, 0"), box.get(":, 0").sub(xmin));
box.set(new NDIndex(":, 1"), box.get(":, 1").sub(ymin));
//mask - convert from NDArray to Mat
Mat maskMat = NDArrayUtils.uint8NDArrayToMat(mask);
//mask - convert from NDArray to Mat - 4 rows, 2 cols
Mat boxMat = NDArrayUtils.floatNDArrayToMat(box, CvType.CV_32S);
// boxMat.reshape(1, new int[]{1, 4, 2});
List<MatOfPoint> pts = new ArrayList<>();
MatOfPoint matOfPoint = NDArrayUtils.matToMatOfPoint(boxMat); // new MatOfPoint(boxMat);
pts.add(matOfPoint);
Imgproc.fillPoly(maskMat, pts, new Scalar(1));
NDArray subBitMap = bitmap.get(ymin + ":" + (ymax + 1) + "," + xmin + ":" + (xmax + 1));
Mat bitMapMat = NDArrayUtils.floatNDArrayToMat(subBitMap);
Scalar score = Core.mean(bitMapMat, maskMat);
float scoreValue = (float) score.val[0];
// release
maskMat.release();
boxMat.release();
bitMapMat.release();
return scoreValue;
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
NDArray img = input.toNDArray(ctx.getNDManager());
int h = input.getHeight();
int w = input.getWidth();
img_height = h;
img_width = w;
// limit the max side
float ratio = 1.0f;
if (Math.max(h, w) > limit_side_len) {
if (h > w) {
ratio = (float) limit_side_len / (float) h;
} else {
ratio = (float) limit_side_len / (float) w;
}
}
int resize_h = (int) (h * ratio);
int resize_w = (int) (w * ratio);
resize_h = Math.round((float) resize_h / 32f) * 32;
resize_w = Math.round((float) resize_w / 32f) * 32;
ratio_h = resize_h / (float) h;
ratio_w = resize_w / (float) w;
img = NDImageUtils.resize(img, resize_w, resize_h);
img = NDImageUtils.toTensor(img);
img =
NDImageUtils.normalize(
img,
new float[]{0.485f, 0.456f, 0.406f},
new float[]{0.229f, 0.224f, 0.225f});
img = img.expandDims(0);
return new NDList(img);
}
@Override
public Batchifier getBatchifier() {
return null;
}
}

314
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/translator/PicoDetLayoutDetectionTranslator.java Normal file
View File

@ -0,0 +1,314 @@
package top.aias.ocr.translator;
import ai.djl.Model;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.modality.cv.output.Rectangle;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDArrays;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.NDManager;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import ai.djl.util.Utils;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
/**
* 布局检测前后处理
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class PicoDetLayoutDetectionTranslator implements Translator<Image, DetectedObjects> {
private List<String> table;
NDArray scale_factor;
Shape input_shape;
Shape ori_shape;
int[] strides = new int[]{8, 16, 32, 64};
float score_threshold = 0.6f; // default = 0.4
float nms_threshold = 0.5f;
int nms_top_k = 1000;
int keep_top_k = 100;
private int width;
private int height;
public PicoDetLayoutDetectionTranslator() {
}
@Override
public void prepare(TranslatorContext ctx) throws IOException {
Model model = ctx.getModel();
try (InputStream is = model.getArtifact("dict.txt").openStream()) {
table = Utils.readLines(is, true);
}
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
NDArray img = input.toNDArray(ctx.getNDManager());
width = input.getWidth();
height = input.getHeight();
img = NDImageUtils.resize(img, 608, 800);
img = img.transpose(2, 0, 1).div(255);
img =
NDImageUtils.normalize(
img, new float[]{0.485f, 0.456f, 0.406f}, new float[]{0.229f, 0.224f, 0.225f});
img = img.expandDims(0);
// (im_scale_y im_scale_x)
scale_factor = ctx.getNDManager().create(new float[]{800f / height, 608f / width});
input_shape = new Shape(800, 608);
ori_shape = input_shape; //new Shape(height, width);
return new NDList(img);
}
@Override
public DetectedObjects processOutput(TranslatorContext ctx, NDList list) {
// NDManager manager = ctx.getNDManager();
try (NDManager manager =
NDManager.newBaseManager(ctx.getNDManager().getDevice(), "PyTorch")) {
NDList raw_boxes = list.subNDList(4);
NDList scores = list;
scores.removeAll(raw_boxes);
int dimension = raw_boxes.get(0).getShape().dimension();
int reg_max = (int) (raw_boxes.get(0).getShape().get(dimension - 1) / 4 - 1);
NDArray out_boxes_list = manager.zeros(new Shape(0, 6));
List<BoundingBox> boxesList = new ArrayList<>();
List<String> namesList = new ArrayList<>();
List<Double> probsList = new ArrayList<>();
NDArray bboxes = manager.zeros(new Shape(0, 4));
NDArray confidences = manager.zeros(new Shape(0, scores.get(0).getShape().get(2))); // CN 10, EN 5
for (int i = 0; i < scores.size(); i++) {
NDArray box_distribute = raw_boxes.get(i);
box_distribute = box_distribute.squeeze();
NDArray score = scores.get(i);
score = score.squeeze(0);
int stride = strides[i];
// centers
float fm_h = input_shape.get(0) / (float) stride;
float fm_w = input_shape.get(1) / (float) stride;
NDArray h_range = manager.arange(fm_h);
NDArray w_range = manager.arange(fm_w);
w_range = w_range.reshape(1, w_range.size());
NDArray ww = manager.zeros(new Shape(0, w_range.size()));
for (int j = 0; j < h_range.size(); j++) {
ww = ww.concat(w_range, 0);
}
h_range = h_range.reshape(h_range.size(), 1);
NDArray hh = manager.zeros(new Shape(h_range.size(), 0));
for (int j = 0; j < w_range.size(); j++) {
hh = hh.concat(h_range, 1);
}
NDArray ct_row = hh.flatten().add(0.5).mul(stride);
NDArray ct_col = ww.flatten().add(0.5).mul(stride);
ct_row = ct_row.reshape(ct_row.size(), 1);
ct_col = ct_col.reshape(ct_col.size(), 1);
NDArray center = ct_col.concat(ct_row, 1).concat(ct_col, 1).concat(ct_row, 1);
// box distribution to distance
NDArray reg_range = manager.arange(reg_max + 1);
NDArray box_distance = box_distribute.reshape(-1, reg_max + 1);
box_distance = box_distance.softmax(1);
box_distance = box_distance.mul(reg_range.expandDims(0));
box_distance = box_distance.sum(new int[]{1}).reshape(-1, 4);
box_distance = box_distance.mul(stride);
// top K candidate
NDArray topk_idx = score.max(new int[]{1}).argSort(0, false);
topk_idx = topk_idx.get(new NDIndex(":" + this.nms_top_k));
center = center.get(topk_idx);
score = score.get(topk_idx);
box_distance = box_distance.get(topk_idx);
// decode box
NDArray decode_box = center.add(manager.create(new int[]{-1, -1, 1, 1}).mul(box_distance));
bboxes = bboxes.concat(decode_box, 0);
confidences = confidences.concat(score, 0);
}
// nms
NDArray picked_box_probs = manager.zeros(new Shape(0, 5));
ArrayList<Integer> picked_labels = new ArrayList<>();
for (int class_index = 0; class_index < confidences.getShape().get(1); class_index++) {
NDArray probs = confidences.get(new NDIndex(":," + class_index));
NDArray mask = probs.gt(this.score_threshold);
probs = probs.get(mask);
if (probs.getShape().get(0) == 0) {
continue;
}
NDArray subset_boxes = bboxes.get(mask);
NDArray box_probs = subset_boxes.concat(probs.reshape(-1, 1), 1);
box_probs = hard_nms(manager, box_probs, this.nms_threshold, this.keep_top_k, 200);
picked_box_probs = picked_box_probs.concat(box_probs);
for (int i = 0; i < box_probs.size(0); i++) {
picked_labels.add(class_index);
}
}
if (picked_box_probs.size() == 0) {
// out_boxes_list.concat(manager.zeros(new Shape(0,4)));
// out_boxes_num.concat(manager.create(0));
} else {
// resize output boxes
NDArray wb = warp_boxes(manager, picked_box_probs.get(new NDIndex(":, :4")));
picked_box_probs.set(new NDIndex(":, :4"), wb);
NDArray im_scale = scale_factor.flip(0).concat(scale_factor.flip(0));
picked_box_probs.set(new NDIndex(":, :4"), picked_box_probs.get(new NDIndex(":, :4")).div(im_scale));
// clas score box
float[] arr = new float[picked_labels.size()];
for (int i = 0; i < picked_labels.size(); i++) {
arr[i] = picked_labels.get(i);
}
int rows = picked_labels.size();
NDArray labels = manager.create(arr).reshape(rows, 1);
NDArray picked_box_prob_1 = picked_box_probs.get(new NDIndex(":, 4")).reshape(rows, 1);
NDArray picked_box_prob_2 = picked_box_probs.get(new NDIndex(":, :4"));
NDArray out_boxes = labels.concat(picked_box_prob_1, 1).concat(picked_box_prob_2, 1);
out_boxes_list = out_boxes_list.concat(out_boxes);
}
for (int i = 0; i < out_boxes_list.size(0); i++) {
NDArray dt = out_boxes_list.get(i);
float[] array = dt.toFloatArray();
// if (array[1] <= 0.5 || array[0] <= -1) continue;
int clsid = (int) array[0];
double score = array[1];
String name = table.get(clsid);
float x = array[2] / width;
float y = array[3] / height;
float w = (array[4] - array[2]) / width;
float h = (array[5] - array[3]) / height;
Rectangle rect = new Rectangle(x, y, w, h);
boxesList.add(rect);
namesList.add(name);
probsList.add(score);
}
return new DetectedObjects(namesList, probsList, boxesList);
}
}
private NDArray warp_boxes(NDManager manager, NDArray boxes) {
int width = (int) ori_shape.get(1);
int height = (int) ori_shape.get(0);
int n = (int) boxes.size(0);
if (n > 0) {
// warp points
NDArray xy = manager.ones(new Shape(n * 4, 3));
NDArray box1 = boxes.get(new NDIndex(":,0")).reshape(n, 1);
NDArray box2 = boxes.get(new NDIndex(":,3")).reshape(n, 1);
NDArray box3 = boxes.get(new NDIndex(":,2")).reshape(n, 1);
NDArray box4 = boxes.get(new NDIndex(":,1")).reshape(n, 1);
boxes = boxes.concat(box1, 1).concat(box2, 1).concat(box3, 1).concat(box4, 1);
boxes = boxes.reshape(n * 4, 2);
xy.set(new NDIndex(":, :2"), boxes);
xy = xy.get(new NDIndex(":, :2")).div(xy.get(new NDIndex(":, 2:3"))).reshape(n, 8);
// create new boxes
NDArray xy0 = xy.get(new NDIndex(":,0")).reshape(n, 1);
NDArray xy2 = xy.get(new NDIndex(":,2")).reshape(n, 1);
NDArray xy4 = xy.get(new NDIndex(":,4")).reshape(n, 1);
NDArray xy6 = xy.get(new NDIndex(":,6")).reshape(n, 1);
NDArray x = xy0.concat(xy2, 1).concat(xy4, 1).concat(xy6, 1);
NDArray xy1 = xy.get(new NDIndex(":,1")).reshape(n, 1);
NDArray xy3 = xy.get(new NDIndex(":,3")).reshape(n, 1);
NDArray xy5 = xy.get(new NDIndex(":,5")).reshape(n, 1);
NDArray xy7 = xy.get(new NDIndex(":,7")).reshape(n, 1);
NDArray y = xy1.concat(xy3, 1).concat(xy5, 1).concat(xy7, 1);
xy = x.min(new int[]{1}).concat(y.min(new int[]{1})).concat(x.max(new int[]{1})).concat(y.max(new int[]{1})).reshape(4, n).transpose();
// clip boxes
xy.set(new NDIndex(":,0"), xy.get(new NDIndex(":,0")).clip(0, width));
xy.set(new NDIndex(":,2"), xy.get(new NDIndex(":,2")).clip(0, width));
xy.set(new NDIndex(":,1"), xy.get(new NDIndex(":,1")).clip(0, height));
xy.set(new NDIndex(":,3"), xy.get(new NDIndex(":,3")).clip(0, height));
return xy;
} else {
return boxes;
}
}
private NDArray hard_nms(NDManager manager, NDArray box_scores, float iou_threshold, int top_k, int candidate_size) {
NDArray scores = box_scores.get(new NDIndex(":, -1"));
NDArray boxes = box_scores.get(new NDIndex(":, :-1"));
NDArray indexes = scores.argSort();
if (candidate_size < indexes.size()) {
indexes = indexes.get(new NDIndex((-candidate_size) + ":"));
}
NDArray picked = manager.zeros(new Shape(0), DataType.INT64);
while (indexes.size() > 0) {
NDArray current = indexes.get(new NDIndex("-1")).reshape(1);
picked = picked.concat(current, 0);
if (top_k == picked.size() || indexes.size() == 1) {
break;
}
NDArray current_box = boxes.get(current);
indexes = indexes.get(new NDIndex(":-1"));
NDArray rest_boxes = boxes.get(indexes);
NDArray iou = iou_of(rest_boxes, current_box.expandDims(0));
iou = iou.squeeze();
if (iou.getShape().dimension() == 0)
iou = iou.reshape(1);
NDArray cutOff = iou.lte(iou_threshold);
indexes = indexes.get(cutOff);
}
return box_scores.get(picked);
}
private NDArray iou_of(NDArray boxes0, NDArray boxes1) {
NDArray overlap_left_top = NDArrays.maximum(boxes0.get(new NDIndex("..., :2")), boxes1.get(new NDIndex("..., :2")));
NDArray overlap_right_bottom = NDArrays.minimum(boxes0.get(new NDIndex("..., 2:")), boxes1.get(new NDIndex("..., 2:")));
NDArray overlap_area = area_of(overlap_left_top, overlap_right_bottom);
NDArray area0 = area_of(boxes0.get(new NDIndex("..., :2")), boxes0.get(new NDIndex("..., 2:")));
NDArray area1 = area_of(boxes1.get(new NDIndex("..., :2")), boxes1.get(new NDIndex("..., 2:")));
return overlap_area.div(area0.add(area1).sub(overlap_area).add(Math.exp(-5)));
}
private NDArray area_of(NDArray left_top, NDArray right_bottom) {
NDArray hw = right_bottom.sub(left_top).clip(0.0f, Float.MAX_VALUE);
return hw.get(new NDIndex("..., 0")).mul(hw.get(new NDIndex("..., 1")));
}
@Override
public Batchifier getBatchifier() {
return null;
}
}

114
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/translator/PpWordDetectionTranslator.java Normal file
View File

@ -0,0 +1,114 @@
package top.aias.ocr.translator;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.paddlepaddle.zoo.cv.objectdetection.BoundFinder;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.stream.IntStream;
/**
* 水平文字检测前后处理
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class PpWordDetectionTranslator implements Translator<Image, DetectedObjects> {
private final int max_side_len;
public PpWordDetectionTranslator(Map<String, ?> arguments) {
max_side_len =
arguments.containsKey("maxLength")
? Integer.parseInt(arguments.get("maxLength").toString())
: 960;
}
@Override
public DetectedObjects processOutput(TranslatorContext ctx, NDList list) {
NDArray result = list.get(0);
result = result.squeeze().mul(255f).toType(DataType.UINT8, true).gt(0.3); // thresh=0.3
boolean[] flattened = result.toBooleanArray();
Shape shape = result.getShape();
int w = (int) shape.get(0);
int h = (int) shape.get(1);
boolean[][] grid = new boolean[w][h];
IntStream.range(0, flattened.length)
.parallel()
.forEach(i -> grid[i / h][i % h] = flattened[i]);
List<BoundingBox> boxes = new BoundFinder(grid).getBoxes();
List<String> names = new ArrayList<>();
List<Double> probs = new ArrayList<>();
int boxSize = boxes.size();
for (int i = 0; i < boxSize; i++) {
names.add("word");
probs.add(1.0);
}
return new DetectedObjects(names, probs, boxes);
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
NDArray img = input.toNDArray(ctx.getNDManager());
int h = input.getHeight();
int w = input.getWidth();
int resize_w = w;
int resize_h = h;
// limit the max side
float ratio = 1.0f;
if (Math.max(resize_h, resize_w) > max_side_len) {
if (resize_h > resize_w) {
ratio = (float) max_side_len / (float) resize_h;
} else {
ratio = (float) max_side_len / (float) resize_w;
}
}
resize_h = (int) (resize_h * ratio);
resize_w = (int) (resize_w * ratio);
if (resize_h % 32 == 0) {
resize_h = resize_h;
} else if (Math.floor((float) resize_h / 32f) <= 1) {
resize_h = 32;
} else {
resize_h = (int) Math.floor((float) resize_h / 32f) * 32;
}
if (resize_w % 32 == 0) {
resize_w = resize_w;
} else if (Math.floor((float) resize_w / 32f) <= 1) {
resize_w = 32;
} else {
resize_w = (int) Math.floor((float) resize_w / 32f) * 32;
}
img = NDImageUtils.resize(img, resize_w, resize_h);
img = NDImageUtils.toTensor(img);
img =
NDImageUtils.normalize(
img,
new float[]{0.485f, 0.456f, 0.406f},
new float[]{0.229f, 0.224f, 0.225f});
img = img.expandDims(0);
return new NDList(img);
}
@Override
public Batchifier getBatchifier() {
return null;
}
}

125
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/translator/PpWordRecognitionTranslator.java Normal file
View File

@ -0,0 +1,125 @@
package top.aias.ocr.translator;
import ai.djl.Model;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import ai.djl.util.Utils;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
/**
* 文字识别前后处理
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class PpWordRecognitionTranslator implements Translator<Image, String> {
private List<String> table;
private final boolean use_space_char;
public PpWordRecognitionTranslator(Map<String, ?> arguments) {
use_space_char =
arguments.containsKey("use_space_char")
? Boolean.parseBoolean(arguments.get("use_space_char").toString())
: true;
}
@Override
public void prepare(TranslatorContext ctx) throws IOException {
Model model = ctx.getModel();
try (InputStream is = model.getArtifact("dict.txt").openStream()) {
table = Utils.readLines(is, true);
table.add(0, "blank");
if(use_space_char){
table.add(" ");
table.add(" ");
}
else{
table.add("");
table.add("");
}
}
}
@Override
public String processOutput(TranslatorContext ctx, NDList list) throws IOException {
StringBuilder sb = new StringBuilder();
NDArray tokens = list.singletonOrThrow();
long[] indices = tokens.get(0).argMax(1).toLongArray();
boolean[] selection = new boolean[indices.length];
Arrays.fill(selection, true);
for (int i = 1; i < indices.length; i++) {
if (indices[i] == indices[i - 1]) {
selection[i] = false;
}
}
// 字符置信度
// float[] probs = new float[indices.length];
// for (int row = 0; row < indices.length; row++) {
// NDArray value = tokens.get(0).get(new NDIndex(""+ row +":" + (row + 1) +"," + indices[row] +":" + ( indices[row] + 1)));
// probs[row] = value.toFloatArray()[0];
// }
int lastIdx = 0;
for (int i = 0; i < indices.length; i++) {
if (selection[i] == true && indices[i] > 0 && !(i > 0 && indices[i] == lastIdx)) {
sb.append(table.get((int) indices[i]));
}
}
return sb.toString();
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
NDArray img = input.toNDArray(ctx.getNDManager(), Image.Flag.COLOR);
int imgC = 3;
int imgH = 48;
int imgW = 320;
float max_wh_ratio = (float) imgW / (float) imgH;
int h = input.getHeight();
int w = input.getWidth();
float wh_ratio = (float) w / (float) h;
max_wh_ratio = Math.max(max_wh_ratio,wh_ratio);
imgW = (int)(imgH * max_wh_ratio);
int resized_w;
if (Math.ceil(imgH * wh_ratio) > imgW) {
resized_w = imgW;
} else {
resized_w = (int) (Math.ceil(imgH * wh_ratio));
}
NDArray resized_image = NDImageUtils.resize(img, resized_w, imgH);
resized_image = resized_image.transpose(2, 0, 1).toType(DataType.FLOAT32,false);
resized_image.divi(255f).subi(0.5f).divi(0.5f);
NDArray padding_im = ctx.getNDManager().zeros(new Shape(imgC, imgH, imgW), DataType.FLOAT32);
padding_im.set(new NDIndex(":,:,0:" + resized_w), resized_image);
padding_im = padding_im.flip(0);
padding_im = padding_im.expandDims(0);
return new NDList(padding_im);
}
@Override
public Batchifier getBatchifier() {
return null;
}
}

251
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/translator/TableStructTranslator.java Normal file
View File

@ -0,0 +1,251 @@
package top.aias.ocr.translator;
import ai.djl.Model;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.Rectangle;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDArrays;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.opencv.OpenCVImageFactory;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import ai.djl.util.Utils;
import top.aias.ocr.bean.TableResult;
import org.apache.commons.lang3.StringUtils;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/**
* 表格识别前后处理
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class TableStructTranslator implements Translator<Image, TableResult> {
private final int maxLength;
private int height;
private int width;
private float xScale = 1.0f;
private float yScale = 1.0f;
public TableStructTranslator(Map<String, ?> arguments) {
maxLength =
arguments.containsKey("maxLength")
? Integer.parseInt(arguments.get("maxLength").toString())
: 488;
}
private Map<String, String> dict_idx_character = new ConcurrentHashMap<>();
private Map<String, String> dict_character_idx = new ConcurrentHashMap<>();
private Map<String, String> dict_idx_elem = new ConcurrentHashMap<>();
private Map<String, String> dict_elem_idx = new ConcurrentHashMap<>();
private String beg_str = "sos";
private String end_str = "eos";
@Override
public void prepare(TranslatorContext ctx) throws IOException {
Model model = ctx.getModel();
// ppocr_keys_v1.txt
try (InputStream is = model.getArtifact("table_structure_dict.txt").openStream()) {
List<String> lines = Utils.readLines(is, false);
String[] substr = lines.get(0).trim().split("\\t");
int characterNum = Integer.parseInt(substr[0]);
int elemNum = Integer.parseInt(substr[1]);
List<String> listCharacter = new ArrayList<>();
List<String> listElem = new ArrayList<>();
for (int i = 1; i < 1 + characterNum; i++) {
listCharacter.add(lines.get(i).trim());
}
for (int i = 1 + characterNum; i < 1 + characterNum + elemNum; i++) {
listElem.add(lines.get(i).trim());
}
listCharacter.add(0, beg_str);
listCharacter.add(end_str);
listElem.add(0, beg_str);
listElem.add(end_str);
for (int i = 0; i < listCharacter.size(); i++) {
dict_idx_character.put("" + i, listCharacter.get(i));
dict_character_idx.put(listCharacter.get(i), "" + i);
}
for (int i = 0; i < listElem.size(); i++) {
dict_idx_elem.put("" + i, listElem.get(i));
dict_elem_idx.put(listElem.get(i), "" + i);
}
}
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
NDArray img = input.toNDArray(ctx.getNDManager(), Image.Flag.COLOR);
height = input.getHeight();
width = input.getWidth();
// img = ResizeTableImage(img, height, width, maxLength);
// img = PaddingTableImage(ctx, img, maxLength);
img = NDImageUtils.resize(img, 488, 488);
// img = NDImageUtils.toTensor(img);
img = img.transpose(2, 0, 1).div(255).flip(0);
img =
NDImageUtils.normalize(
img, new float[] {0.485f, 0.456f, 0.406f}, new float[] {0.229f, 0.224f, 0.225f});
img = img.expandDims(0);
return new NDList(img);
}
@Override
public TableResult processOutput(TranslatorContext ctx, NDList list) {
NDArray locPreds = list.get(0);
NDArray structureProbs = list.get(1);
NDArray structure_idx = structureProbs.argMax(2);
NDArray structure_probs = structureProbs.max(new int[] {2});
List<List<String>> result_list = new ArrayList<>();
List<List<String>> result_pos_list = new ArrayList<>();
List<List<String>> result_score_list = new ArrayList<>();
List<List<String>> result_elem_idx_list = new ArrayList<>();
List<String> res_html_code_list = new ArrayList<>();
List<NDArray> res_loc_list = new ArrayList<>();
// get ignored tokens
int beg_idx = Integer.parseInt(dict_elem_idx.get(beg_str));
int end_idx = Integer.parseInt(dict_elem_idx.get(end_str));
long batch_size = structure_idx.size(0); // len(text_index)
for (int batch_idx = 0; batch_idx < batch_size; batch_idx++) {
List<String> char_list = new ArrayList<>();
List<String> elem_pos_list = new ArrayList<>();
List<String> elem_idx_list = new ArrayList<>();
List<String> score_list = new ArrayList<>();
long len = structure_idx.get(batch_idx).size();
for (int idx = 0; idx < len; idx++) {
int tmp_elem_idx = (int) structure_idx.get(batch_idx).get(idx).toLongArray()[0];
if (idx > 0 && tmp_elem_idx == end_idx) {
break;
}
if (tmp_elem_idx == beg_idx || tmp_elem_idx == end_idx) {
continue;
}
char_list.add(dict_idx_elem.get("" + tmp_elem_idx));
elem_pos_list.add("" + idx);
score_list.add("" + structure_probs.get(batch_idx, idx).toFloatArray()[0]);
elem_idx_list.add("" + tmp_elem_idx);
}
result_list.add(char_list); // structure_str
result_pos_list.add(elem_pos_list);
result_score_list.add(score_list);
result_elem_idx_list.add(elem_idx_list);
}
int batch_num = result_list.size();
for (int bno = 0; bno < batch_num; bno++) {
NDList res_loc = new NDList();
int len = result_list.get(bno).size();
for (int sno = 0; sno < len; sno++) {
String text = result_list.get(bno).get(sno);
if (text.equals("<td>") || text.equals("<td")) {
int pos = Integer.parseInt(result_pos_list.get(bno).get(sno));
res_loc.add(locPreds.get(bno, pos));
}
}
String res_html_code = StringUtils.join(result_list.get(bno), "");
res_html_code_list.add(res_html_code);
NDArray array = NDArrays.stack(res_loc);
res_loc_list.add(array);
}
// structure_str_list result_list
// res_loc res_loc_list
List<BoundingBox> boxes = new ArrayList<>();
long rows = res_loc_list.get(0).size(0);
for (int rno = 0; rno < rows; rno++) {
float[] arr = res_loc_list.get(0).get(rno).toFloatArray();
Rectangle rect = new Rectangle(arr[0], arr[1], (arr[2] - arr[0]), (arr[3] - arr[1]));
boxes.add(rect);
}
List<String> structure_str_list = result_list.get(0);
structure_str_list.add(0, "<table>");
structure_str_list.add(0, "<body>");
structure_str_list.add(0, "<html>");
structure_str_list.add("</table>");
structure_str_list.add("</body>");
structure_str_list.add("</html>");
TableResult result = new TableResult(structure_str_list, boxes);
return result;
}
@Override
public Batchifier getBatchifier() {
return null;
}
private NDArray ResizeTableImage(NDArray img, int height, int width, int maxLen) {
int localMax = Math.max(height, width);
float ratio = maxLen * 1.0f / localMax;
int resize_h = (int) (height * ratio);
int resize_w = (int) (width * ratio);
if(width > height){
xScale = 1.0f;
yScale = ratio;
} else{
xScale = ratio;
yScale = 1.0f;
}
img = NDImageUtils.resize(img, resize_w, resize_h);
return img;
}
private NDArray PaddingTableImage(TranslatorContext ctx, NDArray img, int maxLen) {
Image srcImg = OpenCVImageFactory.getInstance().fromNDArray(img.duplicate());
saveImage(srcImg, "img.png", "build/output");
NDArray paddingImg = ctx.getNDManager().zeros(new Shape(maxLen, maxLen, 3), DataType.UINT8);
// NDManager manager = NDManager.newBaseManager();
// NDArray paddingImg = manager.zeros(new Shape(maxLen, maxLen, 3), DataType.UINT8);
paddingImg.set(
new NDIndex("0:" + img.getShape().get(0) + ",0:" + img.getShape().get(1) + ",:"), img);
// Image image = OpenCVImageFactory.getInstance().fromNDArray(paddingImg);
// saveImage(image, "paddingImg.png", "build/output");
return paddingImg;
}
public void saveImage(Image img, String name, String path) {
Path outputDir = Paths.get(path);
Path imagePath = outputDir.resolve(name);
try {
img.save(Files.newOutputStream(imagePath), "png");
} catch (IOException e) {
e.printStackTrace();
}
}
}

216
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/translator/V2ENTableStructTranslator.java Normal file
View File

@ -0,0 +1,216 @@
package top.aias.ocr.translator;
import ai.djl.Model;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.Rectangle;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import ai.djl.util.Utils;
import top.aias.ocr.bean.TableResult;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/**
* 英文表格识别前后处理
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class V2ENTableStructTranslator implements Translator<Image, TableResult> {
private final int maxLength = 488;
private int height;
private int width;
private float scale = 1.0f;
private float xScale = 1.0f;
private float yScale = 1.0f;
public V2ENTableStructTranslator(Map<String, ?> arguments) {
}
private List<String> dict;
private Map<String, String> dict_idx_character = new ConcurrentHashMap<>();
private Map<String, String> dict_character_idx = new ConcurrentHashMap<>();
private Map<String, String> dict_idx_elem = new ConcurrentHashMap<>();
private Map<String, String> dict_elem_idx = new ConcurrentHashMap<>();
private String beg_str = "sos";
private String end_str = "eos";
private List<String> td_token = new ArrayList<>();
@Override
public void prepare(TranslatorContext ctx) throws IOException {
Model model = ctx.getModel();
try (InputStream is = model.getArtifact("table_structure_dict.txt").openStream()) {
dict = Utils.readLines(is, false);
dict.add(0, beg_str);
if (dict.contains("<td>"))
dict.remove("<td>");
if (!dict.contains("<td></td>"))
dict.add("<td></td>");
dict.add(end_str);
}
td_token.add("<td>");
td_token.add("<td");
td_token.add("<td></td>");
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
NDArray img = input.toNDArray(ctx.getNDManager(), Image.Flag.COLOR);
height = input.getHeight();
width = input.getWidth();
img = ResizeTableImage(img, height, width, maxLength);
img = PaddingTableImage(ctx, img, maxLength);
// img = NDImageUtils.resize(img, 488, 488); // maxLength = 488
// img = NDImageUtils.toTensor(img);
img = img.transpose(2, 0, 1).div(255).flip(0);
img =
NDImageUtils.normalize(
img, new float[]{0.485f, 0.456f, 0.406f}, new float[]{0.229f, 0.224f, 0.225f});
img = img.expandDims(0);
return new NDList(img);
}
@Override
public TableResult processOutput(TranslatorContext ctx, NDList list) {
NDArray bbox_preds = list.get(0);
NDArray structure_probs = list.get(1);
NDArray structure_idx = structure_probs.argMax(2);
structure_probs = structure_probs.max(new int[]{2});
List<List<String>> structure_batch_list = new ArrayList<>();
List<List<NDArray>> bbox_batch_list = new ArrayList<>();
List<List<NDArray>> result_score_list = new ArrayList<>();
// get ignored tokens
int beg_idx = dict.indexOf(beg_str);
int end_idx = dict.indexOf(end_str);
long batch_size = structure_idx.size(0);
for (int batch_idx = 0; batch_idx < batch_size; batch_idx++) {
List<String> structure_list = new ArrayList<>();
List<NDArray> bbox_list = new ArrayList<>();
List<NDArray> score_list = new ArrayList<>();
long len = structure_idx.get(batch_idx).size();
for (int idx = 0; idx < len; idx++) {
int char_idx = (int) structure_idx.get(batch_idx).get(idx).toLongArray()[0];
if (idx > 0 && char_idx == end_idx) {
break;
}
// if (char_idx == beg_idx || char_idx == end_idx) {
// continue;
// }
String text = dict.get(char_idx);
if (td_token.indexOf(text) > -1) {
NDArray bbox = bbox_preds.get(batch_idx, idx);
// bbox.set(new NDIndex("0::2"), bbox.get(new NDIndex("0::2")));
// bbox.set(new NDIndex("1::2"), bbox.get(new NDIndex("1::2")));
bbox_list.add(bbox);
}
structure_list.add(text);
score_list.add(structure_probs.get(batch_idx, idx));
}
structure_batch_list.add(structure_list); // structure_str
bbox_batch_list.add(bbox_list);
result_score_list.add(score_list);
}
List<String> structure_str_list = structure_batch_list.get(0);
List<NDArray> bbox_list = bbox_batch_list.get(0);
structure_str_list.add(0, "<html>");
structure_str_list.add(1, "<body>");
structure_str_list.add(2, "<table>");
structure_str_list.add("</table>");
structure_str_list.add("</body>");
structure_str_list.add("</html>");
List<BoundingBox> boxes = new ArrayList<>();
long rows = bbox_list.size();
for (int rno = 0; rno < rows; rno++) {
NDArray box = bbox_list.get(rno);
float[] arr = new float[4];
arr[0] = box.get(new NDIndex("0::2")).min().toFloatArray()[0];
arr[1] = box.get(new NDIndex("1::2")).min().toFloatArray()[0];
arr[2] = box.get(new NDIndex("0::2")).max().toFloatArray()[0];
arr[3] = box.get(new NDIndex("1::2")).max().toFloatArray()[0];
Rectangle rect = new Rectangle(arr[0] * xScale, arr[1] * yScale, (arr[2] - arr[0]) * xScale, (arr[3] - arr[1]) * yScale);
boxes.add(rect);
}
TableResult result = new TableResult(structure_str_list, boxes);
return result;
}
@Override
public Batchifier getBatchifier() {
return null;
}
private NDArray ResizeTableImage(NDArray img, int height, int width, int maxLen) {
int localMax = Math.max(height, width);
float ratio = maxLen * 1.0f / localMax;
int resize_h = (int) (height * ratio);
int resize_w = (int) (width * ratio);
scale = ratio;
if (width > height) {
xScale = 1f;
yScale = (float) maxLength / (float) resize_h;
} else {
xScale = (float) maxLength / (float) resize_w;
yScale = 1f;
}
img = NDImageUtils.resize(img, resize_w, resize_h);
return img;
}
private NDArray PaddingTableImage(TranslatorContext ctx, NDArray img, int maxLen) {
NDArray paddingImg = ctx.getNDManager().zeros(new Shape(maxLen, maxLen, 3), DataType.UINT8);
paddingImg.set(
new NDIndex("0:" + img.getShape().get(0) + ",0:" + img.getShape().get(1) + ",:"), img);
// Image image = OpenCVImageFactory.getInstance().fromNDArray(paddingImg);
// saveImage(image, "paddingImg.png", "build/output");
return paddingImg;
}
public void saveImage(Image img, String name, String path) {
Path outputDir = Paths.get(path);
Path imagePath = outputDir.resolve(name);
try {
img.save(Files.newOutputStream(imagePath), "png");
} catch (IOException e) {
e.printStackTrace();
}
}
}

216
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/translator/V2TableStructTranslator.java Normal file
View File

@ -0,0 +1,216 @@
package top.aias.ocr.translator;
import ai.djl.Model;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.BoundingBox;
import ai.djl.modality.cv.output.Rectangle;
import ai.djl.modality.cv.util.NDImageUtils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.translate.Batchifier;
import ai.djl.translate.Translator;
import ai.djl.translate.TranslatorContext;
import ai.djl.util.Utils;
import top.aias.ocr.bean.TableResult;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/**
* 中文表格识别前后处理
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class V2TableStructTranslator implements Translator<Image, TableResult> {
private final int maxLength = 488;
private int height;
private int width;
private float scale = 1.0f;
private float xScale;
private float yScale;
public V2TableStructTranslator(Map<String, ?> arguments) {
}
private List<String> dict;
private Map<String, String> dict_idx_character = new ConcurrentHashMap<>();
private Map<String, String> dict_character_idx = new ConcurrentHashMap<>();
private Map<String, String> dict_idx_elem = new ConcurrentHashMap<>();
private Map<String, String> dict_elem_idx = new ConcurrentHashMap<>();
private String beg_str = "sos";
private String end_str = "eos";
private List<String> td_token = new ArrayList<>();
@Override
public void prepare(TranslatorContext ctx) throws IOException {
Model model = ctx.getModel();
try (InputStream is = model.getArtifact("table_structure_dict_ch.txt").openStream()) {
dict = Utils.readLines(is, false);
dict.add(0,beg_str);
if(dict.contains("<td>"))
dict.remove("<td>");
if(!dict.contains("<td></td>"))
dict.add("<td></td>");
dict.add(end_str);
}
td_token.add("<td>");
td_token.add("<td");
td_token.add("<td></td>");
}
@Override
public NDList processInput(TranslatorContext ctx, Image input) {
NDArray img = input.toNDArray(ctx.getNDManager(), Image.Flag.COLOR);
height = input.getHeight();
width = input.getWidth();
img = ResizeTableImage(img, height, width, maxLength);
img = PaddingTableImage(ctx, img, maxLength);
// img = NDImageUtils.resize(img, 488, 488); // maxLength = 488
// img = NDImageUtils.toTensor(img);
img = img.transpose(2, 0, 1).div(255).flip(0);
img =
NDImageUtils.normalize(
img, new float[]{0.485f, 0.456f, 0.406f}, new float[]{0.229f, 0.224f, 0.225f});
img = img.expandDims(0);
return new NDList(img);
}
@Override
public TableResult processOutput(TranslatorContext ctx, NDList list) {
NDArray bbox_preds = list.get(0);
NDArray structure_probs = list.get(1);
NDArray structure_idx = structure_probs.argMax(2);
structure_probs = structure_probs.max(new int[]{2});
List<List<String>> structure_batch_list = new ArrayList<>();
List<List<NDArray>> bbox_batch_list = new ArrayList<>();
List<List<NDArray>> result_score_list = new ArrayList<>();
// get ignored tokens
int beg_idx = dict.indexOf(beg_str);
int end_idx = dict.indexOf(end_str);
long batch_size = structure_idx.size(0);
for (int batch_idx = 0; batch_idx < batch_size; batch_idx++) {
List<String> structure_list = new ArrayList<>();
List<NDArray> bbox_list = new ArrayList<>();
List<NDArray> score_list = new ArrayList<>();
long len = structure_idx.get(batch_idx).size();
for (int idx = 0; idx < len; idx++) {
int char_idx = (int) structure_idx.get(batch_idx).get(idx).toLongArray()[0];
if (idx > 0 && char_idx == end_idx) {
break;
}
// if (char_idx == beg_idx || char_idx == end_idx) {
// continue;
// }
String text = dict.get(char_idx);
if(td_token.indexOf(text)>-1){
NDArray bbox = bbox_preds.get(batch_idx, idx);
// bbox.set(new NDIndex("0::2"), bbox.get(new NDIndex("0::2")));
// bbox.set(new NDIndex("1::2"), bbox.get(new NDIndex("1::2")));
bbox_list.add(bbox);
}
structure_list.add(text);
score_list.add(structure_probs.get(batch_idx, idx));
}
structure_batch_list.add(structure_list); // structure_str
bbox_batch_list.add(bbox_list);
result_score_list.add(score_list);
}
List<String> structure_str_list =structure_batch_list.get(0);
List<NDArray> bbox_list = bbox_batch_list.get(0);
structure_str_list.add(0,"<html>");
structure_str_list.add(1,"<body>");
structure_str_list.add(2,"<table>");
structure_str_list.add("</table>");
structure_str_list.add("</body>");
structure_str_list.add("</html>");
List<BoundingBox> boxes = new ArrayList<>();
long rows = bbox_list.size();
for (int rno = 0; rno < rows; rno++) {
NDArray box = bbox_list.get(rno);
float[] arr = new float[4];
arr[0] = box.get(new NDIndex("0::2")).min().toFloatArray()[0];
arr[1] = box.get(new NDIndex("1::2")).min().toFloatArray()[0];
arr[2] = box.get(new NDIndex("0::2")).max().toFloatArray()[0];
arr[3] = box.get(new NDIndex("1::2")).max().toFloatArray()[0];
Rectangle rect = new Rectangle(arr[0] * xScale, arr[1] * yScale, (arr[2] - arr[0]) * xScale, (arr[3] - arr[1]) * yScale);
boxes.add(rect);
}
TableResult result = new TableResult(structure_str_list, boxes);
return result;
}
@Override
public Batchifier getBatchifier() {
return null;
}
private NDArray ResizeTableImage(NDArray img, int height, int width, int maxLen) {
int localMax = Math.max(height, width);
float ratio = maxLen * 1.0f / localMax;
int resize_h = (int) (height * ratio);
int resize_w = (int) (width * ratio);
scale = ratio;
if(width > height){
xScale = 1f;
yScale = (float)maxLength /(float)resize_h;
} else{
xScale = (float)maxLength /(float)resize_w;
yScale = 1f;
}
img = NDImageUtils.resize(img, resize_w, resize_h);
return img;
}
private NDArray PaddingTableImage(TranslatorContext ctx, NDArray img, int maxLen) {
NDArray paddingImg = ctx.getNDManager().zeros(new Shape(maxLen, maxLen, 3), DataType.UINT8);
paddingImg.set(
new NDIndex("0:" + img.getShape().get(0) + ",0:" + img.getShape().get(1) + ",:"), img);
// Image image = ImageFactory.getInstance().fromNDArray(paddingImg);
// saveImage(image, "paddingImg.png", "build/output");
return paddingImg;
}
public void saveImage(Image img, String name, String path) {
Path outputDir = Paths.get(path);
Path imagePath = outputDir.resolve(name);
try {
img.save(Files.newOutputStream(imagePath), "png");
} catch (IOException e) {
e.printStackTrace();
}
}
}

25
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/Constants.java Normal file
View File

@ -0,0 +1,25 @@
package top.aias.ocr.utils;
/**
* 配置常量
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class Constants {
/**
* win systems
*/
public static final String WIN = "win";
/**
* mac system
*/
public static final String MAC = "mac";
/**
* mac system
*/
public static final String LINUX = "linux";
}

253
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/ConvertHtml2Excel.java Normal file
View File

@ -0,0 +1,253 @@
package top.aias.ocr.utils;
import top.aias.ocr.bean.CrossRangeCellMeta;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.math.NumberUtils;
import org.apache.poi.hssf.usermodel.*;
import org.apache.poi.ss.usermodel.BorderStyle;
import org.apache.poi.ss.usermodel.CellType;
import org.apache.poi.ss.usermodel.HorizontalAlignment;
import org.apache.poi.ss.usermodel.VerticalAlignment;
import org.apache.poi.ss.util.CellRangeAddress;
import org.dom4j.Document;
import org.dom4j.DocumentException;
import org.dom4j.DocumentHelper;
import org.dom4j.Element;
import java.util.ArrayList;
import java.util.List;
/**
* @Auther: xiaoqiang
* @Date: 2020/12/9 9:16
* @Description:
*/
public class ConvertHtml2Excel {
/**
* html表格转excel
* Convert HTML table to Excel
*
* @param tableHtml
* <table>
* ..
* </table>
* @return
*/
public static HSSFWorkbook table2Excel(String tableHtml) {
HSSFWorkbook wb = new HSSFWorkbook();
HSSFSheet sheet = wb.createSheet();
List<CrossRangeCellMeta> crossRowEleMetaLs = new ArrayList<>();
int rowIndex = 0;
try {
Document data = DocumentHelper.parseText(tableHtml);
// 生成表头
// generate header
Element thead = data.getRootElement().element("thead");
HSSFCellStyle titleStyle = getTitleStyle(wb);
int ls=0;//列数 //column number
if (thead != null) {
List<Element> trLs = thead.elements("tr");
for (Element trEle : trLs) {
HSSFRow row = sheet.createRow(rowIndex);
List<Element> thLs = trEle.elements("td");
ls=thLs.size();
makeRowCell(thLs, rowIndex, row, 0, titleStyle, crossRowEleMetaLs);
rowIndex++;
}
}
// 生成表体
// generate body
Element tbody = data.getRootElement().element("tbody");
HSSFCellStyle contentStyle = getContentStyle(wb);
if (tbody != null) {
List<Element> trLs = tbody.elements("tr");
for (Element trEle : trLs) {
HSSFRow row = sheet.createRow(rowIndex);
List<Element> thLs = trEle.elements("th");
int cellIndex = makeRowCell(thLs, rowIndex, row, 0, titleStyle, crossRowEleMetaLs);
List<Element> tdLs = trEle.elements("td");
makeRowCell(tdLs, rowIndex, row, cellIndex, contentStyle, crossRowEleMetaLs);
rowIndex++;
}
}
// 合并表头
// merge header
for (CrossRangeCellMeta crcm : crossRowEleMetaLs) {
sheet.addMergedRegion(new CellRangeAddress(crcm.getFirstRow(), crcm.getLastRow(), crcm.getFirstCol(), crcm.getLastCol()));
setRegionStyle(sheet, new CellRangeAddress(crcm.getFirstRow(), crcm.getLastRow(), crcm.getFirstCol(), crcm.getLastCol()),titleStyle);
}
for(int i=0;i<sheet.getRow(0).getPhysicalNumberOfCells();i++){
sheet.autoSizeColumn(i, true);
//设置列宽
//set column width
if(sheet.getColumnWidth(i)<255*256){
sheet.setColumnWidth(i, sheet.getColumnWidth(i) < 9000 ? 9000 : sheet.getColumnWidth(i));
}else{
sheet.setColumnWidth(i, 15000);
}
}
} catch (DocumentException e) {
e.printStackTrace();
}
return wb;
}
/**
* 生产行内容
* Generate row content
*
* @return 最后一列的cell index - the index of the last cell
*/
/**
* @param tdLs th或者td集合 - th or td list
* @param rowIndex 行号 - row number
* @param row POI行对象 - POI row object
* @param startCellIndex
* @param cellStyle 样式 - style
* @param crossRowEleMetaLs 跨行元数据集合 - row and column span metadata set
* @return
*/
private static int makeRowCell(List<Element> tdLs, int rowIndex, HSSFRow row, int startCellIndex, HSSFCellStyle cellStyle,
List<CrossRangeCellMeta> crossRowEleMetaLs) {
int i = startCellIndex;
for (int eleIndex = 0; eleIndex < tdLs.size(); i++, eleIndex++) {
int captureCellSize = getCaptureCellSize(rowIndex, i, crossRowEleMetaLs);
while (captureCellSize > 0) {
for (int j = 0; j < captureCellSize; j++) {
// 当前行跨列处理（补单元格）
//handle the current row span (fill in cells)
row.createCell(i);
i++;
}
captureCellSize = getCaptureCellSize(rowIndex, i, crossRowEleMetaLs);
}
Element thEle = tdLs.get(eleIndex);
String val = thEle.getTextTrim();
if (StringUtils.isBlank(val)) {
Element e = thEle.element("a");
if (e != null) {
val = e.getTextTrim();
}
}
HSSFCell c = row.createCell(i);
if (NumberUtils.isNumber(val)) {
c.setCellValue(Double.parseDouble(val));
c.setCellType(CellType.NUMERIC);
} else {
c.setCellValue(val);
}
int rowSpan = NumberUtils.toInt(thEle.attributeValue("rowspan"), 1);
int colSpan = NumberUtils.toInt(thEle.attributeValue("colspan"), 1);
c.setCellStyle(cellStyle);
if (rowSpan > 1 || colSpan > 1) {
// 存在跨行或跨列
//exists row and column span
crossRowEleMetaLs.add(new CrossRangeCellMeta(rowIndex, i, rowSpan, colSpan));
}
if (colSpan > 1) {
// 当前行跨列处理（补单元格）
// handle the current row span (fill in cells)
for (int j = 1; j < colSpan; j++) {
i++;
row.createCell(i);
}
}
}
return i;
}
/**
* 设置合并单元格的边框样式
* Set the border style of the merged cells
*
* @param sheet
* @param region
* @param cs
*/
public static void setRegionStyle(HSSFSheet sheet, CellRangeAddress region, HSSFCellStyle cs) {
for (int i = region.getFirstRow(); i <= region.getLastRow(); i++) {
HSSFRow row = sheet.getRow(i);
for (int j = region.getFirstColumn(); j <= region.getLastColumn(); j++) {
HSSFCell cell = row.getCell(j);
cell.setCellStyle(cs);
}
}
}
/**
* 获得因rowSpan占据的单元格
* Get the cells occupied by rowSpan
*
* @param rowIndex 行号 - row number
* @param colIndex 列号 - column number
* @param crossRowEleMetaLs 跨行列元数据 - row and column span metadata
* @return 当前行在某列需要占据单元格 - the number of cells to be occupied in a column on the current row
*/
private static int getCaptureCellSize(int rowIndex, int colIndex, List<CrossRangeCellMeta> crossRowEleMetaLs) {
int captureCellSize = 0;
for (CrossRangeCellMeta crossRangeCellMeta : crossRowEleMetaLs) {
if (crossRangeCellMeta.getFirstRow() < rowIndex && crossRangeCellMeta.getLastRow() >= rowIndex) {
if (crossRangeCellMeta.getFirstCol() <= colIndex && crossRangeCellMeta.getLastCol() >= colIndex) {
captureCellSize = crossRangeCellMeta.getLastCol() - colIndex + 1;
}
}
}
return captureCellSize;
}
/**
* 获得标题样式
* Get the title style
*
* @param workbook
* @return
*/
private static HSSFCellStyle getTitleStyle(HSSFWorkbook workbook) {
//short titlebackgroundcolor = IndexedColors.GREY_25_PERCENT.index;
short fontSize = 12;
String fontName = "宋体";
HSSFCellStyle style = workbook.createCellStyle();
style.setVerticalAlignment(VerticalAlignment.CENTER);
style.setAlignment(HorizontalAlignment.CENTER);
style.setBorderBottom(BorderStyle.THIN); //下边框 bottom border
style.setBorderLeft(BorderStyle.THIN);//左边框 left border
style.setBorderTop(BorderStyle.THIN);//上边框 top border
style.setBorderRight(BorderStyle.THIN);//右边框 right border
//style.setFillPattern(FillPatternType.SOLID_FOREGROUND);
//style.setFillForegroundColor(titlebackgroundcolor);// 背景色 background color
HSSFFont font = workbook.createFont();
font.setFontName(fontName);
font.setFontHeightInPoints(fontSize);
font.setBold(true);
style.setFont(font);
return style;
}
/**
* 获得内容样式
* Get the content style
*
* @param wb
* @return
*/
private static HSSFCellStyle getContentStyle(HSSFWorkbook wb) {
short fontSize = 12;
String fontName = "宋体";
HSSFCellStyle style = wb.createCellStyle();
style.setBorderBottom(BorderStyle.THIN); //下边框 bottom border
style.setBorderLeft(BorderStyle.THIN);//左边框 left border
style.setBorderTop(BorderStyle.THIN);//上边框 top border
style.setBorderRight(BorderStyle.THIN);//右边框 right border
HSSFFont font = wb.createFont();
font.setFontName(fontName);
font.setFontHeightInPoints(fontSize);
style.setFont(font);
style.setAlignment(HorizontalAlignment.CENTER);//水平居中 horizontal center
style.setVerticalAlignment(VerticalAlignment.CENTER);//垂直居中 vertical center
style.setWrapText(true);
return style;
}
}

82
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/DJLImageUtils.java Normal file
View File

@ -0,0 +1,82 @@
package top.aias.ocr.utils;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.DetectedObjects;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
/**
* DJL 图像工具类
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class DJLImageUtils {
public static void saveDJLImage(Image img, String name, String path) {
Path outputDir = Paths.get(path);
Path imagePath = outputDir.resolve(name);
try {
img.save(Files.newOutputStream(imagePath), "png");
} catch (IOException e) {
e.printStackTrace();
}
}
public static void saveBoundingBoxImage(
Image img, DetectedObjects detection, String name, String path) throws IOException {
// Make imageName copy with alpha channel because original imageName was jpg
img.drawBoundingBoxes(detection);
Path outputDir = Paths.get(path);
Files.createDirectories(outputDir);
Path imagePath = outputDir.resolve(name);
// OpenJDK can't save jpg with alpha channel
img.save(Files.newOutputStream(imagePath), "png");
}
public static void drawImageRect(BufferedImage image, int x, int y, int width, int height) {
Graphics2D g = (Graphics2D) image.getGraphics();
try {
g.setColor(new Color(246, 96, 0));
BasicStroke bStroke = new BasicStroke(4, BasicStroke.CAP_BUTT, BasicStroke.JOIN_MITER);
g.setStroke(bStroke);
g.drawRect(x, y, width, height);
} finally {
g.dispose();
}
}
public static void drawImageRect(
BufferedImage image, int x, int y, int width, int height, Color c) {
Graphics2D g = (Graphics2D) image.getGraphics();
try {
g.setColor(c);
BasicStroke bStroke = new BasicStroke(4, BasicStroke.CAP_BUTT, BasicStroke.JOIN_MITER);
g.setStroke(bStroke);
g.drawRect(x, y, width, height);
} finally {
g.dispose();
}
}
public static void drawImageText(BufferedImage image, String text) {
Graphics graphics = image.getGraphics();
int fontSize = 100;
Font font = new Font("楷体", Font.PLAIN, fontSize);
try {
graphics.setFont(font);
graphics.setColor(new Color(246, 96, 0));
int strWidth = graphics.getFontMetrics().stringWidth(text);
graphics.drawString(text, fontSize - (strWidth / 2), fontSize + 30);
} finally {
graphics.dispose();
}
}
}

139
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/FileUtils.java Normal file
View File

@ -0,0 +1,139 @@
package top.aias.ocr.utils;
import ai.djl.util.Utils;
import org.springframework.web.multipart.MultipartFile;
import java.io.*;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
/**
* 文件上传工具包
* File upload tool package
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class FileUtils {
/**
* @param file 文件 - file
* @param path 文件存放路径 - file storage path
* @param fileName 源文件名 - source file name
* @return
*/
public static boolean upload(MultipartFile file, String path, String fileName) {
// 生成新的文件名
// Generate a new file name
//String realPath = path + "/" + FileNameUtils.getFileName(fileName);
Path filePath = Paths.get(path + fileName);
File dest = filePath.toAbsolutePath().toFile();
//判断文件父目录是否存在
// Determine if the parent directory of the file exists
if (!dest.getParentFile().exists()) {
dest.getParentFile().mkdir();
}
try {
//保存文件
// Save the file
file.transferTo(dest);
return true;
} catch (IllegalStateException e) {
e.printStackTrace();
return false;
} catch (IOException e) {
e.printStackTrace();
return false;
}
}
/**
* 获取文件后缀
* Get file suffix
*
* @param fileName
* @return
*/
public static String getSuffix(String fileName) {
return fileName.substring(fileName.lastIndexOf("."));
}
/**
* 生成新的文件名
* Generate a new file name
* @param fileOriginName 源文件名 - source file name
* @return
*/
public static String getFileName(String fileOriginName) {
return UUIDUtils.getUUID() + getSuffix(fileOriginName);
}
/**
* 读取json文件
* Read json file
*
* @param path 文件路径信息 - file path information
* @param fileName 文件名 - file name
* @return
*/
public static String readFile(String path, String fileName) throws IOException {
StringBuilder json = new StringBuilder();
Path filePath = Paths.get(path + fileName);
List<String> lines = Utils.readLines(filePath, true);
lines.stream()
.filter(line -> (line != null && line != ""))
.forEach(
line -> {
json.append(line);
});
return json.toString();
}
/**
* 保存json文件
* Save json file
*
* @param path 文件路径信息 file path information
* @param fileName 文件名 file name
* @param json json信息 json information
* @return
*/
public static void saveFile(String path, String fileName, String json) throws IOException {
Path filePath = Paths.get(path + fileName);
try (PrintStream ps = new PrintStream(new FileOutputStream(filePath.toFile()))) {
ps.print(json);
}
}
/**
* 删除json文件
* Delete json file
*
* @param path 文件路径信息 file path information
* @param fileName 文件名 file name
* @return
*/
public static void removeFile(String path, String fileName) {
Path filePath = Paths.get(path + fileName);
filePath.toFile().delete();
}
/**
* Check & create file path
*
* @param fileRelativePath 文件路径信息 - file path information
* @return
*/
public static void checkAndCreatePath(String fileRelativePath) {
//Check & create file path
Path filePath = Paths.get(fileRelativePath).toAbsolutePath();
File file = filePath.toFile();
if (!file.exists() && !file.isDirectory()) {
file.mkdirs();
}
}
}

261
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/ImageUtils.java Normal file
View File

@ -0,0 +1,261 @@
package top.aias.ocr.utils;
import ai.djl.modality.cv.Image;
import ai.djl.modality.cv.output.DetectedObjects;
import ai.djl.ndarray.NDArray;
import org.opencv.core.Mat;
import org.opencv.core.Point;
import org.opencv.core.Scalar;
import org.opencv.imgproc.Imgproc;
import javax.imageio.ImageIO;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
/**
* 图像工具类
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class ImageUtils {
/**
* 保存BufferedImage图片
*
* @param bufferedImage
* @param name
* @param path
*/
public static void saveImage(BufferedImage bufferedImage, String name, String path) {
try {
Path outputDir = Paths.get(path);
Path imagePath = outputDir.resolve(name);
File output = imagePath.toFile();
ImageIO.write(bufferedImage, "png", output);
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* 保存DJL图片
*
* @param img
* @param name
* @param path
*/
public static void saveImage(Image img, String name, String path) {
Path outputDir = Paths.get(path);
Path imagePath = outputDir.resolve(name);
// OpenJDK 不能保存 jpg 图片的 alpha channel
try {
img.save(Files.newOutputStream(imagePath), "png");
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* 保存图片,含检测框
*
* @param img
* @param detection
* @param name
* @param path
* @throws IOException
*/
public static void saveBoundingBoxImage(
Image img, DetectedObjects detection, String name, String path) throws IOException {
// Make image copy with alpha channel because original image was jpg
img.drawBoundingBoxes(detection);
Path outputDir = Paths.get(path);
Files.createDirectories(outputDir);
Path imagePath = outputDir.resolve(name);
// OpenJDK can't save jpg with alpha channel
img.save(Files.newOutputStream(imagePath), "png");
}
/**
* 画矩形
*
* @param mat
* @param box
* @return
*/
public static void drawRect(Mat mat, NDArray box) {
float[] points = box.toFloatArray();
List<Point> list = new ArrayList<>();
for (int i = 0; i < 4; i++) {
Point point = new Point((int) points[2 * i], (int) points[2 * i + 1]);
list.add(point);
}
Imgproc.line(mat, list.get(0), list.get(1), new Scalar(0, 255, 0), 1);
Imgproc.line(mat, list.get(1), list.get(2), new Scalar(0, 255, 0), 1);
Imgproc.line(mat, list.get(2), list.get(3), new Scalar(0, 255, 0), 1);
Imgproc.line(mat, list.get(3), list.get(0), new Scalar(0, 255, 0), 1);
}
/**
* 画矩形
*
* @param mat
* @param box
* @return
*/
public static void drawRectWithText(Mat mat, NDArray box, String text) {
float[] points = box.toFloatArray();
List<Point> list = new ArrayList<>();
for (int i = 0; i < 4; i++) {
Point point = new Point((int) points[2 * i], (int) points[2 * i + 1]);
list.add(point);
}
Imgproc.line(mat, list.get(0), list.get(1), new Scalar(0, 255, 0), 1);
Imgproc.line(mat, list.get(1), list.get(2), new Scalar(0, 255, 0), 1);
Imgproc.line(mat, list.get(2), list.get(3), new Scalar(0, 255, 0), 1);
Imgproc.line(mat, list.get(3), list.get(0), new Scalar(0, 255, 0), 1);
// 中文乱码
Imgproc.putText(mat, text, list.get(0), Imgproc.FONT_HERSHEY_SCRIPT_SIMPLEX, 1.0, new Scalar(0, 255, 0), 1);
}
/**
* 画检测框(有倾斜角)
*
* @param image
* @param box
*/
public static void drawImageRect(BufferedImage image, NDArray box) {
float[] points = box.toFloatArray();
int[] xPoints = new int[5];
int[] yPoints = new int[5];
for (int i = 0; i < 4; i++) {
xPoints[i] = (int) points[2 * i];
yPoints[i] = (int) points[2 * i + 1];
}
xPoints[4] = xPoints[0];
yPoints[4] = yPoints[0];
// 将绘制图像转换为Graphics2D
Graphics2D g = (Graphics2D) image.getGraphics();
try {
g.setColor(new Color(0, 255, 0));
// 声明画笔属性 ：粗 细（单位像素）末端无修饰 折线处呈尖角
BasicStroke bStroke = new BasicStroke(4, BasicStroke.CAP_BUTT, BasicStroke.JOIN_MITER);
g.setStroke(bStroke);
g.drawPolyline(xPoints, yPoints, 5); // xPoints, yPoints, nPoints
} finally {
g.dispose();
}
}
/**
* 画检测框(有倾斜角)和文本
*
* @param image
* @param box
* @param text
*/
public static void drawImageRectWithText(BufferedImage image, NDArray box, String text) {
float[] points = box.toFloatArray();
int[] xPoints = new int[5];
int[] yPoints = new int[5];
for (int i = 0; i < 4; i++) {
xPoints[i] = (int) points[2 * i];
yPoints[i] = (int) points[2 * i + 1];
}
xPoints[4] = xPoints[0];
yPoints[4] = yPoints[0];
// 将绘制图像转换为Graphics2D
Graphics2D g = (Graphics2D) image.getGraphics();
try {
int fontSize = 32;
Font font = new Font("楷体", Font.PLAIN, fontSize);
g.setFont(font);
g.setColor(new Color(0, 0, 255));
// 声明画笔属性 ：粗 细（单位像素）末端无修饰 折线处呈尖角
BasicStroke bStroke = new BasicStroke(2, BasicStroke.CAP_BUTT, BasicStroke.JOIN_MITER);
g.setStroke(bStroke);
g.drawPolyline(xPoints, yPoints, 5); // xPoints, yPoints, nPoints
g.drawString(text, xPoints[0], yPoints[0]);
} finally {
g.dispose();
}
}
/**
* 画检测框
*
* @param image
* @param x
* @param y
* @param width
* @param height
*/
public static void drawImageRect(BufferedImage image, int x, int y, int width, int height) {
// 将绘制图像转换为Graphics2D
Graphics2D g = (Graphics2D) image.getGraphics();
try {
g.setColor(new Color(0, 255, 0));
// 声明画笔属性 ：粗 细（单位像素）末端无修饰 折线处呈尖角
BasicStroke bStroke = new BasicStroke(2, BasicStroke.CAP_BUTT, BasicStroke.JOIN_MITER);
g.setStroke(bStroke);
g.drawRect(x, y, width, height);
} finally {
g.dispose();
}
}
/**
* 显示文字
*
* @param image
* @param text
* @param x
* @param y
*/
public static void drawImageText(BufferedImage image, String text, int x, int y) {
Graphics graphics = image.getGraphics();
int fontSize = 32;
Font font = new Font("楷体", Font.PLAIN, fontSize);
try {
graphics.setFont(font);
graphics.setColor(new Color(0, 0, 255));
int strWidth = graphics.getFontMetrics().stringWidth(text);
graphics.drawString(text, x, y);
} finally {
graphics.dispose();
}
}
/**
* BufferedImage 转 base64
* @param image
* @param type - JPG、PNG、GIF、BMP 等
* @return
* @throws IOException
*/
public static String toBase64(BufferedImage image, String type) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(image, "jpg", baos);
byte[] bytes = baos.toByteArray();
return Base64.getEncoder().encodeToString(bytes);
}
}

339
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/NDArrayUtils.java Normal file
View File

@ -0,0 +1,339 @@
package top.aias.ocr.utils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDManager;
import ai.djl.ndarray.index.NDIndex;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.MatOfPoint;
import org.opencv.core.Point;
import java.util.ArrayList;
import java.util.List;
/**
* NDArray Utils 工具类
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class NDArrayUtils {
/**
* Sigmoid 激活函数
*
* @param input
* @return
*/
public static NDArray Sigmoid(NDArray input) {
// Sigmoid 函数，即f(x)=1/(1+e-x)
return input.neg().exp().add(1).pow(-1);
}
/**
* np.arctan2和np.arctan都是计算反正切值的NumPy函数，但它们的参数和返回值不同。一般来说，np.arctan2的参数为(y, x)，
* 返回值为[-π, π]之间的弧度值；而np.arctan的参数为x，返回值为[-π/2, π/2]之间的弧度值。两者之间的换算关系是：
* np.arctan(y/x) = np.arctan2(y, x)（当x>0时），
* 或 np.pi + np.arctan(y/x) = np.arctan2(y, x) （当x<0且y>=0时），
* 或 np.pi - np.arctan(y/x) = np.arctan2(y, x) （当x<0且y<0时）。
* @param y
* @param x
* @return
*/
public static NDArray arctan2(NDArray y, NDArray x) {
NDArray x_neg = x.lt(0).toType(DataType.INT32, false);
NDArray y_pos = y.gte(0).toType(DataType.INT32, false);
NDArray y_neg = y.lt(0).toType(DataType.INT32, false);
NDArray theta = y.div(x).atan();
// np.arctan(y/x) + np.pi = np.arctan2(y, x) （当x<0且y>=0时）
theta = theta.add(x_neg.mul(y_pos).mul((float) Math.PI));
// np.arctan(y/x) - np.pi = np.arctan2(y, x) （当x<0且y<0时）
theta = theta.add(x_neg.mul(y_neg).mul(-(float) Math.PI));
theta = theta.mul(180).div((float) Math.PI);
return theta;
}
/**
* 最大池化
*
* @param manager
* @param heat
* @param ksize
* @param stride
* @param padding
* @return
*/
public static NDArray maxPool(NDManager manager, NDArray heat, int ksize, int stride, int padding) {
int rows = (int) (heat.getShape().get(0));
int cols = (int) (heat.getShape().get(1));
// hmax = F.max_pool2d( heat, (ksize, ksize), stride=1, padding=(ksize-1)//2)
NDArray max_pool2d = manager.zeros(new Shape(rows + 2 * padding, cols + 2 * padding));
max_pool2d.set(new NDIndex(padding + ":" + (rows + padding) + ","+ padding + ":" + (cols + padding)), heat);
float[][] max_pool2d_arr = NDArrayUtils.floatNDArrayToArray(max_pool2d);
float[][] arr = new float[rows][cols];
for (int row = 0; row < rows; row++) {
for (int col = 0; col < cols; col++) {
float max = max_pool2d_arr[row][col];
for (int i = row; i < row + ksize; i++) {
for (int j = col; j < col + ksize; j++) {
if (max_pool2d_arr[i][j] > max) {
max = max_pool2d_arr[i][j];
}
}
}
arr[row][col] = max;
}
}
NDArray hmax = manager.create(arr).reshape(rows, cols);
return hmax;
}
/**
* mat To MatOfPoint
*
* @param mat
* @return
*/
public static MatOfPoint matToMatOfPoint(Mat mat) {
int rows = mat.rows();
MatOfPoint matOfPoint = new MatOfPoint();
List<Point> list = new ArrayList<>();
for (int i = 0; i < rows; i++) {
Point point = new Point((float) mat.get(i, 0)[0], (float) mat.get(i, 1)[0]);
list.add(point);
}
matOfPoint.fromList(list);
return matOfPoint;
}
/**
* int NDArray To int[][] Array
*
* @param ndArray
* @return
*/
public static int[][] intNDArrayToArray(NDArray ndArray) {
int rows = (int) (ndArray.getShape().get(0));
int cols = (int) (ndArray.getShape().get(1));
int[][] arr = new int[rows][cols];
int[] arrs = ndArray.toIntArray();
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
arr[i][j] = arrs[i * cols + j];
}
}
return arr;
}
/**
* float NDArray To float[][] Array
*
* @param ndArray
* @return
*/
public static float[][] floatNDArrayToArray(NDArray ndArray) {
int rows = (int) (ndArray.getShape().get(0));
int cols = (int) (ndArray.getShape().get(1));
float[][] arr = new float[rows][cols];
float[] arrs = ndArray.toFloatArray();
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
arr[i][j] = arrs[i * cols + j];
}
}
return arr;
}
/**
* mat To double[][] Array
*
* @param mat
* @return
*/
public static double[][] matToDoubleArray(Mat mat) {
int rows = mat.rows();
int cols = mat.cols();
double[][] doubles = new double[rows][cols];
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
doubles[i][j] = mat.get(i, j)[0];
}
}
return doubles;
}
/**
* mat To float[][] Array
*
* @param mat
* @return
*/
public static float[][] matToFloatArray(Mat mat) {
int rows = mat.rows();
int cols = mat.cols();
float[][] floats = new float[rows][cols];
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
floats[i][j] = (float) mat.get(i, j)[0];
}
}
return floats;
}
/**
* mat To byte[][] Array
*
* @param mat
* @return
*/
public static byte[][] matToUint8Array(Mat mat) {
int rows = mat.rows();
int cols = mat.cols();
byte[][] bytes = new byte[rows][cols];
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
bytes[i][j] = (byte) mat.get(i, j)[0];
}
}
return bytes;
}
/**
* float NDArray To Mat
*
* @param ndArray
* @param cvType
* @return
*/
public static Mat floatNDArrayToMat(NDArray ndArray, int cvType) {
int rows = (int) (ndArray.getShape().get(0));
int cols = (int) (ndArray.getShape().get(1));
Mat mat = new Mat(rows, cols, cvType);
float[] arrs = ndArray.toFloatArray();
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
mat.put(i, j, arrs[i * cols + j]);
}
}
return mat;
}
/**
* float NDArray To Mat
*
* @param ndArray
* @return
*/
public static Mat floatNDArrayToMat(NDArray ndArray) {
int rows = (int) (ndArray.getShape().get(0));
int cols = (int) (ndArray.getShape().get(1));
Mat mat = new Mat(rows, cols, CvType.CV_32F);
float[] arrs = ndArray.toFloatArray();
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
mat.put(i, j, arrs[i * cols + j]);
}
}
return mat;
}
/**
* uint8 NDArray To Mat
*
* @param ndArray
* @return
*/
public static Mat uint8NDArrayToMat(NDArray ndArray) {
int rows = (int) (ndArray.getShape().get(0));
int cols = (int) (ndArray.getShape().get(1));
Mat mat = new Mat(rows, cols, CvType.CV_8U);
byte[] arrs = ndArray.toByteArray();
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
mat.put(i, j, arrs[i * cols + j]);
}
}
return mat;
}
/**
* float[][] Array To Mat
* @param arr
* @return
*/
public static Mat floatArrayToMat(float[][] arr) {
int rows = arr.length;
int cols = arr[0].length;
Mat mat = new Mat(rows, cols, CvType.CV_32F);
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
mat.put(i, j, arr[i][j]);
}
}
return mat;
}
/**
* uint8Array To Mat
* @param arr
* @return
*/
public static Mat uint8ArrayToMat(byte[][] arr) {
int rows = arr.length;
int cols = arr[0].length;
Mat mat = new Mat(rows, cols, CvType.CV_8U);
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
mat.put(i, j, arr[i][j]);
}
}
return mat;
}
/**
* list 转 Mat
*
* @param points
* @return
*/
public static Mat toMat(List<ai.djl.modality.cv.output.Point> points) {
Mat mat = new Mat(points.size(), 2, CvType.CV_32F);
for (int i = 0; i < points.size(); i++) {
ai.djl.modality.cv.output.Point point = points.get(i);
mat.put(i, 0, (float) point.getX());
mat.put(i, 1, (float) point.getY());
}
return mat;
}
}

247
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/OpenCVUtils.java Normal file
View File

@ -0,0 +1,247 @@
package top.aias.ocr.utils;
import ai.djl.ndarray.NDArray;
import top.aias.ocr.bean.Point;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.MatOfPoint;
import org.opencv.core.Scalar;
import org.opencv.imgproc.Imgproc;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.util.ArrayList;
import java.util.List;
/**
* OpenCV Utils 工具类
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class OpenCVUtils {
/**
* Mat to BufferedImage
*
* @param mat
* @return
*/
public static BufferedImage mat2Image(Mat mat) {
int width = mat.width();
int height = mat.height();
byte[] data = new byte[width * height * (int) mat.elemSize()];
Imgproc.cvtColor(mat, mat, 4);
mat.get(0, 0, data);
BufferedImage ret = new BufferedImage(width, height, 5);
ret.getRaster().setDataElements(0, 0, width, height, data);
return ret;
}
/**
* BufferedImage to Mat
*
* @param img
* @return
*/
public static Mat image2Mat(BufferedImage img) {
int width = img.getWidth();
int height = img.getHeight();
byte[] data = ((DataBufferByte) img.getRaster().getDataBuffer()).getData();
Mat mat = new Mat(height, width, CvType.CV_8UC3);
mat.put(0, 0, data);
return mat;
}
/**
* list 转 Mat
*
* @param points
* @return
*/
public static Mat toMat(List<Point> points) {
Mat mat = new Mat(points.size(), 2, CvType.CV_32F);
for (int i = 0; i < points.size(); i++) {
Point point = points.get(i);
mat.put(i, 0, (float) point.getX());
mat.put(i, 1, (float) point.getY());
}
return mat;
}
/**
* 透视变换
*
* @param src
* @param dst
* @param warp_mat
* @return
*/
public static Mat warpPerspective(Mat src, Mat dst, Mat warp_mat) {
Mat dstClone = dst.clone();
// org.opencv.core.Mat mat = new org.opencv.core.Mat(dst.rows(), dst.cols(), CvType.CV_8UC3);
Imgproc.warpPerspective(src, dstClone, warp_mat, dst.size());
return dstClone;
}
/**
* 透视变换
*
* @param src
* @param srcPoints
* @param dstPoints
* @return
*/
public static Mat perspectiveTransform(Mat src, Mat srcPoints, Mat dstPoints) {
Mat dst = src.clone();
Mat warp_mat = Imgproc.getPerspectiveTransform(srcPoints, dstPoints);
Imgproc.warpPerspective(src, dst, warp_mat, dst.size());
warp_mat.release();
return dst;
}
/**
* 透视变换
*
* @param src
* @param dst
* @param srcPoints
* @param dstPoints
* @return
*/
public static Mat perspectiveTransform(Mat src, Mat dst, Mat srcPoints, Mat dstPoints) {
Mat dstClone = dst.clone();
Mat warp_mat = Imgproc.getPerspectiveTransform(srcPoints, dstPoints);
Imgproc.warpPerspective(src, dstClone, warp_mat, dst.size());
warp_mat.release();
return dstClone;
}
/**
* 图片裁剪
*
* @param points
* @return
*/
public static int[] imgCrop(float[] points) {
int[] wh = new int[2];
float[] lt = java.util.Arrays.copyOfRange(points, 0, 2);
float[] rt = java.util.Arrays.copyOfRange(points, 2, 4);
float[] rb = java.util.Arrays.copyOfRange(points, 4, 6);
float[] lb = java.util.Arrays.copyOfRange(points, 6, 8);
wh[0] = (int) Math.max(PointUtils.distance(lt, rt), PointUtils.distance(rb, lb));
wh[1] = (int) Math.max(PointUtils.distance(lt, lb), PointUtils.distance(rt, rb));
return wh;
}
/**
* 转正图片
*
* @param mat
* @param points
* @return
*/
public static Mat perspectiveTransform(Mat mat, float[] points) {
float[] lt = java.util.Arrays.copyOfRange(points, 0, 2);
float[] rt = java.util.Arrays.copyOfRange(points, 2, 4);
float[] rb = java.util.Arrays.copyOfRange(points, 4, 6);
float[] lb = java.util.Arrays.copyOfRange(points, 6, 8);
int img_crop_width = (int) Math.max(PointUtils.distance(lt, rt), PointUtils.distance(rb, lb));
int img_crop_height = (int) Math.max(PointUtils.distance(lt, lb), PointUtils.distance(rt, rb));
List<Point> srcPoints = new ArrayList<>();
srcPoints.add(new Point((int)lt[0], (int)lt[1]));
srcPoints.add(new Point((int)rt[0], (int)rt[1]));
srcPoints.add(new Point((int)rb[0], (int)rb[1]));
srcPoints.add(new Point((int)lb[0], (int)lb[1]));
List<Point> dstPoints = new ArrayList<>();
dstPoints.add(new Point(0, 0));
dstPoints.add(new Point(img_crop_width, 0));
dstPoints.add(new Point(img_crop_width, img_crop_height));
dstPoints.add(new Point(0, img_crop_height));
Mat srcPoint2f = toMat(srcPoints);
Mat dstPoint2f = toMat(dstPoints);
Mat cvMat = OpenCVUtils.perspectiveTransform(mat, srcPoint2f, dstPoint2f);
srcPoint2f.release();
dstPoint2f.release();
return cvMat;
}
/**
* 转正图片 - 废弃
*
* @param mat
* @param points
* @return
*/
public Mat perspectiveTransformOld(Mat mat, float[] points) {
List<org.opencv.core.Point> pointList = new ArrayList<>();
float[][] srcArr = new float[4][2];
float min_X = Float.MAX_VALUE;
float min_Y = Float.MAX_VALUE;
float max_X = -1;
float max_Y = -1;
for (int j = 0; j < 4; j++) {
org.opencv.core.Point pt = new org.opencv.core.Point(points[2 * j], points[2 * j + 1]);
pointList.add(pt);
srcArr[j][0] = points[2 * j];
srcArr[j][1] = points[2 * j + 1];
if (points[2 * j] > max_X) {
max_X = points[2 * j];
}
if (points[2 * j] < min_X) {
min_X = points[2 * j];
}
if (points[2 * j + 1] > max_Y) {
max_Y = points[2 * j + 1];
}
if (points[2 * j + 1] < min_Y) {
min_Y = points[2 * j + 1];
}
}
Mat src = NDArrayUtils.floatArrayToMat(srcArr);
float width = max_Y - min_Y;
float height = max_X - min_X;
float[][] dstArr = new float[4][2];
dstArr[0] = new float[]{0, 0};
dstArr[1] = new float[]{width - 1, 0};
dstArr[2] = new float[]{width - 1, height - 1};
dstArr[3] = new float[]{0, height - 1};
Mat dst = NDArrayUtils.floatArrayToMat(dstArr);
return OpenCVUtils.perspectiveTransform(mat, src, dst);
}
/**
* 画边框
*
* @param mat
* @param squares
* @param topK
*/
public static void drawSquares(Mat mat, NDArray squares, int topK) {
for (int i = 0; i < topK; i++) {
float[] points = squares.get(i).toFloatArray();
List<MatOfPoint> matOfPoints = new ArrayList<>();
MatOfPoint matOfPoint = new MatOfPoint();
matOfPoints.add(matOfPoint);
List<org.opencv.core.Point> pointList = new ArrayList<>();
for (int j = 0; j < 4; j++) {
org.opencv.core.Point pt = new org.opencv.core.Point(points[2 * j], points[2 * j + 1]);
pointList.add(pt);
Imgproc.circle(mat, pt, 10, new Scalar(0, 255, 255), -1);
Imgproc.putText(mat, "" + j, pt, Imgproc.FONT_HERSHEY_SCRIPT_SIMPLEX, 1.0, new Scalar(0, 255, 0), 1);
}
matOfPoint.fromList(pointList);
Imgproc.polylines(mat, matOfPoints, true, new Scalar(200, 200, 0), 5);
}
}
}

343
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/PointUtils.java Normal file
View File

@ -0,0 +1,343 @@
package top.aias.ocr.utils;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDArrays;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.NDManager;
import ai.djl.ndarray.types.Shape;
import top.aias.ocr.bean.Point;
import java.util.List;
/**
* 点工具类
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class PointUtils {
/**
* 计算两点距离
* @param point1
* @param point2
* @return
*/
public static float distance(float[] point1, float[] point2) {
float disX = point1[0] - point2[0];
float disY = point1[1] - point2[1];
float dis = (float) Math.sqrt(disX * disX + disY * disY);
return dis;
}
/**
* 计算两点距离
* @param point1
* @param point2
* @return
*/
public static float distance(Point point1, Point point2) {
double disX = point1.getX() - point2.getX();
double disY = point1.getY() - point2.getY();
float dis = (float) Math.sqrt(disX * disX + disY * disY);
return dis;
}
/**
* sort the points based on their x-coordinates
* 顺时针排序
*
* @param pts
* @return
*/
private static NDArray order_points_clockwise(NDArray pts) {
NDList list = new NDList();
long[] indexes = pts.get(":, 0").argSort().toLongArray();
// grab the left-most and right-most points from the sorted
// x-roodinate points
Shape s1 = pts.getShape();
NDArray leftMost1 = pts.get(indexes[0] + ",:");
NDArray leftMost2 = pts.get(indexes[1] + ",:");
NDArray leftMost = leftMost1.concat(leftMost2).reshape(2, 2);
NDArray rightMost1 = pts.get(indexes[2] + ",:");
NDArray rightMost2 = pts.get(indexes[3] + ",:");
NDArray rightMost = rightMost1.concat(rightMost2).reshape(2, 2);
// now, sort the left-most coordinates according to their
// y-coordinates so we can grab the top-left and bottom-left
// points, respectively
indexes = leftMost.get(":, 1").argSort().toLongArray();
NDArray lt = leftMost.get(indexes[0] + ",:");
NDArray lb = leftMost.get(indexes[1] + ",:");
indexes = rightMost.get(":, 1").argSort().toLongArray();
NDArray rt = rightMost.get(indexes[0] + ",:");
NDArray rb = rightMost.get(indexes[1] + ",:");
list.add(lt);
list.add(rt);
list.add(rb);
list.add(lb);
NDArray rect = NDArrays.concat(list).reshape(4, 2);
return rect;
}
/**
* 计算四边形的面积
* 根据海伦公式（Heron's formula）计算面积
*
* @param arr
* @return
*/
public static double getQuadArea(NDManager manager, double[][] arr) {
NDArray ndArray = manager.create(arr).reshape(4, 2);
ndArray = order_points_clockwise(ndArray);
double[] array = ndArray.toDoubleArray();
double x1 = array[0];
double y1 = array[1];
double x2 = array[2];
double y2 = array[3];
double x3 = array[4];
double y3 = array[5];
double x4 = array[6];
double y4 = array[7];
double totalArea;
if (isInTriangle(x2, y2, x3, y3, x4, y4, x1, y1)) { // 判断点 (x1, y1) 是否在三角形 (x2,y2)，(x3,y3)，(x4,y4) 内
double area1 = getTriangleArea(x2, y2, x3, y3, x1, y1);
double area2 = getTriangleArea(x2, y2, x4, y4, x1, y1);
double area3 = getTriangleArea(x3, y3, x4, y4, x1, y1);
totalArea = area1 + area2 + area3;
} else if (isInTriangle(x1, y1, x3, y3, x4, y4, x2, y2)) {// 判断点 (x2, y2) 是否在三角形 (x1,y1)，(x3,y3)，(x4,y4) 内
double area1 = getTriangleArea(x1, y1, x3, y3, x2, y2);
double area2 = getTriangleArea(x1, y1, x4, y4, x2, y2);
double area3 = getTriangleArea(x3, y3, x4, y4, x2, y2);
totalArea = area1 + area2 + area3;
} else if (isInTriangle(x1, y1, x2, y2, x4, y4, x3, y3)) {// 判断点 (x3, y3) 是否在三角形 (x1,y1)，(x2,y2)，(x4,y4) 内
double area1 = getTriangleArea(x1, y1, x2, y2, x3, y3);
double area2 = getTriangleArea(x1, y1, x4, y4, x3, y3);
double area3 = getTriangleArea(x2, y2, x4, y4, x3, y3);
totalArea = area1 + area2 + area3;
} else if (isInTriangle(x1, y1, x2, y2, x3, y3, x4, y4)) {// 判断点 (x4, y4) 是否在三角形 (x1,y1)，(x2,y2)，(x3,y3) 内
double area1 = getTriangleArea(x1, y1, x2, y2, x4, y4);
double area2 = getTriangleArea(x1, y1, x3, y3, x4, y4);
double area3 = getTriangleArea(x2, y2, x3, y3, x4, y4);
totalArea = area1 + area2 + area3;
} else {
double area1 = getTriangleArea(x1, y1, x2, y2, x3, y3);
double area2 = getTriangleArea(x1, y1, x3, y3, x4, y4);
totalArea = area1 + area2;
}
return totalArea;
}
/**
* 判断点 (px, py) 是否在三角形 (x1,y1)，(x2,y2)，(x3,y3) 内
*
* @param x1
* @param y1
* @param x2
* @param y2
* @param x3
* @param y3
* @param px
* @param py
* @return
*/
public static boolean isInTriangle(double x1, double y1, double x2, double y2, double x3, double y3, double px, double py) {
if(!isTriangle(x1, y1, x2, y2, px, py))
return false;
double area1 = getTriangleArea(x1, y1, x2, y2, px, py);
if(!isTriangle(x1, y1, x3, y3, px, py))
return false;
double area2 = getTriangleArea(x1, y1, x3, y3, px, py);
if(!isTriangle(x2, y2, x3, y3, px, py))
return false;
double area3 = getTriangleArea(x2, y2, x3, y3, px, py);
if(!isTriangle(x1, y1, x2, y2, x3, y3))
return false;
double totalArea = getTriangleArea(x1, y1, x2, y2, x3, y3);
double delta = Math.abs(totalArea - (area1 + area2 + area3));
if (delta < 1)
return true;
else
return false;
}
/**
* 给定3个点坐标(x1,y1)，(x2,y2)，(x3,y3)，给出判断是否能组成三角形
* @param x1
* @param y1
* @param x2
* @param y2
* @param x3
* @param y3
* @return
*/
public static boolean isTriangle(double x1, double y1, double x2, double y2, double x3, double y3) {
double a = Math.sqrt(Math.pow(x1-x2, 2) + Math.pow(y1-y2, 2));
double b = Math.sqrt(Math.pow(x1-x3, 2) + Math.pow(y1-y3, 2));
double c = Math.sqrt(Math.pow(x2-x3, 2) + Math.pow(y2-y3, 2));
return a + b > c && b + c > a && a + c > b;
}
/**
* 计算三角形的面积
* 根据海伦公式（Heron's formula）计算三角形面积
*
* @param x1
* @param y1
* @param x2
* @param y2
* @param x3
* @param y3
* @return
*/
public static double getTriangleArea(double x1, double y1, double x2, double y2, double x3, double y3) {
double a = Math.sqrt(Math.pow(x2 - x1, 2) + Math.pow(y2 - y1, 2));
double b = Math.sqrt(Math.pow(x3 - x2, 2) + Math.pow(y3 - y2, 2));
double c = Math.sqrt(Math.pow(x1 - x3, 2) + Math.pow(y1 - y3, 2));
double p = (a + b + c) / 2;
double area = Math.sqrt(p * (p - a) * (p - b) * (p - c));
return area;
}
public static ai.djl.modality.cv.output.Point getCenterPoint(List<Point> points) {
double sumX = 0;
double sumY = 0;
for (Point point : points) {
sumX = sumX + point.getX();
sumY = sumY + point.getY();
}
ai.djl.modality.cv.output.Point centerPoint = new ai.djl.modality.cv.output.Point(sumX / 4, sumY / 4);
return centerPoint;
}
/**
* 点坐标变换
*
* @param manager
* @param mat
* @param point
* @return
*/
public static Point transformPoint(NDManager manager, org.opencv.core.Mat mat, Point point) {
double[][] pointsArray = new double[3][3];
for (int i = 0; i < 3; i++) {
for (int j = 0; j < 3; j++) {
pointsArray[i][j] = mat.get(i, j)[0];
}
}
NDArray ndPoints = manager.create(pointsArray);
double[] vector = new double[3];
vector[0] = point.getX();
vector[1] = point.getY();
vector[2] = 1f;
NDArray vPoints = manager.create(vector);
vPoints = vPoints.reshape(3, 1);
NDArray result = ndPoints.matMul(vPoints);
double[] dArray = result.toDoubleArray();
if (dArray[2] != 0) {
point.setX((int) (dArray[0] / dArray[2]));
point.setY((int) (dArray[1] / dArray[2]));
}
return point;
}
/**
* 坐标变换
*
* @param manager
* @param mat
* @param points
* @return
*/
public static List<Point> transformPoints(NDManager manager, org.opencv.core.Mat mat, List<Point> points) {
int cols = mat.cols();
int rows = mat.rows();
double[][] pointsArray = new double[rows][cols];
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
pointsArray[i][j] = mat.get(i, j)[0];
}
}
NDArray ndPoints = manager.create(pointsArray);
double[] vector = new double[3];
for (int i = 0; i < points.size(); i++) {
vector[0] = points.get(i).getX();
vector[1] = points.get(i).getY();
vector[2] = 1f;
NDArray vPoints = manager.create(vector);
vPoints = vPoints.reshape(3, 1);
NDArray result = ndPoints.matMul(vPoints);
double[] dArray = result.toDoubleArray();
if (dArray.length > 2) {
if (dArray[2] != 0) {
points.get(i).setX((int) (dArray[0] / dArray[2]));
points.get(i).setY((int) (dArray[1] / dArray[2]));
}
} else {
points.get(i).setX((int) (dArray[0]));
points.get(i).setY((int) (dArray[1]));
}
}
return points;
}
/**
* 获取左上角和有下角的坐标
* Get (x1,y1,x2,y2) coordinations
*
* @param points
* @return
*/
public static int[] rectXYXY(List<Point> points) {
int left = points.get(0).getX();
int top = points.get(0).getY();
int right = points.get(2).getX();
int bottom = points.get(2).getY();
return new int[]{left, top, right, bottom};
}
/**
* 获取左上角的坐标，宽度，高度
* Get (x1,y1,w,h) coordinations
*
* @param points
* @return
*/
public static int[] rectXYWH(List<Point> points) {
int minX = Integer.MAX_VALUE;
int minY = Integer.MAX_VALUE;
int maxX = Integer.MIN_VALUE;
int maxY = Integer.MIN_VALUE;
for (Point point : points) {
int x = point.getX();
int y = point.getY();
if (x < minX)
minX = x;
if (x > maxX)
maxX = x;
if (y < minY)
minY = y;
if (y > maxY)
maxY = y;
}
int w = maxX - minX;
int h = maxY - minY;
return new int[]{minX, minY, w, h};
}
}

19
6_web_app/ocr_web_app/ocr_backend/src/main/java/top/aias/ocr/utils/UUIDUtils.java Normal file
View File

@ -0,0 +1,19 @@
package top.aias.ocr.utils;
import java.util.UUID;
/**
* 生成文件名
* Generate file name
*
* @author Calvin
* @mail 179209347@qq.com
* @website www.aias.top
*/
public class UUIDUtils {
public static String getUUID() {
return UUID.randomUUID().toString().replace("-", "");
}
}

11
6_web_app/ocr_web_app/ocr_backend/src/main/main.iml Normal file
View File

@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/java" isTestSource="false" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>

27
6_web_app/ocr_web_app/ocr_backend/src/main/resources/application-linux.yml Normal file
View File

@ -0,0 +1,27 @@
# Server Port
server:
port: 8089
tomcat:
uri-encoding: UTF-8
baseUri: http://127.0.0.1:${server.port}
model:
# 设置为 CPU 核心数 (Core Number)
poolSize: 4
table:
# 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测
layout: /home/models/ocr/picodet_lcnet_x1_0_fgd_layout_table_infer_onnx.zip
# 英文表格识别
rec: /home/models/ocr/en_ppstructure_mobile_v2.0_SLANet_infer.zip
# 中文表格识别
# /home/models/ocr/ch_ppstructure_mobile_v2.0_SLANet_infer.zip
ocrv4:
# server detection model URI
det: /home/models/ocr/ch_PP-OCRv4_det_infer.zip
# server recognition model URI
rec: /home/models/ocr/ch_PP-OCRv4_rec_infer.zip
mlsd:
# mlsd model URI
model: /home/models/ocr/mlsd_traced_model_onnx.zip

26
6_web_app/ocr_web_app/ocr_backend/src/main/resources/application-mac.yml Normal file
View File

@ -0,0 +1,26 @@
# Server Port
server:
port: 8089
tomcat:
uri-encoding: UTF-8
baseUri: http://127.0.0.1:${server.port}
model:
# 设置为 CPU 核心数 (Core Number)
poolSize: 4
table:
# 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测
layout: /Users/calvin/ocr/ocr_backend/models/picodet_lcnet_x1_0_fgd_layout_table_infer_onnx.zip
# 英文表格识别
rec: /Users/calvin/ocr/ocr_backend/models/en_ppstructure_mobile_v2.0_SLANet_infer.zip
# 中文表格识别
# rec: /Users/calvin/ocr/ocr_backend/models/ch_ppstructure_mobile_v2.0_SLANet_infer.zip
ocrv4:
# server detection model URI
det: /Users/calvin/ocr/ocr_backend/models/ch_PP-OCRv4_det_infer.zip
# server recognition model URI
rec: /Users/calvin/ocr/ocr_backend/models/ch_PP-OCRv4_rec_infer.zip
mlsd:
# mlsd model URI
model: /Users/calvin/ocr/ocr_backend/models/mlsd_traced_model_onnx.zip

26
6_web_app/ocr_web_app/ocr_backend/src/main/resources/application-online.yml Normal file
View File

@ -0,0 +1,26 @@
# Server Port
server:
port: 8089
tomcat:
uri-encoding: UTF-8
baseUri: http://127.0.0.1:${server.port}
model:
# 设置为 CPU 核心数 (Core Number)
poolSize: 4
table:
# 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测
layout: https://aias-home.oss-cn-beijing.aliyuncs.com/models/ocr/picodet_lcnet_x1_0_fgd_layout_table_infer_onnx.zip
# 英文表格识别
rec: https://aias-home.oss-cn-beijing.aliyuncs.com/models/ocr/en_ppstructure_mobile_v2.0_SLANet_infer.zip
# 中文表格识别
# rec: https://aias-home.oss-cn-beijing.aliyuncs.com/models/ocr/ch_ppstructure_mobile_v2.0_SLANet_infer.zip
ocrv4:
# server detection model URI
det: https://aias-home.oss-cn-beijing.aliyuncs.com/models/ocr/ch_PP-OCRv4_det_infer.zip
# server recognition model URI
rec: https://aias-home.oss-cn-beijing.aliyuncs.com/models/ocr/ch_PP-OCRv4_rec_infer.zip
mlsd:
# mlsd model URI
model: https://aias-home.oss-cn-beijing.aliyuncs.com/models/ocr/mlsd_traced_model_onnx.zip

27
6_web_app/ocr_web_app/ocr_backend/src/main/resources/application-win.yml Normal file
View File

@ -0,0 +1,27 @@
# Server Port
server:
port: 8089
tomcat:
uri-encoding: UTF-8
baseUri: http://127.0.0.1:${server.port}
model:
# 设置为 CPU 核心数 (Core Number)
poolSize: 4
table:
# 表格数据集训练的版面分析模型，支持中英文文档表格区域的检测
layout: D:\\ai_projects\\AIAS\\6_web_app\\ocr_web_app\\ocr_backend\\models\\picodet_lcnet_x1_0_fgd_layout_table_infer_onnx.zip
# 英文表格识别
rec: D:\\ai_projects\\AIAS\\6_web_app\\ocr_web_app\\ocr_backend\\models\\en_ppstructure_mobile_v2.0_SLANet_infer.zip
# 中文表格识别
# D:\\ai_projects\\AIAS\\6_web_app\\ocr_web_app\\ocr_backend\\models\\ch_ppstructure_mobile_v2.0_SLANet_infer.zip
ocrv4:
# server detection model URI
det: D:\\ai_projects\\AIAS\\6_web_app\\ocr_web_app\\ocr_backend\\models\\ch_PP-OCRv4_det_infer.zip
# server recognition model URI
rec: D:\\ai_projects\\AIAS\\6_web_app\\ocr_web_app\\ocr_backend\\models\\ch_PP-OCRv4_rec_infer.zip
mlsd:
# mlsd model URI
model: D:\\ai_projects\\AIAS\\6_web_app\\ocr_web_app\\ocr_backend\\models\\mlsd_traced_model_onnx.zip

36
6_web_app/ocr_web_app/ocr_backend/src/main/resources/application.yml Normal file
View File

@ -0,0 +1,36 @@
spring:
profiles:
active: win
servlet:
multipart:
enabled: true
max-file-size: 30MB
max-request-size: 30MB
http:
encoding:
charset: utf-8
enabled: true
force: true
messages:
encoding: UTF-8
# Swagger-ui
swagger:
enabled: true
# File path
file:
mac:
path: file/
linux:
path: file/
windows:
path: D:\\
# File max size - MB
maxSize: 100
# Verify image transformation result
image:
debug: true

17
6_web_app/ocr_web_app/ocr_backend/src/main/resources/log4j2.xml Normal file
View File

@ -0,0 +1,17 @@
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<Console name="console" target="SYSTEM_OUT">
<PatternLayout
pattern="[%-5level] - %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="info" additivity="false">
<AppenderRef ref="console"/>
</Root>
<Logger name="me.calvin" level="${sys:me.calvin.logging.level:-info}" additivity="false">
<AppenderRef ref="console"/>
</Logger>
</Loggers>
</Configuration>

12
6_web_app/ocr_web_app/ocr_backend/src/test/test.iml Normal file
View File

@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="JAVA_MODULE" version="4">
<component name="NewModuleRootManager" inherit-compiler-output="true">
<exclude-output />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/java" isTestSource="true" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="module" module-name="main" />
</component>
</module>

14
6_web_app/ocr_web_app/ocr_ui/.editorconfig Normal file
View File

@ -0,0 +1,14 @@
# http://editorconfig.org
root = true
[*]
charset = utf-8
indent_style = space
indent_size = 2
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
[*.md]
insert_final_newline = false
trim_trailing_whitespace = false

6
6_web_app/ocr_web_app/ocr_ui/.env.development Normal file
View File

@ -0,0 +1,6 @@
# just a flag
ENV = 'development'
# base api 127.0.0.1
VUE_APP_BASE_API = 'http://127.0.0.1:8089'

6
6_web_app/ocr_web_app/ocr_ui/.env.production Normal file
View File

@ -0,0 +1,6 @@
# just a flag
ENV = 'production'
# base api 127.0.0.1
VUE_APP_BASE_API = 'http://127.0.0.1:8089'

8
6_web_app/ocr_web_app/ocr_ui/.env.staging Normal file
View File

@ -0,0 +1,8 @@
NODE_ENV = production
# just a flag
ENV = 'staging'
# base api
VUE_APP_BASE_API = '/stage-api'

4
6_web_app/ocr_web_app/ocr_ui/.eslintignore Normal file
View File

@ -0,0 +1,4 @@
build/*.js
src/assets
public
dist

198
6_web_app/ocr_web_app/ocr_ui/.eslintrc.js Normal file
View File

@ -0,0 +1,198 @@
module.exports = {
root: true,
parserOptions: {
parser: 'babel-eslint',
sourceType: 'module'
},
env: {
browser: true,
node: true,
es6: true,
},
extends: ['plugin:vue/recommended', 'eslint:recommended'],
// add your custom rules here
//it is base on https://github.com/vuejs/eslint-config-vue
rules: {
"vue/max-attributes-per-line": [2, {
"singleline": 10,
"multiline": {
"max": 1,
"allowFirstLine": false
}
}],
"vue/singleline-html-element-content-newline": "off",
"vue/multiline-html-element-content-newline":"off",
"vue/name-property-casing": ["error", "PascalCase"],
"vue/no-v-html": "off",
'accessor-pairs': 2,
'arrow-spacing': [2, {
'before': true,
'after': true
}],
'block-spacing': [2, 'always'],
'brace-style': [2, '1tbs', {
'allowSingleLine': true
}],
'camelcase': [0, {
'properties': 'always'
}],
'comma-dangle': [2, 'never'],
'comma-spacing': [2, {
'before': false,
'after': true
}],
'comma-style': [2, 'last'],
'constructor-super': 2,
'curly': [2, 'multi-line'],
'dot-location': [2, 'property'],
'eol-last': 2,
'eqeqeq': ["error", "always", {"null": "ignore"}],
'generator-star-spacing': [2, {
'before': true,
'after': true
}],
'handle-callback-err': [2, '^(err|error)$'],
'indent': [2, 2, {
'SwitchCase': 1
}],
'jsx-quotes': [2, 'prefer-single'],
'key-spacing': [2, {
'beforeColon': false,
'afterColon': true
}],
'keyword-spacing': [2, {
'before': true,
'after': true
}],
'new-cap': [2, {
'newIsCap': true,
'capIsNew': false
}],
'new-parens': 2,
'no-array-constructor': 2,
'no-caller': 2,
'no-console': 'off',
'no-class-assign': 2,
'no-cond-assign': 2,
'no-const-assign': 2,
'no-control-regex': 0,
'no-delete-var': 2,
'no-dupe-args': 2,
'no-dupe-class-members': 2,
'no-dupe-keys': 2,
'no-duplicate-case': 2,
'no-empty-character-class': 2,
'no-empty-pattern': 2,
'no-eval': 2,
'no-ex-assign': 2,
'no-extend-native': 2,
'no-extra-bind': 2,
'no-extra-boolean-cast': 2,
'no-extra-parens': [2, 'functions'],
'no-fallthrough': 2,
'no-floating-decimal': 2,
'no-func-assign': 2,
'no-implied-eval': 2,
'no-inner-declarations': [2, 'functions'],
'no-invalid-regexp': 2,
'no-irregular-whitespace': 2,
'no-iterator': 2,
'no-label-var': 2,
'no-labels': [2, {
'allowLoop': false,
'allowSwitch': false
}],
'no-lone-blocks': 2,
'no-mixed-spaces-and-tabs': 2,
'no-multi-spaces': 2,
'no-multi-str': 2,
'no-multiple-empty-lines': [2, {
'max': 1
}],
'no-native-reassign': 2,
'no-negated-in-lhs': 2,
'no-new-object': 2,
'no-new-require': 2,
'no-new-symbol': 2,
'no-new-wrappers': 2,
'no-obj-calls': 2,
'no-octal': 2,
'no-octal-escape': 2,
'no-path-concat': 2,
'no-proto': 2,
'no-redeclare': 2,
'no-regex-spaces': 2,
'no-return-assign': [2, 'except-parens'],
'no-self-assign': 2,
'no-self-compare': 2,
'no-sequences': 2,
'no-shadow-restricted-names': 2,
'no-spaced-func': 2,
'no-sparse-arrays': 2,
'no-this-before-super': 2,
'no-throw-literal': 2,
'no-trailing-spaces': 2,
'no-undef': 2,
'no-undef-init': 2,
'no-unexpected-multiline': 2,
'no-unmodified-loop-condition': 2,
'no-unneeded-ternary': [2, {
'defaultAssignment': false
}],
'no-unreachable': 2,
'no-unsafe-finally': 2,
'no-unused-vars': [2, {
'vars': 'all',
'args': 'none'
}],
'no-useless-call': 2,
'no-useless-computed-key': 2,
'no-useless-constructor': 2,
'no-useless-escape': 0,
'no-whitespace-before-property': 2,
'no-with': 2,
'one-var': [2, {
'initialized': 'never'
}],
'operator-linebreak': [2, 'after', {
'overrides': {
'?': 'before',
':': 'before'
}
}],
'padded-blocks': [2, 'never'],
'quotes': [2, 'single', {
'avoidEscape': true,
'allowTemplateLiterals': true
}],
'semi': [2, 'never'],
'semi-spacing': [2, {
'before': false,
'after': true
}],
'space-before-blocks': [2, 'always'],
'space-before-function-paren': [2, 'never'],
'space-in-parens': [2, 'never'],
'space-infix-ops': 2,
'space-unary-ops': [2, {
'words': true,
'nonwords': false
}],
'spaced-comment': [2, 'always', {
'markers': ['global', 'globals', 'eslint', 'eslint-disable', '*package', '!', ',']
}],
'template-curly-spacing': [2, 'never'],
'use-isnan': 2,
'valid-typeof': 2,
'wrap-iife': [2, 'any'],
'yield-star-spacing': [2, 'both'],
'yoda': [2, 'never'],
'prefer-const': 2,
'no-debugger': process.env.NODE_ENV === 'production' ? 2 : 0,
'object-curly-spacing': [2, 'always', {
objectsInObjects: false
}],
'array-bracket-spacing': [2, 'never']
}
}

5
6_web_app/ocr_web_app/ocr_ui/.travis.yml Normal file
View File

@ -0,0 +1,5 @@
language: node_js
node_js: 10
script: npm run test
notifications:
email: false

14
6_web_app/ocr_web_app/ocr_ui/babel.config.js Normal file
View File

@ -0,0 +1,14 @@
module.exports = {
presets: [
// https://github.com/vuejs/vue-cli/tree/master/packages/@vue/babel-preset-app
'@vue/cli-plugin-babel/preset'
],
'env': {
'development': {
// babel-plugin-dynamic-import-node plugin only does one thing by converting all import() to require().
// This plugin can significantly increase the speed of hot updates, when you have a large number of pages.
// https://panjiachen.github.io/vue-element-admin-site/guide/advanced/lazy-loading.html
'plugins': ['dynamic-import-node']
}
}
}

35
6_web_app/ocr_web_app/ocr_ui/build/index.js Normal file
View File

@ -0,0 +1,35 @@
const { run } = require('runjs')
const chalk = require('chalk')
const config = require('../vue.config.js')
const rawArgv = process.argv.slice(2)
const args = rawArgv.join(' ')
if (process.env.npm_config_preview || rawArgv.includes('--preview')) {
const report = rawArgv.includes('--report')
run(`vue-cli-service build ${args}`)
const port = 9526
const publicPath = config.publicPath
var connect = require('connect')
var serveStatic = require('serve-static')
const app = connect()
app.use(
publicPath,
serveStatic('./dist', {
index: ['index.html', '/']
})
)
app.listen(port, function () {
console.log(chalk.green(`> Preview at http://localhost:${port}${publicPath}`))
if (report) {
console.log(chalk.green(`> Report at http://localhost:${port}${publicPath}report.html`))
}
})
} else {
run(`vue-cli-service build ${args}`)
}

24
6_web_app/ocr_web_app/ocr_ui/jest.config.js Normal file
View File

@ -0,0 +1,24 @@
module.exports = {
moduleFileExtensions: ['js', 'jsx', 'json', 'vue'],
transform: {
'^.+\\.vue$': 'vue-jest',
'.+\\.(css|styl|less|sass|scss|svg|png|jpg|ttf|woff|woff2)$':
'jest-transform-stub',
'^.+\\.jsx?$': 'babel-jest'
},
moduleNameMapper: {
'^@/(.*)$': '<rootDir>/src/$1'
},
snapshotSerializers: ['jest-serializer-vue'],
testMatch: [
'**/tests/unit/**/*.spec.(js|jsx|ts|tsx)|**/__tests__/*.(js|jsx|ts|tsx)'
],
collectCoverageFrom: ['src/utils/**/*.{js,vue}', '!src/utils/auth.js', '!src/utils/request.js', 'src/components/**/*.{js,vue}'],
coverageDirectory: '<rootDir>/tests/unit/coverage',
// 'collectCoverage': true,
'coverageReporters': [
'lcov',
'text-summary'
],
testURL: 'http://localhost/'
}

9
6_web_app/ocr_web_app/ocr_ui/jsconfig.json Normal file
View File

@ -0,0 +1,9 @@
{
"compilerOptions": {
"baseUrl": "./",
"paths": {
"@/*": ["src/*"]
}
},
"exclude": ["node_modules", "dist"]
}

85
6_web_app/ocr_web_app/ocr_ui/nginx.conf Normal file
View File

@ -0,0 +1,85 @@
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}

65
6_web_app/ocr_web_app/ocr_ui/package.json Normal file
View File

@ -0,0 +1,65 @@
{
"name": "ocr-ui",
"version": "1.0.0",
"description": "OCR UI",
"author": "Calvin <179209347@qq.com>",
"scripts": {
"dev": "vue-cli-service serve",
"build:prod": "vue-cli-service build",
"build:stage": "vue-cli-service build --mode staging",
"preview": "node build/index.js --preview",
"lint": "eslint --ext .js,.vue src",
"test:unit": "jest --clearCache && vue-cli-service test:unit",
"test:ci": "npm run lint && npm run test:unit"
},
"dependencies": {
"axios": "0.18.1",
"core-js": "3.6.5",
"easy-circular-progress": "1.0.4",
"echarts": "^4.2.1",
"element-ui": "2.13.2",
"js-base64": "^2.6.4",
"js-cookie": "2.2.0",
"normalize.css": "7.0.0",
"nprogress": "0.2.0",
"path-to-regexp": "2.4.0",
"vertx3-eventbus-client": "^3.9.4",
"vue": "2.6.10",
"vue-count-to": "^1.0.13",
"vue-json-viewer": "^2.2.18",
"vue-router": "3.0.6",
"vuex": "3.1.0"
},
"devDependencies": {
"@vue/cli-plugin-babel": "4.4.4",
"@vue/cli-plugin-eslint": "4.4.4",
"@vue/cli-plugin-unit-jest": "4.4.4",
"@vue/cli-service": "4.4.4",
"@vue/test-utils": "1.0.0-beta.29",
"autoprefixer": "9.5.1",
"babel-eslint": "10.1.0",
"babel-jest": "23.6.0",
"babel-plugin-dynamic-import-node": "2.3.3",
"chalk": "2.4.2",
"connect": "3.6.6",
"eslint": "6.7.2",
"eslint-plugin-vue": "6.2.2",
"html-webpack-plugin": "3.2.0",
"mockjs": "1.0.1-beta3",
"runjs": "4.3.2",
"sass": "1.26.8",
"sass-loader": "8.0.2",
"script-ext-html-webpack-plugin": "2.1.3",
"serve-static": "1.13.2",
"vue-template-compiler": "2.6.10"
},
"browserslist": [
"> 1%",
"last 2 versions"
],
"engines": {
"node": ">=8.9",
"npm": ">= 3.0.0"
},
"license": "apache2.0"
}

8
6_web_app/ocr_web_app/ocr_ui/postcss.config.js Normal file
View File

@ -0,0 +1,8 @@
// https://github.com/michael-ciniawsky/postcss-load-config
module.exports = {
'plugins': {
// to edit target browsers: use "browserslist" field in package.json
'autoprefixer': {}
}
}

BIN
6_web_app/ocr_web_app/ocr_ui/public/favicon.ico Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

17
6_web_app/ocr_web_app/ocr_ui/public/index.html Normal file
View File

@ -0,0 +1,17 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
<link rel="icon" href="<%= BASE_URL %>favicon.ico">
<title><%= webpackConfig.name %></title>
</head>
<body>
<noscript>
<strong>We're sorry but <%= webpackConfig.name %> doesn't work properly without JavaScript enabled. Please enable it to continue.</strong>
</noscript>
<div id="app"></div>
<!-- built files will be auto injected -->
</body>
</html>

11
6_web_app/ocr_web_app/ocr_ui/src/App.vue Normal file
View File

@ -0,0 +1,11 @@
<template>
<div id="app">
<router-view />
</div>
</template>
<script>
export default {
name: 'App'
}
</script>

24
6_web_app/ocr_web_app/ocr_ui/src/api/inference.js Normal file
View File

@ -0,0 +1,24 @@
import request from '@/utils/request'
export function generalInfoForImageUrl(data) {
return request({
url: '/inference/generalInfoForImageUrl',
method: 'get',
params: {
url: data.url
}
})
}
export function mlsdForImageUrl(data) {
return request({
url: '/inference/mlsdForImageUrl',
method: 'get',
params: {
url: data.url
}
})
}
export default { generalInfoForImageUrl, mlsdForImageUrl }

24
6_web_app/ocr_web_app/ocr_ui/src/api/table.js Normal file
View File

@ -0,0 +1,24 @@
import request from '@/utils/request'
export function tableInfoForImageUrl(data) {
return request({
url: '/table/tableInfoForImageUrl',
method: 'get',
params: {
url: data.url
}
})
}
export function autoTableInfoForImageUrl(data) {
return request({
url: '/table/autoTableInfoForImageUrl',
method: 'get',
params: {
url: data.url
}
})
}
export default { tableInfoForImageUrl, autoTableInfoForImageUrl }

BIN
6_web_app/ocr_web_app/ocr_ui/src/assets/404_images/404.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 96 KiB

BIN
6_web_app/ocr_web_app/ocr_ui/src/assets/404_images/404_cloud.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.7 KiB

55
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/base.scss Normal file
View File

@ -0,0 +1,55 @@
// flex row
@mixin flex-row {
display: flex;
flex-direction: row;
}
@mixin flex-row-between {
@include flex-row();
justify-content: space-between;
}
@mixin flex-row-between-center {
@include flex-row-between();
align-items: center
}
@mixin flex-row-center {
@include flex-row();
justify-content: center
}
@mixin flex-row-all-center {
@include flex-row-center;
align-items: center
}
@mixin all-height($height) {
height: $height;
line-height: $height
}
@mixin ellipsis($width) {
width: $width;
display: inline-block;
overflow: hidden;
white-space: nowrap;
text-overflow: ellipsis
}
// flex column
@mixin flex-column {
display: flex;
flex-direction: column
}
@mixin flex-column-center {
@include flex-column();
justify-content: center
}
@mixin flex-column-all-center {
@include flex-column-center;
align-items: center
}

99
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/btn.scss Normal file
View File

@ -0,0 +1,99 @@
@import 'variables';
@mixin colorBtn($color) {
background: $color;
&:hover {
color: $color;
&:before,
&:after {
background: $color;
}
}
}
.blue-btn {
@include colorBtn($blue)
}
.light-blue-btn {
@include colorBtn($light-blue)
}
.red-btn {
@include colorBtn($red)
}
.pink-btn {
@include colorBtn($pink)
}
.green-btn {
@include colorBtn($green)
}
.tiffany-btn {
@include colorBtn($tiffany)
}
.yellow-btn {
@include colorBtn($yellow)
}
.pan-btn {
font-size: 14px;
color: #fff;
padding: 14px 36px;
border-radius: 8px;
border: none;
outline: none;
transition: 600ms ease all;
position: relative;
display: inline-block;
&:hover {
background: #fff;
&:before,
&:after {
width: 100%;
transition: 600ms ease all;
}
}
&:before,
&:after {
content: '';
position: absolute;
top: 0;
right: 0;
height: 2px;
width: 0;
transition: 400ms ease all;
}
&::after {
right: inherit;
top: inherit;
left: 0;
bottom: 0;
}
}
.custom-button {
display: inline-block;
line-height: 1;
white-space: nowrap;
cursor: pointer;
background: #fff;
color: #fff;
-webkit-appearance: none;
text-align: center;
box-sizing: border-box;
outline: 0;
margin: 0;
padding: 10px 15px;
font-size: 14px;
border-radius: 4px;
}

117
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/eladmin.scss Normal file
View File

@ -0,0 +1,117 @@
.head-container {
padding-bottom: 10px;
.filter-item {
display: inline-block;
vertical-align: middle;
margin: 0 3px 10px 0;
input {
height: 30.5px;
line-height: 30.5px;
}
}
.el-form-item-label {
margin: 0 3px 9px 0;
display: inline-block;
text-align: right;
vertical-align: middle;
font-size: 14px;
color: #606266;
line-height: 30.5px;
padding: 0 7px 0 7px;
}
.el-button+.el-button {
margin-left: 0 !important;
}
.el-select__caret.el-input__icon.el-icon-arrow-up{
line-height: 30.5px;
}
.date-item {
display: inline-block;
vertical-align: middle;
margin-bottom: 10px;
height: 30.5px !important;
width: 230px !important;
}
}
.el-avatar {
display: inline-block;
text-align: center;
background: #ccc;
color: #fff;
white-space: nowrap;
position: relative;
overflow: hidden;
vertical-align: middle;
width: 32px;
height: 32px;
line-height: 32px;
border-radius: 16px;
}
.logo-con{
height: 60px;
padding: 13px 0 0;
img{
height: 32px;
width: 135px;
display: block;
//margin: 0 auto;
}
}
#el-login-footer {
height: 40px;
line-height: 40px;
position: fixed;
bottom: 0;
width: 100%;
text-align: center;
color: #fff;
font-family: Arial, serif;
font-size: 12px;
letter-spacing: 1px;
}
#el-main-footer {
background: none repeat scroll 0 0 white;
border-top: 1px solid #e7eaec;
overflow: hidden;
padding: 10px 6px 0 6px;
height: 33px;
font-size: 0.7rem !important;
color: #7a8b9a;
letter-spacing: 0.8px;
font-family: Arial, sans-serif !important;
position: fixed;
bottom: 0;
z-index: 99;
width: 100%;
}
.eladmin-upload {
border: 1px dashed #c0ccda;
border-radius: 5px;
height: 45px;
line-height: 45px;
width: 368px;
}
.my-blockquote{
margin: 0 0 10px;
padding: 15px;
line-height: 22px;
border-left: 5px solid #00437B;
border-radius: 0 2px 2px 0;
background-color: #f2f2f2;
}
.my-code{
position: relative;
padding: 15px;
line-height: 20px;
border-left: 5px solid #ddd;
color: #333;
font-family: Courier New, serif;
font-size: 12px
}
.el-tabs{
margin-bottom: 25px;
}

79
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/element-ui.scss Normal file
View File

@ -0,0 +1,79 @@
// cover some element-ui styles
.el-breadcrumb__inner,
.el-breadcrumb__inner a {
font-weight: 400 !important;
}
.el-upload {
input[type="file"] {
display: none !important;
}
}
.el-upload__input {
display: none;
}
.cell {
.el-tag {
margin-right: 0;
}
}
.small-padding {
.cell {
padding-left: 5px;
padding-right: 5px;
}
}
.fixed-width {
.el-button--mini {
padding: 7px 10px;
width: 60px;
}
}
.status-col {
.cell {
padding: 0 10px;
text-align: center;
.el-tag {
margin-right: 0;
}
}
}
// to fixed https://github.com/ElemeFE/element/issues/2461
.el-dialog {
transform: none;
left: 0;
position: relative;
margin: 0 auto;
}
// refine element ui upload
.upload-container {
.el-upload {
width: 100%;
.el-upload-dragger {
width: 100%;
height: 200px;
}
}
}
// dropdown
.el-dropdown-menu {
a {
display: block
}
}
// fix date-picker ui bug in filter-item
.el-range-editor.el-input__inner {
display: inline-flex !important;
}

31
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/element-variables.scss Normal file
View File

@ -0,0 +1,31 @@
/**
* I think element-ui's default theme color is too light for long-term use.
* So I modified the default color and you can modify it to your liking.
**/
/* theme color */
$--color-primary: #1890ff;
$--color-success: #13ce66;
$--color-warning: #FFBA00;
$--color-danger: #ff4949;
// $--color-info: #1E1E1E;
$--button-font-weight: 400;
// $--color-text-regular: #1f2d3d;
$--border-color-light: #dfe4ed;
$--border-color-lighter: #e6ebf5;
$--table-border:1px solid#dfe6ec;
/* icon font path, required */
$--font-path: '~element-ui/lib/theme-chalk/fonts';
@import "../../../node_modules/element-ui/packages/theme-chalk/src/index";
// the :export directive is the magic sauce for webpack
// https://www.bluematador.com/blog/how-to-share-variables-between-js-and-sass
:export {
theme: $--color-primary;
}

182
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/index.scss Normal file
View File

@ -0,0 +1,182 @@
@import 'variables';
@import 'mixin';
@import 'transition';
@import 'element-ui';
@import 'sidebar';
@import 'btn';
@import 'eladmin';
body {
height: 100%;
-moz-osx-font-smoothing: grayscale;
-webkit-font-smoothing: antialiased;
text-rendering: optimizeLegibility;
font-family: Helvetica Neue, Helvetica, PingFang SC, Hiragino Sans GB, Microsoft YaHei, Arial, sans-serif;
}
label {
font-weight: 700;
}
html {
height: 100%;
box-sizing: border-box;
}
#app {
height: 100%;
}
*,
*:before,
*:after {
box-sizing: inherit;
}
.no-padding {
padding: 0 !important;
}
.padding-content {
padding: 4px 0;
}
a:focus,
a:active {
outline: none;
}
a,
a:focus,
a:hover {
cursor: pointer;
color: inherit;
text-decoration: none;
}
div:focus {
outline: none;
}
.fr {
float: right;
}
.fl {
float: left;
}
.pr-5 {
padding-right: 5px;
}
.pl-5 {
padding-left: 5px;
}
.block {
display: block;
}
.pointer {
cursor: pointer;
}
.inlineBlock {
display: block;
}
.clearfix {
&:after {
visibility: hidden;
display: block;
font-size: 0;
content: " ";
clear: both;
height: 0;
}
}
aside {
background: #eef1f6;
padding: 8px 24px;
margin-bottom: 20px;
border-radius: 2px;
display: block;
line-height: 32px;
font-size: 16px;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, Cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif;
color: #2c3e50;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
a {
color: #337ab7;
cursor: pointer;
&:hover {
color: rgb(32, 160, 255);
}
}
}
//main-container全局样式
.app-container {
padding: 20px 20px 45px 20px;
}
.components-container {
margin: 30px 50px;
position: relative;
}
.pagination-container {
margin-top: 30px;
}
.text-center {
text-align: center
}
.sub-navbar {
height: 50px;
line-height: 50px;
position: relative;
width: 100%;
text-align: right;
padding-right: 20px;
transition: 600ms ease position;
background: linear-gradient(90deg, rgba(32, 182, 249, 1) 0%, rgba(32, 182, 249, 1) 0%, rgba(33, 120, 241, 1) 100%, rgba(33, 120, 241, 1) 100%);
.subtitle {
font-size: 20px;
color: #fff;
}
&.draft {
background: #d0d0d0;
}
&.deleted {
background: #d0d0d0;
}
}
.link-type,
.link-type:focus {
color: #337ab7;
cursor: pointer;
&:hover {
color: rgb(32, 160, 255);
}
}
//refine vue-multiselect plugin
.multiselect {
line-height: 16px;
}
.multiselect--active {
z-index: 1000 !important;
}

66
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/mixin.scss Normal file
View File

@ -0,0 +1,66 @@
@mixin clearfix {
&:after {
content: "";
display: table;
clear: both;
}
}
@mixin scrollBar {
&::-webkit-scrollbar-track-piece {
background: #d3dce6;
}
&::-webkit-scrollbar {
width: 6px;
}
&::-webkit-scrollbar-thumb {
background: #99a9bf;
border-radius: 20px;
}
}
@mixin relative {
position: relative;
width: 100%;
height: 100%;
}
@mixin pct($pct) {
width: #{$pct};
position: relative;
margin: 0 auto;
}
@mixin triangle($width, $height, $color, $direction) {
$width: $width/2;
$color-border-style: $height solid $color;
$transparent-border-style: $width solid transparent;
height: 0;
width: 0;
@if $direction==up {
border-bottom: $color-border-style;
border-left: $transparent-border-style;
border-right: $transparent-border-style;
}
@else if $direction==right {
border-left: $color-border-style;
border-top: $transparent-border-style;
border-bottom: $transparent-border-style;
}
@else if $direction==down {
border-top: $color-border-style;
border-left: $transparent-border-style;
border-right: $transparent-border-style;
}
@else if $direction==left {
border-right: $color-border-style;
border-top: $transparent-border-style;
border-bottom: $transparent-border-style;
}
}

209
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/sidebar.scss Normal file
View File

@ -0,0 +1,209 @@
#app {
.main-container {
min-height: 100%;
transition: margin-left .28s;
margin-left: $sideBarWidth;
position: relative;
}
.sidebar-container {
transition: width 0.28s;
width: $sideBarWidth !important;
background-color: $menuBg;
height: 100%;
position: fixed;
font-size: 0;
top: 0;
bottom: 0;
left: 0;
z-index: 1001;
overflow: hidden;
// reset element-ui css
.horizontal-collapse-transition {
transition: 0s width ease-in-out, 0s padding-left ease-in-out, 0s padding-right ease-in-out;
}
.scrollbar-wrapper {
overflow-x: hidden !important;
}
.el-scrollbar__bar.is-vertical {
right: 0;
}
.el-scrollbar {
height: 100%;
}
&.has-logo {
.el-scrollbar {
height: calc(100% - 50px);
}
}
.is-horizontal {
display: none;
}
a {
display: inline-block;
width: 100%;
overflow: hidden;
}
.svg-icon {
margin-right: 16px;
}
.el-menu {
border: none;
height: 100%;
width: 100% !important;
}
// menu hover
.submenu-title-noDropdown,
.el-submenu__title {
&:hover {
background-color: $menuHover !important;
}
}
.is-active>.el-submenu__title {
color: $subMenuActiveText !important;
}
& .nest-menu .el-submenu>.el-submenu__title,
& .el-submenu .el-menu-item {
min-width: $sideBarWidth !important;
background-color: $subMenuBg !important;
&:hover {
background-color: $subMenuHover !important;
}
}
}
.hideSidebar {
.sidebar-container {
width: 54px !important;
}
.main-container {
margin-left: 54px;
}
.submenu-title-noDropdown {
padding: 0 !important;
position: relative;
.el-tooltip {
padding: 0 !important;
.svg-icon {
margin-left: 20px;
}
}
}
.el-submenu {
overflow: hidden;
&>.el-submenu__title {
padding: 0 !important;
.svg-icon {
margin-left: 20px;
}
.el-submenu__icon-arrow {
display: none;
}
}
}
.el-menu--collapse {
.el-submenu {
&>.el-submenu__title {
&>span {
height: 0;
width: 0;
overflow: hidden;
visibility: hidden;
display: inline-block;
}
}
}
}
}
.el-menu--collapse .el-menu .el-submenu {
min-width: $sideBarWidth !important;
}
// mobile responsive
.mobile {
.main-container {
margin-left: 0;
}
.sidebar-container {
transition: transform .28s;
width: $sideBarWidth !important;
}
&.hideSidebar {
.sidebar-container {
pointer-events: none;
transition-duration: 0.3s;
transform: translate3d(-$sideBarWidth, 0, 0);
}
}
}
.withoutAnimation {
.main-container,
.sidebar-container {
transition: none;
}
}
}
// when menu collapsed
.el-menu--vertical {
&>.el-menu {
.svg-icon {
margin-right: 16px;
}
}
.nest-menu .el-submenu>.el-submenu__title,
.el-menu-item {
&:hover {
// you can use $subMenuHover
background-color: $menuHover !important;
}
}
// the scroll bar appears when the subMenu is too long
>.el-menu--popup {
max-height: 100vh;
overflow-y: auto;
&::-webkit-scrollbar-track-piece {
background: #d3dce6;
}
&::-webkit-scrollbar {
width: 6px;
}
&::-webkit-scrollbar-thumb {
background: #99a9bf;
border-radius: 20px;
}
}
}

48
6_web_app/ocr_web_app/ocr_ui/src/assets/styles/transition.scss Normal file
View File

@ -0,0 +1,48 @@
// global transition css
/* fade */
.fade-enter-active,
.fade-leave-active {
transition: opacity 0.28s;
}
.fade-enter,
.fade-leave-active {
opacity: 0;
}
/* fade-transform */
.fade-transform-leave-active,
.fade-transform-enter-active {
transition: all .5s;
}
.fade-transform-enter {
opacity: 0;
transform: translateX(-30px);
}
.fade-transform-leave-to {
opacity: 0;
transform: translateX(30px);
}
/* breadcrumb transition */
.breadcrumb-enter-active,
.breadcrumb-leave-active {
transition: all .5s;
}
.breadcrumb-enter,
.breadcrumb-leave-active {
opacity: 0;
transform: translateX(20px);
}
.breadcrumb-move {
transition: all .5s;
}
.breadcrumb-leave-active {
position: absolute;
}

Some files were not shown because too many files have changed in this diff Show More