test/AIAS
1
0
Fork 0
mirror of https://gitee.com/mymagicpower/AIAS.git synced 2024-12-04 21:27:50 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

upgrade milvus to v2.0

Browse Source
This commit is contained in:
Calvin 2022-02-13 19:08:17 +08:00
parent e10b6fe497
commit 2952dc03f5
15 changed files with 773 additions and 1026 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

79
6_biomedicine/dna_sequence_search/README.md
View File

@ -5,7 +5,7 @@ http://aias.top/
本例子提供了DNA序列搜索，支持上传文件文件，使用spark mlib计算模型提取特征，并基于milvus向量引擎进行后续检索。
<div align="center">
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine_sdks/dna_sequence_search/arc.png" width = "600"/>
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine/dna_sequence_search/arc.png" width = "600"/>
</div>
#### 引擎特性
@ -31,7 +31,7 @@ DNA序列测定方法有光学测序和芯片测序两种。
超参数minDF则指定词汇表中的词语至少要在多少个不同文档中出现。
模型训练，推理使用了spark mlib:
<div align="center">
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine_sdks/dna_sequence_search/spark_mlib.png" width = "400"/>
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine/dna_sequence_search/spark_mlib.png" width = "400"/>
</div>
- [算法详细介绍](http://spark.apache.org/docs/latest/ml-features.html#countvectorizer)
@ -39,7 +39,7 @@ DNA序列测定方法有光学测序和芯片测序两种。
#### 向量引擎索引策略
<div align="center">
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine_sdks/dna_sequence_search/milvus.png" width = "600"/>
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine/dna_sequence_search/milvus.png" width = "600"/>
</div>
@ -100,7 +100,7 @@ file:
linux:
path: /home/aias/file/
windows:
path: D:/aias/file/
path: file:/D:/aias/file/
# 文件大小 /M
maxSize: 3000
...
@ -114,51 +114,30 @@ java -jar dna-sequence-search-0.1.0.jar
```
## 3. 后端向量引擎部署（docker）
### 3. 后端向量引擎部署（Milvus 2.0）
#### 3.1 环境要求：
- 需要安装docker运行环境，Mac环境可以使用Docker Desktop
#### 3.2 拉取Milvus向量引擎镜像（用于计算特征值向量相似度）
[安装文档](https://github.com/milvus-io/docs/blob/master/v0.10.0/site/zh-CN/quick_start/install_milvus/cpu_milvus_docker.md)
##### 最新版本请参考官网
- Milvus向量引擎参考链接
[Milvus向量引擎官网](https://milvus.io/cn/docs/overview.md)
[Milvus向量引擎Github](https://github.com/milvus-io)
下载 milvus-standalone-docker-compose.yml 配置文件并保存为 docker-compose.yml
[单机版安装文档](https://milvus.io/docs/v2.0.0/install_standalone-docker.md)
```bash
sudo docker pull milvusdb/milvus:0.10.0-cpu-d061620-5f3c00
wget https://github.com/milvus-io/milvus/releases/download/v2.0.0/milvus-standalone-docker-compose.yml -O docker-compose.yml
```
#### 3.3 下载配置文件
[vector_engine.zip](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/image_search/vector_engine.zip)
#### 3.4 启动 Docker 容器
/Users/calvin/vector_engine为主机路径，根据需要修改。conf下为引擎所需的配置文件。
#### 3.3 启动 Docker 容器
```bash
docker run -d --name milvus_cpu_0.10.0 \
-p 19530:19530 \
-p 19121:19121 \
-p 9091:9091 \
-v /Users/calvin/vector_engine/db:/var/lib/milvus/db \
-v /Users/calvin/vector_engine/conf:/var/lib/milvus/conf \
-v /Users/calvin/vector_engine/logs:/var/lib/milvus/logs \
-v /Users/calvin/vector_engine/wal:/var/lib/milvus/wal \
milvusdb/milvus:0.10.0-cpu-d061620-5f3c00
sudo docker-compose up -d
```
#### 3.5 编辑向量引擎连接配置信息
- application.yml
- 根据需要编辑向量引擎连接ip地址127.0.0.1为容器所在的主机ip
```bash
##################### 向量引擎 ###############################
################## 向量引擎 ################
search:
host: 127.0.0.1
port: 19530
indexFileSize: 1024 # maximum size (in MB) of each index file
nprobe: 16
nlist: 16384
collectionName: dna #collection name
```
#### 4. 打开浏览器
@ -166,11 +145,11 @@ search:
- 上传数据文件
1). 点击上传按钮上传文件.
[测试数据](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine_sdks/dna_sequence_search/human_data.txt)
[测试数据](https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine/dna_sequence_search/human_data.txt)
2). 点击特征提取按钮.
等待文件解析，模型训练，特征提取，特征存入向量引擎。通过console可以看到进度信息。
<div align="center">
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine_sdks/dna_sequence_search/storage.png" width = "600"/>
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine/dna_sequence_search/storage.png" width = "600"/>
</div>
- DNA序列搜索
@ -179,42 +158,26 @@ search:
ATGCCCCAACTAAATACTACCGTATGGCCCACCATAATTACCCCCATACTCCTTACACTATTCCTCATCACCCAACTAAAAATATTAAACACAAACTACCACCTACCTCCCTCACCAAAGCCCATAAAAATAAAAAATTATAACAAACCCTGAGAACCAAAATGAACGAAAATCTGTTCGCTTCATTCATTGCCCCCACAATCCTAG
```
<div align="center">
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine_sdks/dna_sequence_search/search.png" width = "600"/>
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine/dna_sequence_search/search.png" width = "600"/>
</div>
## 5. 帮助信息
- swagger接口文档:
http://localhost:8089/swagger-ui.html
<div align="center">
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine_sdks/dna_sequence_search/swagger.png" width = "600"/>
<img src="https://aias-home.oss-cn-beijing.aliyuncs.com/AIAS/6_biomedicine/dna_sequence_search/swagger.png" width = "600"/>
</div>
- 初始化向量引擎(清空数据):
me.aias.tools.MilvusInit.java
```bash
String host = "127.0.0.1";
int port = 19530;
final String collectionName = "dna"; // collection name
MilvusClient client = new MilvusGrpcClient();
// Connect to Milvus server
ConnectParam connectParam = new ConnectParam.Builder().withHost(host).withPort(port).build();
try {
Response connectResponse = client.connect(connectParam);
} catch (ConnectFailedException e) {
e.printStackTrace();
}
// 检查 collection 是否存在
HasCollectionResponse hasCollection = hasCollection(client, collectionName);
if (hasCollection.hasCollection()) {
dropIndex(client, collectionName);
dropCollection(client, collectionName);
}
...
me.aias.tools.MilvusInit.java
```
- Milvus向量引擎参考链接
[Milvus向量引擎官网](https://milvus.io/cn/docs/overview.md)
[Milvus向量引擎Github](https://github.com/milvus-io)
### 官网：
[官网链接](http://www.aias.top/)

2
6_biomedicine/dna_sequence_search/dna-search-ui/package.json
View File

@ -15,7 +15,7 @@
"dependencies": {
"crypto-js": "^3.1.9-1",
"axios": "0.18.1",
"core-js": "3.6.5",
"core-js": "3.21.0",
"easy-circular-progress": "1.0.4",
"echarts": "^4.2.1",
"element-ui": "2.13.2",

319
6_biomedicine/dna_sequence_search/dna_sequence_search/dna_sequence_search.iml
View File

@ -1,319 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
<component name="CheckStyle-IDEA-Module">
<option name="configuration">
<map />
</option>
</component>
<component name="FacetManager">
<facet type="Spring" name="Spring">
<configuration />
</facet>
<facet type="web" name="Web">
<configuration>
<webroots />
<sourceRoots>
<root url="file://$MODULE_DIR$/src/main/java" />
<root url="file://$MODULE_DIR$/src/main/resources" />
</sourceRoots>
</configuration>
</facet>
</component>
<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
<output url="file://$MODULE_DIR$/target/classes" />
<output-test url="file://$MODULE_DIR$/target/test-classes" />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/src/main/resources" type="java-resource" />
<sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
<sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
<excludeFolder url="file://$MODULE_DIR$/target" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-context:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-autoconfigure:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-logging:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: ch.qos.logback:logback-classic:1.2.3" level="project" />
<orderEntry type="library" name="Maven: ch.qos.logback:logback-core:1.2.3" level="project" />
<orderEntry type="library" name="Maven: org.apache.logging.log4j:log4j-to-slf4j:2.11.2" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:jul-to-slf4j:1.7.28" level="project" />
<orderEntry type="library" name="Maven: javax.annotation:javax.annotation-api:1.3.2" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-core:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-jcl:5.1.10.RELEASE" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: org.yaml:snakeyaml:1.23" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-web:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-json:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.9" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.9" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.9" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-tomcat:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.apache.tomcat.embed:tomcat-embed-core:9.0.26" level="project" />
<orderEntry type="library" name="Maven: org.apache.tomcat.embed:tomcat-embed-el:9.0.26" level="project" />
<orderEntry type="library" name="Maven: org.apache.tomcat.embed:tomcat-embed-websocket:9.0.26" level="project" />
<orderEntry type="library" name="Maven: org.hibernate.validator:hibernate-validator:6.0.17.Final" level="project" />
<orderEntry type="library" name="Maven: javax.validation:validation-api:2.0.1.Final" level="project" />
<orderEntry type="library" name="Maven: org.jboss.logging:jboss-logging:3.3.3.Final" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-web:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-webmvc:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-expression:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-aop:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-aop:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.aspectj:aspectjweaver:1.9.4" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework.boot:spring-boot-starter-test:2.1.9.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework.boot:spring-boot-test:2.1.9.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework.boot:spring-boot-test-autoconfigure:2.1.9.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: com.jayway.jsonpath:json-path:2.4.0" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: net.minidev:json-smart:2.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: net.minidev:accessors-smart:1.2" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.ow2.asm:asm:5.0.4" level="project" />
<orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.assertj:assertj-core:3.11.1" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.mockito:mockito-core:2.23.4" level="project" />
<orderEntry type="library" name="Maven: net.bytebuddy:byte-buddy:1.9.16" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: net.bytebuddy:byte-buddy-agent:1.9.16" level="project" />
<orderEntry type="library" name="Maven: org.objenesis:objenesis:2.6" level="project" />
<orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.hamcrest:hamcrest-library:1.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.skyscreamer:jsonassert:1.5.0" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: com.vaadin.external.google:android-json:0.0.20131108.vaadin1" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework:spring-test:5.1.10.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.xmlunit:xmlunit-core:2.6.3" level="project" />
<orderEntry type="library" name="Maven: javax.xml.bind:jaxb-api:2.3.1" level="project" />
<orderEntry type="library" name="Maven: org.springframework.data:spring-data-commons:2.1.2.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-beans:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.28" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.8.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.logging.log4j:log4j-slf4j-impl:2.16.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.logging.log4j:log4j-api:2.11.2" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: org.apache.logging.log4j:log4j-core:2.11.2" level="project" />
<orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
<orderEntry type="library" name="Maven: io.milvus:milvus-sdk-java:0.8.2" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven.plugins:maven-gpg-plugin:1.6" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-plugin-api:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-project:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-settings:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-profile:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-artifact-manager:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven.wagon:wagon-provider-api:1.0-beta-6" level="project" />
<orderEntry type="library" name="Maven: backport-util-concurrent:backport-util-concurrent:3.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-plugin-registry:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.plexus:plexus-interpolation:1.11" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.plexus:plexus-container-default:1.0-alpha-9-stable-1" level="project" />
<orderEntry type="library" name="Maven: classworlds:classworlds:1.1-alpha-2" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-artifact:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-repository-metadata:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-model:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.plexus:plexus-utils:3.0.20" level="project" />
<orderEntry type="library" name="Maven: org.sonatype.plexus:plexus-sec-dispatcher:1.4" level="project" />
<orderEntry type="library" name="Maven: org.sonatype.plexus:plexus-cipher:1.4" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: io.grpc:grpc-netty-shaded:1.27.2" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: io.grpc:grpc-core:1.27.2" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: com.google.android:annotations:4.1.1.4" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: io.perfmark:perfmark-api:0.19.0" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-protobuf:1.27.2" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-api:1.27.2" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-context:1.27.2" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.18" level="project" />
<orderEntry type="library" name="Maven: com.google.protobuf:protobuf-java:3.11.0" level="project" />
<orderEntry type="library" name="Maven: com.google.api.grpc:proto-google-common-protos:1.17.0" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-protobuf-lite:1.27.2" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-stub:1.27.2" level="project" />
<orderEntry type="library" name="Maven: com.google.protobuf:protobuf-java-util:3.11.0" level="project" />
<orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.3.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-text:1.6" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-collections4:4.4" level="project" />
<orderEntry type="library" name="Maven: org.json:json:20190722" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-sql_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: com.univocity:univocity-parsers:2.9.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-sketch_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-core_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: com.thoughtworks.paranamer:paranamer:2.8" level="project" />
<orderEntry type="library" name="Maven: org.apache.avro:avro:1.8.2" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-compress:1.8.1" level="project" />
<orderEntry type="library" name="Maven: org.tukaani:xz:1.5" level="project" />
<orderEntry type="library" name="Maven: org.apache.avro:avro-mapred:hadoop2:1.8.2" level="project" />
<orderEntry type="library" name="Maven: org.apache.avro:avro-ipc:1.8.2" level="project" />
<orderEntry type="library" name="Maven: com.twitter:chill_2.12:0.9.5" level="project" />
<orderEntry type="library" name="Maven: com.esotericsoftware:kryo-shaded:4.0.2" level="project" />
<orderEntry type="library" name="Maven: com.esotericsoftware:minlog:1.3.0" level="project" />
<orderEntry type="library" name="Maven: com.twitter:chill-java:0.9.5" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-client:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-common:2.7.4" level="project" />
<orderEntry type="library" name="Maven: xmlenc:xmlenc:0.52" level="project" />
<orderEntry type="library" name="Maven: commons-httpclient:commons-httpclient:3.1" level="project" />
<orderEntry type="library" name="Maven: commons-io:commons-io:2.4" level="project" />
<orderEntry type="library" name="Maven: commons-collections:commons-collections:3.2.2" level="project" />
<orderEntry type="library" name="Maven: org.mortbay.jetty:jetty-sslengine:6.1.26" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: javax.servlet.jsp:jsp-api:2.1" level="project" />
<orderEntry type="library" name="Maven: commons-configuration:commons-configuration:1.6" level="project" />
<orderEntry type="library" name="Maven: commons-digester:commons-digester:1.8" level="project" />
<orderEntry type="library" name="Maven: commons-beanutils:commons-beanutils:1.7.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-auth:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15" level="project" />
<orderEntry type="library" name="Maven: org.apache.directory.server:apacheds-i18n:2.0.0-M15" level="project" />
<orderEntry type="library" name="Maven: org.apache.directory.api:api-asn1-api:1.0.0-M20" level="project" />
<orderEntry type="library" name="Maven: org.apache.directory.api:api-util:1.0.0-M20" level="project" />
<orderEntry type="library" name="Maven: org.apache.curator:curator-client:2.7.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.htrace:htrace-core:3.1.0-incubating" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-hdfs:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.mortbay.jetty:jetty-util:6.1.26" level="project" />
<orderEntry type="library" name="Maven: xerces:xercesImpl:2.9.1" level="project" />
<orderEntry type="library" name="Maven: xml-apis:xml-apis:1.4.01" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-mapreduce-client-app:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-mapreduce-client-common:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-yarn-client:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-yarn-server-common:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-mapreduce-client-shuffle:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-yarn-api:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-mapreduce-client-core:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-yarn-common:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-jaxrs:1.9.13" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-xc:1.9.13" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-mapreduce-client-jobclient:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.hadoop:hadoop-annotations:2.7.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-launcher_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-kvstore_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.fusesource.leveldbjni:leveldbjni-all:1.8" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-network-common_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-network-shuffle_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-unsafe_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: javax.activation:activation:1.1.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.curator:curator-recipes:2.7.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.curator:curator-framework:2.7.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.zookeeper:zookeeper:3.4.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.yetus:audience-annotations:0.5.0" level="project" />
<orderEntry type="library" name="Maven: javax.servlet:javax.servlet-api:4.0.1" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:jcl-over-slf4j:1.7.28" level="project" />
<orderEntry type="library" name="Maven: log4j:log4j:1.2.17" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:slf4j-log4j12:1.7.28" level="project" />
<orderEntry type="library" name="Maven: com.ning:compress-lzf:1.0.3" level="project" />
<orderEntry type="library" name="Maven: org.xerial.snappy:snappy-java:1.1.7.5" level="project" />
<orderEntry type="library" name="Maven: org.lz4:lz4-java:1.7.1" level="project" />
<orderEntry type="library" name="Maven: com.github.luben:zstd-jni:1.4.4-3" level="project" />
<orderEntry type="library" name="Maven: org.roaringbitmap:RoaringBitmap:0.7.45" level="project" />
<orderEntry type="library" name="Maven: org.roaringbitmap:shims:0.7.45" level="project" />
<orderEntry type="library" name="Maven: commons-net:commons-net:3.1" level="project" />
<orderEntry type="library" name="Maven: org.scala-lang.modules:scala-xml_2.12:1.2.0" level="project" />
<orderEntry type="library" name="Maven: org.scala-lang:scala-library:2.12.10" level="project" />
<orderEntry type="library" name="Maven: org.scala-lang:scala-reflect:2.12.10" level="project" />
<orderEntry type="library" name="Maven: org.json4s:json4s-jackson_2.12:3.6.6" level="project" />
<orderEntry type="library" name="Maven: org.json4s:json4s-core_2.12:3.6.6" level="project" />
<orderEntry type="library" name="Maven: org.json4s:json4s-ast_2.12:3.6.6" level="project" />
<orderEntry type="library" name="Maven: org.json4s:json4s-scalap_2.12:3.6.6" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jersey.core:jersey-client:2.27" level="project" />
<orderEntry type="library" name="Maven: javax.ws.rs:javax.ws.rs-api:2.1" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.hk2.external:javax.inject:2.5.0-b42" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jersey.core:jersey-common:2.27" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.hk2:osgi-resource-locator:1.0.1" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jersey.core:jersey-server:2.27" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jersey.media:jersey-media-jaxb:2.27" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jersey.containers:jersey-container-servlet:2.27" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jersey.containers:jersey-container-servlet-core:2.27" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jersey.inject:jersey-hk2:2.27" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.hk2:hk2-locator:2.5.0-b42" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.hk2.external:aopalliance-repackaged:2.5.0-b42" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.hk2:hk2-api:2.5.0-b42" level="project" />
<orderEntry type="library" name="Maven: javax.inject:javax.inject:1" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.hk2:hk2-utils:2.5.0-b42" level="project" />
<orderEntry type="library" name="Maven: org.javassist:javassist:3.22.0-CR2" level="project" />
<orderEntry type="library" name="Maven: io.netty:netty-all:4.1.39.Final" level="project" />
<orderEntry type="library" name="Maven: com.clearspring.analytics:stream:2.9.6" level="project" />
<orderEntry type="library" name="Maven: io.dropwizard.metrics:metrics-core:4.0.6" level="project" />
<orderEntry type="library" name="Maven: io.dropwizard.metrics:metrics-jvm:4.0.6" level="project" />
<orderEntry type="library" name="Maven: io.dropwizard.metrics:metrics-json:4.0.6" level="project" />
<orderEntry type="library" name="Maven: io.dropwizard.metrics:metrics-graphite:4.0.6" level="project" />
<orderEntry type="library" name="Maven: io.dropwizard.metrics:metrics-jmx:4.0.6" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.module:jackson-module-scala_2.12:2.9.9" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.module:jackson-module-paranamer:2.9.9" level="project" />
<orderEntry type="library" name="Maven: org.apache.ivy:ivy:2.4.0" level="project" />
<orderEntry type="library" name="Maven: oro:oro:2.0.8" level="project" />
<orderEntry type="library" name="Maven: net.razorvine:pyrolite:4.30" level="project" />
<orderEntry type="library" name="Maven: net.sf.py4j:py4j:0.10.9" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-crypto:1.0.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-catalyst_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.janino:janino:3.0.15" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.janino:commons-compiler:3.0.15" level="project" />
<orderEntry type="library" name="Maven: org.antlr:antlr4-runtime:4.7.1" level="project" />
<orderEntry type="library" name="Maven: commons-codec:commons-codec:1.11" level="project" />
<orderEntry type="library" name="Maven: org.apache.arrow:arrow-vector:0.15.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.arrow:arrow-format:0.15.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.arrow:arrow-memory:0.15.1" level="project" />
<orderEntry type="library" name="Maven: com.google.flatbuffers:flatbuffers-java:1.9.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-tags_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.orc:orc-core:1.5.10" level="project" />
<orderEntry type="library" name="Maven: org.apache.orc:orc-shims:1.5.10" level="project" />
<orderEntry type="library" name="Maven: commons-lang:commons-lang:2.6" level="project" />
<orderEntry type="library" name="Maven: io.airlift:aircompressor:0.10" level="project" />
<orderEntry type="library" name="Maven: org.threeten:threeten-extra:1.5.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.orc:orc-mapreduce:1.5.10" level="project" />
<orderEntry type="library" name="Maven: org.apache.hive:hive-storage-api:2.7.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.parquet:parquet-column:1.10.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.parquet:parquet-common:1.10.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.parquet:parquet-encoding:1.10.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.parquet:parquet-hadoop:1.10.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.parquet:parquet-format:2.4.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.parquet:parquet-jackson:1.10.1" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-mapper-asl:1.9.13" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-core-asl:1.9.13" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-databind:2.9.9.3" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-core:2.9.9" level="project" />
<orderEntry type="library" name="Maven: org.apache.xbean:xbean-asm7-shaded:4.15" level="project" />
<orderEntry type="library" name="Maven: org.spark-project.spark:unused:1.0.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-mllib_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.scala-lang.modules:scala-parser-combinators_2.12:1.1.2" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-streaming_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-graphx_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: com.github.fommil.netlib:core:1.1.2" level="project" />
<orderEntry type="library" name="Maven: net.sourceforge.f2j:arpack_combined_all:0.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.spark:spark-mllib-local_2.12:3.0.1" level="project" />
<orderEntry type="library" name="Maven: org.scalanlp:breeze_2.12:1.0" level="project" />
<orderEntry type="library" name="Maven: org.scalanlp:breeze-macros_2.12:1.0" level="project" />
<orderEntry type="library" name="Maven: net.sf.opencsv:opencsv:2.3" level="project" />
<orderEntry type="library" name="Maven: com.github.wendykierp:JTransforms:3.1" level="project" />
<orderEntry type="library" name="Maven: pl.edu.icm:JLargeArrays:1.5" level="project" />
<orderEntry type="library" name="Maven: com.chuusai:shapeless_2.12:2.3.3" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:macro-compat_2.12:1.1.1" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:spire_2.12:0.17.0-M1" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:spire-macros_2.12:0.17.0-M1" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:spire-platform_2.12:0.17.0-M1" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:spire-util_2.12:0.17.0-M1" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:machinist_2.12:0.6.8" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:algebra_2.12:2.0.0-M2" level="project" />
<orderEntry type="library" name="Maven: org.typelevel:cats-kernel_2.12:2.0.0-M4" level="project" />
<orderEntry type="library" name="Maven: org.scala-lang.modules:scala-collection-compat_2.12:2.1.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-math3:3.4.1" level="project" />
<orderEntry type="library" name="Maven: org.glassfish.jaxb:jaxb-runtime:2.3.1" level="project" />
<orderEntry type="library" name="Maven: com.sun.istack:istack-commons-runtime:3.0.7" level="project" />
<orderEntry type="library" name="Maven: javax.activation:javax.activation-api:1.2.0" level="project" />
<orderEntry type="library" name="Maven: org.projectlombok:lombok:1.18.10" level="project" />
<orderEntry type="library" name="Maven: com.google.http-client:google-http-client:1.19.0" level="project" />
<orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:1.3.9" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.10" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.12" level="project" />
<orderEntry type="library" name="Maven: commons-cli:commons-cli:1.4" level="project" />
<orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.70" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-swagger2:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-spi:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-core:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-schema:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-swagger-common:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-spring-web:2.9.2" level="project" />
<orderEntry type="library" name="Maven: com.google.guava:guava:20.0" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml:classmate:1.4.0" level="project" />
<orderEntry type="library" name="Maven: org.springframework.plugin:spring-plugin-core:1.2.0.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.plugin:spring-plugin-metadata:1.2.0.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.mapstruct:mapstruct:1.2.0.Final" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-swagger-ui:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.swagger:swagger-annotations:1.5.21" level="project" />
<orderEntry type="library" name="Maven: io.swagger:swagger-models:1.5.21" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-annotations:2.9.0" level="project" />
<orderEntry type="library" name="Maven: cn.hutool:hutool-all:5.3.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.ant:ant:1.9.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.ant:ant-launcher:1.9.4" level="project" />
<orderEntry type="library" name="Maven: eu.bitwalker:UserAgentUtils:1.21" level="project" />
<orderEntry type="library" name="Maven: de.siegmar:fastcsv:1.0.1" level="project" />
</component>
</module>

2
6_biomedicine/dna_sequence_search/dna_sequence_search/pom.xml
View File

@ -65,7 +65,7 @@
<dependency>
<groupId>io.milvus</groupId>
<artifactId>milvus-sdk-java</artifactId>
<version>0.8.2</version>
<version>2.0.0</version>
</dependency>
<!-- Spark Mlib -->
<dependency>

235
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/common/milvus/ConnectionPool.java Normal file
View File

@ -0,0 +1,235 @@
package me.aias.common.milvus;
import io.milvus.client.MilvusClient;
import io.milvus.client.MilvusServiceClient;
import io.milvus.param.ConnectParam;
import java.util.Enumeration;
import java.util.Vector;
public class ConnectionPool {
private String host = ""; // Milvus 主机
private int port; // Milvus 端口号
private static volatile ConnectionPool uniqueInstance;
private int initialConnections = 10; // 连接池的初始大小
private int incrementalConnections = 5; // 连接池自动增加的大小
private int maxConnections = 50; // 连接池最大的大小
private Vector connections = null; // 存放连接池中连接的向量, 存放的对象为 PooledConnection 型
private ConnectionPool(String host, int port) {
this.host = host;
this.port = port;
}
public static ConnectionPool getInstance(String host, String port, boolean refresh) {
if (uniqueInstance == null || refresh) {
synchronized (ConnectionPool.class) {
if (uniqueInstance == null || refresh) {
uniqueInstance = new ConnectionPool(host, Integer.parseInt(port));
try {
uniqueInstance.createPool();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
}
return uniqueInstance;
}
private void createPool() { // synchronized
if (connections != null) {
return; // 假如己经创建，则返回
}
// 创建保存连接的向量 , 初始时有 0 个元素
connections = new Vector();
// 根据 initialConnections 中设置的值，创建连接。
createConnections(this.initialConnections);
System.out.println(" Milvus连接池创建成功！");
}
private void createConnections(int numConnections) {
// 循环创建指定数目的数据库连接
for (int x = 0; x < numConnections; x++) {
// 是否连接池中的Milvus连接数量己经达到最大？最大值由类成员 maxConnections
if (this.maxConnections > 0 && this.connections.size() >= this.maxConnections) {
break;
}
// 增加一个连接到连接池中（Vector connections）
connections.addElement(new PooledConnection(newConnection()));
System.out.println(" Milvus连接己创建 ......");
}
}
private MilvusClient newConnection() {
// 创建一个 Milvus 客户端
ConnectParam connectParam = ConnectParam.newBuilder()
.withHost(host)
.withPort(port)
.build();
MilvusServiceClient milvusClient = new MilvusServiceClient(connectParam);
// 返回创建的新的Milvus连接
return milvusClient;
}
public synchronized MilvusClient getConnection() {
// 确保连接池己被创建
if (connections == null) {
return null; // 连接池还没创建，则返回 null
}
MilvusClient client = getFreeConnection(); // 获得一个可用的数据库连接
// 假如目前没有可以使用的连接，即所有的连接都在使用中
while (client == null) {
// 等一会再试 250 ms
wait(250);
client = getFreeConnection(); // 重新再试，直到获得可用的连接，假如
// getFreeConnection() 返回的为 null
// 则表明创建一批连接后也不可获得可用连接
}
return client; // 返回获得的可用的连接
}
private MilvusClient getFreeConnection() {
// 从连接池中获得一个可用的Milvus连接
MilvusClient client = findFreeConnection();
if (client == null) {
// 假如目前连接池中没有可用的连接
// 创建一些连接
createConnections(incrementalConnections);
// 重新从池中查找是否有可用连接
client = findFreeConnection();
if (client == null) {
// 假如创建连接后仍获得不到可用的连接，则返回 null
return null;
}
}
return client;
}
private MilvusClient findFreeConnection() {
MilvusClient client = null;
PooledConnection pConn = null;
// 获得连接池中所有的对象
Enumeration enumerate = connections.elements();
// 遍历所有的对象，看是否有可用的连接
while (enumerate.hasMoreElements()) {
pConn = (PooledConnection) enumerate.nextElement();
if (!pConn.isBusy()) {
// 假如此对象不忙，则获得它的数据库连接并把它设为忙
client = pConn.getConnection();
pConn.setBusy(true);
break; // 己经找到一个可用的连接，退出
}
}
return client; // 返回找到到的可用连接
}
public void returnConnection(MilvusClient client) {
// 确保连接池存在，假如连接没有创建（不存在），直接返回
if (connections == null) {
System.out.println(" 连接池不存在，无法返回此连接到连接池中 !");
return;
}
PooledConnection pConn = null;
Enumeration enumerate = connections.elements();
// 遍历连接池中的所有连接，找到这个要返回的连接对象
while (enumerate.hasMoreElements()) {
pConn = (PooledConnection) enumerate.nextElement();
// 先找到连接池中的要返回的连接对象
if (client == pConn.getConnection()) {
// 找到了 , 设置此连接为空闲状态
pConn.setBusy(false);
break;
}
}
}
public synchronized void refreshConnections() {
// 确保连接池己创新存在
if (connections == null) {
System.out.println(" 连接池不存在，无法刷新 !");
return;
}
PooledConnection pConn = null;
Enumeration enumerate = connections.elements();
while (enumerate.hasMoreElements()) {
// 获得一个连接对象
pConn = (PooledConnection) enumerate.nextElement();
// 假如对象忙则等 5 秒 ,5 秒后直接刷新
if (pConn.isBusy()) {
wait(5000); // 等 5 秒
}
// 关闭此连接，用一个新的连接代替它。
closeConnection(pConn.getConnection());
pConn.setConnection(newConnection());
pConn.setBusy(false);
}
}
public synchronized void closeConnectionPool() {
// 确保连接池存在，假如不存在，返回
if (connections == null) {
System.out.println(" 连接池不存在，无法关闭 !");
return;
}
PooledConnection pConn = null;
Enumeration enumerate = connections.elements();
while (enumerate.hasMoreElements()) {
pConn = (PooledConnection) enumerate.nextElement();
// 假如忙，等 5 秒
if (pConn.isBusy()) {
wait(5000); // 等 5 秒
}
// 5 秒后直接关闭它
closeConnection(pConn.getConnection());
// 从连接池向量中删除它
connections.removeElement(pConn);
}
// 置连接池为空
connections = null;
}
private void closeConnection(MilvusClient client) {
client.close();
}
private void wait(int mSeconds) {
try {
Thread.sleep(mSeconds);
} catch (InterruptedException e) {
}
}
class PooledConnection {
MilvusClient client = null; // Milvus连接
boolean busy = false; // 此连接是否正在使用的标志，默认没有正在使用
// 构造函数，根据一个 Connection 构告一个 PooledConnection 对象
public PooledConnection(MilvusClient client) {
this.client = client;
}
// 返回此对象中的连接
public MilvusClient getConnection() {
return client;
}
// 设置此对象的连接
public void setConnection(MilvusClient client) {
this.client = client;
}
// 获得对象连接是否忙
public boolean isBusy() {
return busy;
}
// 设置对象的连接正在忙
public void setBusy(boolean busy) {
this.busy = busy;
}
}
}

37
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/common/milvus/MilvusConnector.java
View File

@ -1,37 +0,0 @@
package me.aias.common.milvus;
import io.milvus.client.*;
import lombok.extern.slf4j.Slf4j;
/**
* Milvus Client Connector
*
* @author Calvin
* @date 2021-12-19
**/
@Slf4j
public final class MilvusConnector {
// 创建一个 Milvus 客户端
private MilvusClient client = null;
public void init(String host, int port) {
client = new MilvusGrpcClient();
// Connect to Milvus server
ConnectParam connectParam = new ConnectParam.Builder().withHost(host).withPort(port).build();
try {
Response connectResponse = client.connect(connectParam);
} catch (ConnectFailedException e) {
log.error("Failed to connect to Milvus server: " + e.toString());
}
}
public void close() throws InterruptedException {
this.client.disconnect();
}
public MilvusClient getClient() {
return client;
}
}

26
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/config/MilvusConfiguration.java
View File

@ -1,26 +0,0 @@
package me.aias.config;
import me.aias.common.milvus.MilvusConnector;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
/**
* Milvus配置类
* @author Calvin
* @date 2021-12-12
**/
@Configuration
public class MilvusConfiguration {
@Value("${search.host}")
private String host;
@Value("${search.port}")
private int port;
@Bean
public MilvusConnector milvusConnector() {
MilvusConnector milvus = new MilvusConnector();
milvus.init(host, port);
return milvus;
}
}

2
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/config/ModelConfiguration.java
View File

@ -14,7 +14,7 @@ import org.springframework.context.annotation.Configuration;
@Configuration
public class ModelConfiguration {
// 设定词汇表的最大量为768
@Value("${search.dimension}")
@Value("${search.size}")
private int size;
@Bean

41
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/controller/DNASeqController.java
View File

@ -1,7 +1,6 @@
package me.aias.controller;
import io.milvus.client.ConnectFailedException;
import io.milvus.client.HasCollectionResponse;
import io.milvus.param.R;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import lombok.RequiredArgsConstructor;
@ -10,13 +9,8 @@ import me.aias.common.sentence.VectorizerModel;
import me.aias.common.utils.DataUtils;
import me.aias.common.utils.FeatureUtils;
import me.aias.common.utils.FileUtils;
import me.aias.config.FileProperties;
import me.aias.domain.DNAInfoDto;
import me.aias.domain.LocalStorage;
import me.aias.domain.ResEnum;
import me.aias.domain.ResultRes;
import me.aias.domain.*;
import me.aias.service.DNAService;
import me.aias.service.FeatureService;
import me.aias.service.LocalStorageService;
import me.aias.service.SearchService;
import org.apache.spark.ml.linalg.DenseVector;
@ -52,7 +46,6 @@ import java.util.concurrent.ConcurrentHashMap;
@Api(tags = "DNA数据管理")
@RequestMapping("/api/text")
public class DNASeqController {
private final FileProperties properties;
@Autowired
private VectorizerModel vectorizerModel;
@ -62,18 +55,9 @@ public class DNASeqController {
@Autowired
private SearchService searchService;
@Autowired
private FeatureService featureService;
@Autowired
private LocalStorageService localStorageService;
@Value("${search.dimension}")
int dimension;
@Value("${search.collectionName}")
String collectionName;
@ApiOperation(value = "提取DNA特征值")
@GetMapping("/extractFeatures")
public ResponseEntity<Object> extractFeatures(@RequestParam(value = "id") String id) {
@ -107,6 +91,7 @@ public class DNASeqController {
// 解析DNA信息
ConcurrentHashMap<Long, DNAInfoDto> map = textService.getMap();
long size = map.size();
int dimension = 0;
for (int i = 0; i < rowList.size(); i++) {
textInfoDto = new DNAInfoDto();
String label = rowList.get(i).getString(0);
@ -129,21 +114,25 @@ public class DNASeqController {
// 将向量插入向量引擎
try {
HasCollectionResponse response = searchService.hasCollection(this.collectionName);
if (!response.hasCollection()) {
searchService.createCollection(this.collectionName, dimension);
searchService.createIndex(this.collectionName);
R<Boolean> response = searchService.hasCollection();
if (!response.getData()) {
searchService.initSearchEngine(dimension);
}
searchService.insertVectors(this.collectionName, list);
List<Long> vectorIds = new ArrayList<>();
List<List<Float>> vectors = new ArrayList<>();
for (DNAInfoDto textInfo : list) {
vectorIds.add(textInfo.getId());
vectors.add(textInfo.getFeature());
}
searchService.insert(vectorIds, vectors);
textService.addTexts(list);
} catch (ConnectFailedException e) {
} catch (Exception e) {
e.printStackTrace();
log.error(e.getMessage());
return new ResponseEntity<>(ResultRes.error(ResEnum.MILVUS_CONNECTION_ERROR.KEY, ResEnum.MILVUS_CONNECTION_ERROR.VALUE), HttpStatus.OK);
}
return new ResponseEntity<>(ResultRes.success(), HttpStatus.OK);
return new ResponseEntity<>(ResultBean.success(), HttpStatus.OK);
}
}

51
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/controller/SearchController.java
View File

@ -1,8 +1,9 @@
package me.aias.controller;
import com.google.common.collect.Lists;
import io.milvus.client.ConnectFailedException;
import io.milvus.client.SearchResponse;
import io.milvus.Response.SearchResultsWrapper;
import io.milvus.grpc.SearchResults;
import io.milvus.param.R;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import lombok.RequiredArgsConstructor;
@ -49,9 +50,6 @@ public class SearchController {
@Autowired
private FeatureService featureService;
@Value("${search.collectionName}")
String collectionName;
@GetMapping("/sequence")
@ApiOperation(value = "DNA序列搜索", nickname = "search")
public ResultBean search(@RequestParam("sequence") String sequence, @RequestParam(value = "topK") String topk) {
@ -63,8 +61,8 @@ public class SearchController {
new StructField("kmers", new ArrayType(DataTypes.StringType, false), false, Metadata.empty())
});
Long topK = Long.parseLong(topk);
List<Float> vectorToSearch = null;
Integer topK = Integer.parseInt(topk);
List<Float> vectorToSearch;
try {
//获取数据 DataFrames
List<Row> rawData = DataUtils.getRawData("", sequence);
@ -81,50 +79,27 @@ public class SearchController {
try {
// 根据向量搜索
SearchResponse searchResponse = searchService.search(this.collectionName, topK, vectorsToSearch);
List<List<Long>> resultIds = searchResponse.getResultIdsList();
if (resultIds == null || resultIds.size() == 0) {
return ResultBean.failure().add(ResEnum.INFO_NOT_FOUND.KEY, ResEnum.INFO_NOT_FOUND.VALUE);
}
List<String> idList = Lists.transform(resultIds.get(0), (entity) -> {
return entity.toString();
});
R<SearchResults> searchResponse = searchService.search(topK, vectorsToSearch);
SearchResultsWrapper wrapper = new SearchResultsWrapper(searchResponse.getData().getResults());
List<SearchResultsWrapper.IDScore> scores = wrapper.getIDScore(0);
// 根据ID获取文本信息
ConcurrentHashMap<Long, DNAInfoDto> map = textService.getMap();
List<DNAInfoRes> textInfoResList = new ArrayList<>();
for (String uid : idList) {
Long id = Long.parseLong(uid);
DNAInfoDto dnaInfoDto = map.get(id);
for (SearchResultsWrapper.IDScore score : scores) {
DNAInfoDto dnaInfoDto = map.get(score.getLongID());
DNAInfoRes textInfoRes = new DNAInfoRes();
Float score = maxScoreForTextId(searchResponse, id);
textInfoRes.setId(id);
textInfoRes.setScore(score);
textInfoRes.setId(score.getLongID());
textInfoRes.setScore(score.getScore());
textInfoRes.setLabel(dnaInfoDto.getLabel());
textInfoRes.setSequence(dnaInfoDto.getSequence());
textInfoResList.add(textInfoRes);
}
return ResultBean.success().add("result", textInfoResList);
// return new ResponseEntity<>(ResultRes.success(textInfoResList, textInfoResList.size()), HttpStatus.OK);
} catch (ConnectFailedException e) {
} catch (Exception e) {
e.printStackTrace();
log.error(e.getMessage());
return ResultBean.failure().add(ResEnum.MILVUS_CONNECTION_ERROR.KEY, ResEnum.MILVUS_CONNECTION_ERROR.VALUE);
}
}
private Float maxScoreForTextId(SearchResponse searchResponse, Long id) {
float maxScore = -1;
List<SearchResponse.QueryResult> list = searchResponse.getQueryResultsList().get(0);
for (SearchResponse.QueryResult result : list) {
if (result.getVectorId() == id.longValue()) {
if (result.getDistance() > maxScore) {
maxScore = result.getDistance();
}
}
}
return maxScore;
}
}

103
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/service/SearchService.java
View File

@ -1,80 +1,71 @@
package me.aias.service;
import io.milvus.client.*;
import me.aias.domain.DNAInfoDto;
import io.milvus.client.MilvusClient;
import io.milvus.grpc.MutationResult;
import io.milvus.grpc.QueryResults;
import io.milvus.grpc.SearchResults;
import io.milvus.param.R;
import io.milvus.param.RpcStatus;
import me.aias.common.milvus.ConnectionPool;
import java.util.List;
/**
* 搜索服务接口
*
* @author Calvin
* @date 2021-12-19
**/
public interface SearchService {
// 引擎初始化
void initSearchEngine() throws ConnectFailedException;
// 重置向量引擎
void clearSearchEngine();
// 初始化向量引擎
void initSearchEngine(Integer dimension);
// 获取连接池
ConnectionPool getConnectionPool(boolean refresh);
// 获取Milvus Client
MilvusClient getClient(ConnectionPool connPool);
// 检查是否存在 collection
HasCollectionResponse hasCollection(MilvusClient client, String collectionName);
void returnConnection(ConnectionPool connPool, MilvusClient client);
HasCollectionResponse hasCollection(String collectionName) throws ConnectFailedException;
// 检查是否存在 collection
R<Boolean> hasCollection(MilvusClient milvusClient);
R<Boolean> hasCollection();
// 创建 collection
Response createCollection(
MilvusClient client, String collectionName, long dimension, long indexFileSize);
R<RpcStatus> createCollection(MilvusClient milvusClient, Integer dimension, long timeoutMiliseconds) ;
Response createCollection(String collectionName, long dimension) throws ConnectFailedException;
// 加载 collection
R<RpcStatus> loadCollection(MilvusClient milvusClient);
// 释放 collection
R<RpcStatus> releaseCollection(MilvusClient milvusClient);
// 删除 collection
Response dropCollection(MilvusClient client, String collectionName);
R<RpcStatus> dropCollection(MilvusClient milvusClient);
// 查看 collection 信息
Response getCollectionStats(MilvusClient client, String collectionName);
// 创建 分区
R<RpcStatus> createPartition(MilvusClient milvusClient, String partitionName);
void insertVectors(String collectionName, Long id, List<Float> feature) throws ConnectFailedException;
// 删除 分区
R<RpcStatus> dropPartition(MilvusClient milvusClient, String partitionName);
void insertVectors(String collectionName, List<Long> vectorIds, List<List<Float>> vectors) throws ConnectFailedException;
void insertVectors(String collectionName, List<DNAInfoDto> list) throws ConnectFailedException;
InsertResponse insertVectors(
MilvusClient client, String collectionName, List<Long> vectorIds, List<List<Float>> vectors);
// 查询向量数量
long count(MilvusClient client, String collectionName);
// 根据ID获取向量
GetEntityByIDResponse getEntityByID(
MilvusClient client, String collectionName, List<Long> vectorIds);
// 搜索向量
SearchResponse search(String collectionName, long topK, List<List<Float>> vectorsToSearch) throws ConnectFailedException;
SearchResponse search(
MilvusClient client,
String collectionName,
int nprobe,
long topK,
List<List<Float>> vectorsToSearch);
// 删除向量
Response deleteVectorsByIds(MilvusClient client, String collectionName, List<Long> vectorIds);
// 是否存在分区
R<Boolean> hasPartition(MilvusClient milvusClient, String partitionName);
// 创建 index
Response createIndex(MilvusClient client, String collectionName);
Response createIndex(String collectionName) throws ConnectFailedException;
// 查看索引信息
GetIndexInfoResponse getIndexInfo(MilvusClient client, String collectionName);
R<RpcStatus> createIndex(MilvusClient client);
// 删除 index
Response dropIndex(MilvusClient client, String collectionName);
R<RpcStatus> dropIndex(MilvusClient client);
// 压缩 collection
Response compactCollection(MilvusClient client, String collectionName);
// 插入向量
R<MutationResult> insert(List<Long> vectorIds, List<List<Float>> vectors);
// 检查 collection 中是否有 partition "tag"
HasPartitionResponse hasPartition(MilvusClient client, String collectionName, String tag);
// 查询向量
R<QueryResults> query(String expr);
// 搜索向量
R<SearchResults> search(Integer topK, List<List<Float>> vectorsToSearch);
// 删除向量
R<MutationResult> delete(String expr);
}

560
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/service/impl/SearchServiceImpl.java
View File

@ -1,24 +1,37 @@
package me.aias.service.impl;
import com.google.gson.JsonObject;
import io.milvus.client.*;
import io.milvus.Response.QueryResultsWrapper;
import io.milvus.Response.SearchResultsWrapper;
import io.milvus.client.MilvusClient;
import io.milvus.grpc.DataType;
import io.milvus.grpc.MutationResult;
import io.milvus.grpc.QueryResults;
import io.milvus.grpc.SearchResults;
import io.milvus.param.IndexType;
import io.milvus.param.MetricType;
import io.milvus.param.R;
import io.milvus.param.RpcStatus;
import io.milvus.param.collection.*;
import io.milvus.param.dml.DeleteParam;
import io.milvus.param.dml.InsertParam;
import io.milvus.param.dml.QueryParam;
import io.milvus.param.dml.SearchParam;
import io.milvus.param.index.CreateIndexParam;
import io.milvus.param.index.DropIndexParam;
import io.milvus.param.partition.CreatePartitionParam;
import io.milvus.param.partition.DropPartitionParam;
import io.milvus.param.partition.HasPartitionParam;
import lombok.extern.slf4j.Slf4j;
import me.aias.common.milvus.MilvusConnector;
import me.aias.domain.DNAInfoDto;
import me.aias.common.milvus.ConnectionPool;
import me.aias.service.SearchService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.TimeUnit;
/**
* 搜素服务
*
* @author Calvin
* @date 2021-12-19
**/
@Slf4j
@Service
public class SearchServiceImpl implements SearchService {
@ -26,16 +39,19 @@ public class SearchServiceImpl implements SearchService {
String host;
@Value("${search.port}")
String port;
String port;
@Value("${search.dimension}")
String dimension;
@Value("${search.indexType}")
String indexType;
@Value("${search.metricType}")
String metricType;
@Value("${search.collectionName}")
String collectionName;
@Value("${search.indexFileSize}")
String indexFileSize;
@Value("${search.partitionName}")
String partitionName;
@Value("${search.nprobe}")
String nprobe;
@ -43,246 +59,344 @@ public class SearchServiceImpl implements SearchService {
@Value("${search.nlist}")
String nlist;
@Autowired
private MilvusConnector milvusConnector;
private static final String ID_FIELD = "id";
private static final String VECTOR_FIELD = "feature";
public void initSearchEngine() {
MilvusClient client = milvusConnector.getClient();
// 检查 collection 是否存在
HasCollectionResponse hasCollection = this.hasCollection(client, collectionName);
if (hasCollection.hasCollection()) {
this.dropCollection(client, collectionName);
this.dropIndex(client, collectionName);
// 重置向量引擎
public void clearSearchEngine() {
ConnectionPool connPool = this.getConnectionPool(false);
MilvusClient client = connPool.getConnection();
try {
this.releaseCollection(client);
this.dropPartition(client, partitionName);
this.dropIndex(client);
this.dropCollection(client);
} finally {
returnConnection(connPool, client);
}
}
// 检查是否存在 collection
public HasCollectionResponse hasCollection(MilvusClient client, String collectionName) {
HasCollectionResponse response = client.hasCollection(collectionName);
return response;
// 初始化向量引擎
public void initSearchEngine(Integer dimension) {
ConnectionPool connPool = this.getConnectionPool(false);
MilvusClient client = connPool.getConnection();
try {
this.createCollection(client, dimension, 2000);
this.loadCollection(client);
this.createPartition(client, partitionName);
this.createIndex(client);
} finally {
returnConnection(connPool, client);
}
}
public ConnectionPool getConnectionPool(boolean refresh) {
ConnectionPool connPool = ConnectionPool.getInstance(host, port, refresh);
return connPool;
}
public MilvusClient getClient(ConnectionPool connPool) {
MilvusClient client = connPool.getConnection();
return client;
}
public void returnConnection(ConnectionPool connPool, MilvusClient client) {
// 释放 Milvus client 回连接池
connPool.returnConnection(client);
// 关闭 Milvus 连接池
// connPool.closeConnectionPool();
}
// 检查是否存在 collection
public HasCollectionResponse hasCollection(String collectionName) {
MilvusClient client = milvusConnector.getClient();
HasCollectionResponse response = client.hasCollection(collectionName);
public R<Boolean> hasCollection(MilvusClient client) {
R<Boolean> response = client.hasCollection(HasCollectionParam.newBuilder()
.withCollectionName(collectionName)
.build());
return response;
}
public R<Boolean> hasCollection() {
ConnectionPool connPool = this.getConnectionPool(false);
MilvusClient client = connPool.getConnection();
try {
R<Boolean> response = hasCollection(client);
return response;
} finally {
returnConnection(connPool, client);
}
}
// 创建 collection
public Response createCollection(
MilvusClient client, String collectionName, long dimension, long indexFileSize) {
// 选择内积 IP (Inner Product) 作为距离计算方式 MetricType.IP
final MetricType metricType = MetricType.IP;
CollectionMapping collectionMapping =
new CollectionMapping.Builder(collectionName, dimension)
.withIndexFileSize(indexFileSize)
.withMetricType(metricType)
.build();
Response createCollectionResponse = client.createCollection(collectionMapping);
return createCollectionResponse;
public R<RpcStatus> createCollection(MilvusClient milvusClient, Integer dimension, long timeoutMiliseconds) {
System.out.println("========== createCollection() ==========");
FieldType fieldType1 = FieldType.newBuilder()
.withName(ID_FIELD)
.withDescription("id")
.withDataType(DataType.Int64)
.withPrimaryKey(true)
.withAutoID(false) // 使用数据库生成的id
.build();
FieldType fieldType2 = FieldType.newBuilder()
.withName(VECTOR_FIELD)
.withDescription("embedding")
.withDataType(DataType.FloatVector)
.withDimension(dimension)
.build();
CreateCollectionParam createCollectionReq = CreateCollectionParam.newBuilder()
.withCollectionName(collectionName)
.withDescription("info")
.withShardsNum(2)
.addFieldType(fieldType1)
.addFieldType(fieldType2)
.build();
R<RpcStatus> response = milvusClient.withTimeout(timeoutMiliseconds, TimeUnit.MILLISECONDS)
.createCollection(createCollectionReq);
return response;
}
public Response createCollection(String collectionName, long dimension) {
// 选择内积 IP (Inner Product) 作为距离计算方式 MetricType.IP
final MetricType metricType = MetricType.IP;
CollectionMapping collectionMapping =
new CollectionMapping.Builder(collectionName, dimension)
.withIndexFileSize(Long.parseLong(indexFileSize))
.withMetricType(metricType)
.build();
MilvusClient client = milvusConnector.getClient();
Response createCollectionResponse = client.createCollection(collectionMapping);
return createCollectionResponse;
// 加载 collection
public R<RpcStatus> loadCollection(MilvusClient milvusClient) {
System.out.println("========== loadCollection() ==========");
R<RpcStatus> response = milvusClient.loadCollection(LoadCollectionParam.newBuilder()
.withCollectionName(collectionName)
.build());
return response;
}
// 释放 collection
public R<RpcStatus> releaseCollection(MilvusClient milvusClient) {
System.out.println("========== releaseCollection() ==========");
R<RpcStatus> response = milvusClient.releaseCollection(ReleaseCollectionParam.newBuilder()
.withCollectionName(collectionName)
.build());
return response;
}
// 删除 collection
public Response dropCollection(MilvusClient client, String collectionName) {
// Drop collection
Response dropCollectionResponse = client.dropCollection(collectionName);
return dropCollectionResponse;
public R<RpcStatus> dropCollection(MilvusClient milvusClient) {
System.out.println("========== dropCollection() ==========");
R<RpcStatus> response = milvusClient.dropCollection(DropCollectionParam.newBuilder()
.withCollectionName(collectionName)
.build());
return response;
}
// 查看 collection 信息
public Response getCollectionStats(MilvusClient client, String collectionName) {
Response getCollectionStatsResponse = client.getCollectionStats(collectionName);
// if (getCollectionStatsResponse.ok()) {
// // JSON 格式 collection 信息
// String jsonString = getCollectionStatsResponse.getMessage();
// System.out.format("Collection 信息: %s\n", jsonString);
// }
return getCollectionStatsResponse;
// 创建 分区
public R<RpcStatus> createPartition(MilvusClient milvusClient, String partitionName) {
System.out.println("========== createPartition() ==========");
R<RpcStatus> response = milvusClient.createPartition(CreatePartitionParam.newBuilder()
.withCollectionName(collectionName)
.withPartitionName(partitionName)
.build());
return response;
}
// 插入向量
public void insertVectors(String collectionName, List<DNAInfoDto> list) {
MilvusClient client = milvusConnector.getClient();
List<Long> vectorIds = new ArrayList<>();
List<List<Float>> vectors = new ArrayList<>();
for (DNAInfoDto textInfo : list) {
vectorIds.add(textInfo.getId());
vectors.add(textInfo.getFeature());
}
this.insertVectors(client, collectionName, vectorIds, vectors);
// 删除 分区
public R<RpcStatus> dropPartition(MilvusClient milvusClient, String partitionName) {
System.out.println("========== dropPartition() ==========");
R<RpcStatus> response = milvusClient.dropPartition(DropPartitionParam.newBuilder()
.withCollectionName(collectionName)
.withPartitionName(partitionName)
.build());
return response;
}
// 插入向量
public void insertVectors(String collectionName, Long id, List<Float> feature) {
MilvusClient client = milvusConnector.getClient();
List<Long> vectorIds = new ArrayList<>();
List<List<Float>> vectors = new ArrayList<>();
vectorIds.add(id);
vectors.add(feature);
this.insertVectors(client, collectionName, vectorIds, vectors);
}
// 插入向量
public void insertVectors(String collectionName, List<Long> vectorIds, List<List<Float>> vectors) {
MilvusClient client = milvusConnector.getClient();
this.insertVectors(client, collectionName, vectorIds, vectors);
}
public InsertResponse insertVectors(
MilvusClient client, String collectionName, List<Long> vectorIds, List<List<Float>> vectors) {
// 需要主动指定ID，如：图片的ID，用来关联图片资源，页面显示使用等
InsertParam insertParam =
new InsertParam.Builder(collectionName)
.withVectorIds(vectorIds)
.withFloatVectors(vectors)
.build();
InsertResponse insertResponse = client.insert(insertParam);
// 返回向量ID列表，向量ID如果不主动赋值，系统自动生成并返回
// List<Long> vectorIds = insertResponse.getVectorIds();
return insertResponse;
}
// 刷新数据
public Response flushData(MilvusClient client, String collectionName) {
// Flush data in collection
Response flushResponse = client.flush(collectionName);
return flushResponse;
}
// 查询向量数量
public long count(MilvusClient client, String collectionName) {
// 获取数据条数
CountEntitiesResponse ountEntitiesResponse = client.countEntities(collectionName);
long rows = ountEntitiesResponse.getCollectionEntityCount();
return rows;
}
// 根据ID获取向量
public GetEntityByIDResponse getEntityByID(
MilvusClient client, String collectionName, List<Long> vectorIds) {
GetEntityByIDResponse getEntityByIDResponse =
client.getEntityByID(collectionName, vectorIds.subList(0, 5));
return getEntityByIDResponse;
}
// 搜索向量
public SearchResponse search(String collectionName, long topK, List<List<Float>> vectorsToSearch) {
// 索引类型不同，参数也可能不同，查询文档选择最优参数
JsonObject searchParamsJson = new JsonObject();
searchParamsJson.addProperty("nprobe", Integer.parseInt(nprobe));
SearchParam searchParam =
new SearchParam.Builder(collectionName)
.withFloatVectors(vectorsToSearch)
.withTopK(topK)
.withParamsInJson(searchParamsJson.toString())
.build();
MilvusClient client = milvusConnector.getClient();
SearchResponse searchResponse = client.search(searchParam);
return searchResponse;
}
public SearchResponse search(
MilvusClient client,
String collectionName,
int nprobe,
long topK,
List<List<Float>> vectorsToSearch) {
// 索引类型不同，参数也可能不同，查询文档选择最优参数
JsonObject searchParamsJson = new JsonObject();
searchParamsJson.addProperty("nprobe", nprobe);
SearchParam searchParam =
new SearchParam.Builder(collectionName)
.withFloatVectors(vectorsToSearch)
.withTopK(topK)
.withParamsInJson(searchParamsJson.toString())
.build();
SearchResponse searchResponse = client.search(searchParam);
return searchResponse;
}
// 删除向量
public Response deleteVectorsByIds(
MilvusClient client, String collectionName, List<Long> vectorIds) {
Response deleteByIdsResponse = client.deleteEntityByID(collectionName, vectorIds);
// Flush, 使删除数据生效
Response flushResponse = client.flush(collectionName);
return deleteByIdsResponse;
// 是否存在分区
public R<Boolean> hasPartition(MilvusClient milvusClient, String partitionName) {
System.out.println("========== hasPartition() ==========");
R<Boolean> response = milvusClient.hasPartition(HasPartitionParam.newBuilder()
.withCollectionName(collectionName)
.withPartitionName(partitionName)
.build());
return response;
}
// 创建 index
public Response createIndex(MilvusClient client, String collectionName) {
// 索引类型在配置页面设置 IndexType. IVFLAT
final IndexType indexType = IndexType.IVF_SQ8;
// 每种索引有自己的可选参数 - 在配置页面设置
JsonObject indexParamsJson = new JsonObject();
indexParamsJson.addProperty("nlist", Integer.parseInt(nlist));
Index index =
new Index.Builder(collectionName, indexType)
.withParamsInJson(indexParamsJson.toString())
.build();
public R<RpcStatus> createIndex(MilvusClient milvusClient) {
System.out.println("========== createIndex() ==========");
String INDEX_PARAM = "{\"nlist\":" + nlist + "}";
Response createIndexResponse = client.createIndex(index);
return createIndexResponse;
}
IndexType INDEX_TYPE;
switch (indexType.toUpperCase()) {
case "IVF_FLAT":
INDEX_TYPE = IndexType.IVF_FLAT;
break;
case "IVF_SQ8":
INDEX_TYPE = IndexType.IVF_SQ8;
break;
case "IVF_PQ":
INDEX_TYPE = IndexType.IVF_PQ;
break;
case "HNSW":
INDEX_TYPE = IndexType.HNSW;
break;
case "ANNOY":
INDEX_TYPE = IndexType.ANNOY;
break;
case "RHNSW_FLAT":
INDEX_TYPE = IndexType.RHNSW_FLAT;
break;
case "RHNSW_PQ":
INDEX_TYPE = IndexType.RHNSW_PQ;
break;
case "RHNSW_SQ":
INDEX_TYPE = IndexType.RHNSW_SQ;
break;
case "BIN_IVF_FLAT":
INDEX_TYPE = IndexType.BIN_IVF_FLAT;
break;
default:
INDEX_TYPE = IndexType.IVF_FLAT;
break;
}
public Response createIndex(String collectionName) {
// 索引类型在配置页面设置 IndexType.IVF_SQ8 IVFLAT
final IndexType indexType = IndexType.IVF_SQ8;
// 每种索引有自己的可选参数 - 在配置页面设置
JsonObject indexParamsJson = new JsonObject();
indexParamsJson.addProperty("nlist", Integer.parseInt(nlist));
Index index =
new Index.Builder(collectionName, indexType)
.withParamsInJson(indexParamsJson.toString())
.build();
MilvusClient client = milvusConnector.getClient();
Response createIndexResponse = client.createIndex(index);
return createIndexResponse;
}
// 查看索引信息
public GetIndexInfoResponse getIndexInfo(MilvusClient client, String collectionName) {
GetIndexInfoResponse getIndexInfoResponse = client.getIndexInfo(collectionName);
// System.out.format("索引信息: %s\n",search.service.SearchServiceImpl.getIndexInfo(client,
// collectionName).getIndex().toString());
return getIndexInfoResponse;
MetricType METRIC_TYPE;
MetricType metricTypeEnum = MetricType.valueOf(metricType.toUpperCase());
switch (metricTypeEnum) {
case L2:
METRIC_TYPE = MetricType.L2;
break;
case IP:
METRIC_TYPE = MetricType.IP;
break;
case HAMMING:
METRIC_TYPE = MetricType.HAMMING;
break;
case JACCARD:
METRIC_TYPE = MetricType.JACCARD;
break;
case TANIMOTO:
METRIC_TYPE = MetricType.TANIMOTO;
break;
case SUBSTRUCTURE:
METRIC_TYPE = MetricType.SUBSTRUCTURE;
break;
case SUPERSTRUCTURE:
METRIC_TYPE = MetricType.SUPERSTRUCTURE;
break;
default:
METRIC_TYPE = MetricType.L2;
break;
}
R<RpcStatus> response = milvusClient.createIndex(CreateIndexParam.newBuilder()
.withCollectionName(collectionName)
.withFieldName(VECTOR_FIELD)
.withIndexType(INDEX_TYPE)
.withMetricType(METRIC_TYPE)
.withExtraParam(INDEX_PARAM)
.withSyncMode(Boolean.TRUE)
.build());
return response;
}
// 删除 index
public Response dropIndex(MilvusClient client, String collectionName) {
Response dropIndexResponse = client.dropIndex(collectionName);
return dropIndexResponse;
public R<RpcStatus> dropIndex(MilvusClient milvusClient) {
System.out.println("========== dropIndex() ==========");
R<RpcStatus> response = milvusClient.dropIndex(DropIndexParam.newBuilder()
.withCollectionName(collectionName)
.withFieldName(VECTOR_FIELD)
.build());
return response;
}
// 压缩 collection
public Response compactCollection(MilvusClient client, String collectionName) {
// 压缩 collection, 从磁盘抹除删除的数据，并在后台重建索引（如果压缩后的数据比indexFileSize还要大）
// 在主动压缩前，数据只是软删除
Response compactResponse = client.compact(collectionName);
return compactResponse;
// 插入向量
public R<MutationResult> insert(List<Long> vectorIds, List<List<Float>> vectors) {
System.out.println("========== insert() ==========");
ConnectionPool connPool = this.getConnectionPool(false);
MilvusClient milvusClient = connPool.getConnection();
try {
List<InsertParam.Field> fields = new ArrayList<>();
fields.add(new InsertParam.Field(ID_FIELD, DataType.Int64, vectorIds));
fields.add(new InsertParam.Field(VECTOR_FIELD, DataType.FloatVector, vectors));
InsertParam insertParam = InsertParam.newBuilder()
.withCollectionName(collectionName)
.withPartitionName(partitionName)
.withFields(fields)
.build();
R<MutationResult> response = milvusClient.insert(insertParam);
return response;
} finally {
returnConnection(connPool, milvusClient);
}
}
// 检查 collection 中是否有 partition "tag"
public HasPartitionResponse hasPartition(MilvusClient client, String collectionName, String tag) {
HasPartitionResponse hasPartitionResponse = client.hasPartition(collectionName, tag);
return hasPartitionResponse;
// 查询向量
// queryExpr = ID_FIELD + " == 60";
public R<QueryResults> query(String expr) {
System.out.println("========== query() ==========");
ConnectionPool connPool = this.getConnectionPool(false);
MilvusClient milvusClient = connPool.getConnection();
try {
List<String> fields = Arrays.asList(ID_FIELD);
QueryParam test = QueryParam.newBuilder()
.withCollectionName(collectionName)
.withExpr(expr)
.withOutFields(fields)
.build();
R<QueryResults> response = milvusClient.query(test);
QueryResultsWrapper wrapper = new QueryResultsWrapper(response.getData());
System.out.println(ID_FIELD + ":" + wrapper.getFieldWrapper(ID_FIELD).getFieldData().toString());
System.out.println("Query row count: " + wrapper.getFieldWrapper(ID_FIELD).getRowCount());
return response;
} finally {
returnConnection(connPool, milvusClient);
}
}
// 搜索向量
public R<SearchResults> search(Integer topK, List<List<Float>> vectorsToSearch) {
System.out.println("========== searchImage() ==========");
ConnectionPool connPool = this.getConnectionPool(false);
MilvusClient milvusClient = connPool.getConnection();
try {
String SEARCH_PARAM = "{\"nprobe\":" + nprobe + "}";
List<String> outFields = Arrays.asList(ID_FIELD);
SearchParam searchParam = SearchParam.newBuilder()
.withCollectionName(collectionName)
.withMetricType(MetricType.IP)
.withOutFields(outFields)
.withTopK(topK)
.withVectors(vectorsToSearch)
.withVectorFieldName(VECTOR_FIELD)
// .withExpr(expr)
.withParams(SEARCH_PARAM)
.build();
R<SearchResults> response = milvusClient.search(searchParam);
SearchResultsWrapper wrapper = new SearchResultsWrapper(response.getData().getResults());
for (int i = 0; i < vectorsToSearch.size(); ++i) {
System.out.println("Search result of No." + i);
List<SearchResultsWrapper.IDScore> scores = wrapper.getIDScore(i);
System.out.println(scores);
System.out.println("Output field data for No." + i);
}
return response;
} finally {
returnConnection(connPool, milvusClient);
}
}
// 删除向量
// String deleteExpr = ID_FIELD + " in " + deleteIds.toString();
public R<MutationResult> delete(String expr) {
System.out.println("========== delete() ==========");
ConnectionPool connPool = this.getConnectionPool(false);
MilvusClient milvusClient = connPool.getConnection();
try {
DeleteParam build = DeleteParam.newBuilder()
.withCollectionName(collectionName)
.withPartitionName(partitionName)
.withExpr(expr)
.build();
R<MutationResult> response = milvusClient.delete(build);
return response;
} finally {
returnConnection(connPool, milvusClient);
}
}
}

107
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/java/me/aias/tools/MilvusInit.java
View File

@ -1,6 +1,14 @@
package me.aias.tools;
import io.milvus.client.*;
import io.milvus.client.MilvusServiceClient;
import io.milvus.param.ConnectParam;
import io.milvus.param.R;
import io.milvus.param.RpcStatus;
import io.milvus.param.collection.DropCollectionParam;
import io.milvus.param.collection.HasCollectionParam;
import io.milvus.param.collection.ReleaseCollectionParam;
import io.milvus.param.index.DropIndexParam;
import io.milvus.param.partition.DropPartitionParam;
/**
* 搜索引擎初始化工具
@ -9,49 +17,92 @@ import io.milvus.client.*;
* @date 2021-12-12
**/
public class MilvusInit {
private static final MilvusServiceClient milvusClient;
public static void main(String[] args) throws InterruptedException {
static {
ConnectParam connectParam = ConnectParam.newBuilder()
.withHost("127.0.0.1")
.withPort(19530)
.build();
milvusClient = new MilvusServiceClient(connectParam);
}
String host = "127.0.0.1";
int port = 19530;
final String collectionName = "dna"; // collection name
private static final String COLLECTION_NAME = "dna";// collection name
private static final String VECTOR_FIELD = "feature";
MilvusClient client = new MilvusGrpcClient();
// Connect to Milvus server
ConnectParam connectParam = new ConnectParam.Builder().withHost(host).withPort(port).build();
public static void main(String[] args) {
try {
Response connectResponse = client.connect(connectParam);
} catch (ConnectFailedException e) {
// 检查 collection 是否存在，不存在会抛异常
hasCollection();
releaseCollection();
dropPartition("p1");
dropIndex();
dropCollection();
} catch (Exception e) {
e.printStackTrace();
}
// 检查 collection 是否存在
HasCollectionResponse hasCollection = hasCollection(client, collectionName);
if (hasCollection.hasCollection()) {
dropIndex(client, collectionName);
dropCollection(client, collectionName);
}
// 关闭 Milvus 连接
client.disconnect();
milvusClient.close();
}
// 检查是否存在 collection
public static HasCollectionResponse hasCollection(MilvusClient client, String collectionName) {
HasCollectionResponse response = client.hasCollection(collectionName);
private static R<Boolean> hasCollection() {
System.out.println("========== hasCollection() ==========");
R<Boolean> response = milvusClient.hasCollection(HasCollectionParam.newBuilder()
.withCollectionName(COLLECTION_NAME)
.build());
handleResponseStatus(response);
System.out.println(response);
return response;
}
// 删除 collection
public static Response dropCollection(MilvusClient client, String collectionName) {
// Drop collection
Response dropCollectionResponse = client.dropCollection(collectionName);
return dropCollectionResponse;
private static R<RpcStatus> dropCollection() {
System.out.println("========== dropCollection() ==========");
R<RpcStatus> response = milvusClient.dropCollection(DropCollectionParam.newBuilder()
.withCollectionName(COLLECTION_NAME)
.build());
System.out.println(response);
return response;
}
private static R<RpcStatus> releaseCollection() {
System.out.println("========== releaseCollection() ==========");
R<RpcStatus> response = milvusClient.releaseCollection(ReleaseCollectionParam.newBuilder()
.withCollectionName(COLLECTION_NAME)
.build());
handleResponseStatus(response);
System.out.println(response);
return response;
}
// 删除 index
public static Response dropIndex(MilvusClient client, String collectionName) {
Response dropIndexResponse = client.dropIndex(collectionName);
return dropIndexResponse;
private static R<RpcStatus> dropIndex() {
System.out.println("========== dropIndex() ==========");
R<RpcStatus> response = milvusClient.dropIndex(DropIndexParam.newBuilder()
.withCollectionName(COLLECTION_NAME)
.withFieldName(VECTOR_FIELD)
.build());
handleResponseStatus(response);
System.out.println(response);
return response;
}
private static R<RpcStatus> dropPartition(String partitionName) {
System.out.println("========== dropPartition() ==========");
R<RpcStatus> response = milvusClient.dropPartition(DropPartitionParam.newBuilder()
.withCollectionName(COLLECTION_NAME)
.withPartitionName(partitionName)
.build());
handleResponseStatus(response);
System.out.println(response);
return response;
}
private static void handleResponseStatus(R<?> r) {
if (r.getStatus() != R.Status.Success.getCode()) {
throw new RuntimeException(r.getMessage());
}
}
}

28
6_biomedicine/dna_sequence_search/dna_sequence_search/src/main/resources/application.yml
View File

@ -23,12 +23,30 @@ file:
# 文件大小 /M
maxSize: 3000
##################### 向量引擎 ###############################
# Vector Engine - Milvus
search:
host: 127.0.0.1
port: 19530
indexFileSize: 1024 # maximum size (in MB) of each index file
nprobe: 16
# https://milvus.io/cn/docs/v2.0.0/build_index.md
# https://milvus.io/cn/docs/v2.0.0/metric.md#floating
# For floating point vectors:
# IVF_FLAT
# IVF_SQ8
# IVF_PQ
# HNSW
# ANNOY
# RHNSW_FLAT
# RHNSW_PQ
# RHNSW_SQ
# For binary vectors:
# BIN_IVF_FLAT
indexType: IVF_FLAT
# 选择内积 IP (Inner Product) 作为距离计算方式 MetricType.IP
metricType: IP
nprobe: 256
nlist: 16384
dimension: 768 #dimension of each vector
collectionName: dna #collection name
size: 768 # 设定词汇表的最大量为768
collectionName: dna #collection name
partitionName: p1

207
6_biomedicine/molecular_search/molecular-search/molecular-search.iml
View File

@ -1,207 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
<component name="CheckStyle-IDEA-Module">
<option name="configuration">
<map />
</option>
</component>
<component name="FacetManager">
<facet type="Spring" name="Spring">
<configuration />
</facet>
<facet type="web" name="Web">
<configuration>
<webroots />
<sourceRoots>
<root url="file://$MODULE_DIR$/src/main/java" />
<root url="file://$MODULE_DIR$/src/main/resources" />
</sourceRoots>
</configuration>
</facet>
</component>
<component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
<output url="file://$MODULE_DIR$/target/classes" />
<output-test url="file://$MODULE_DIR$/target/test-classes" />
<content url="file://$MODULE_DIR$">
<sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
<sourceFolder url="file://$MODULE_DIR$/src/main/resources" type="java-resource" />
<sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
<sourceFolder url="file://$MODULE_DIR$/src/test/resources" type="java-test-resource" />
<excludeFolder url="file://$MODULE_DIR$/target" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-context:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-autoconfigure:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-logging:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: ch.qos.logback:logback-classic:1.2.3" level="project" />
<orderEntry type="library" name="Maven: ch.qos.logback:logback-core:1.2.3" level="project" />
<orderEntry type="library" name="Maven: org.apache.logging.log4j:log4j-to-slf4j:2.11.2" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:jul-to-slf4j:1.7.28" level="project" />
<orderEntry type="library" name="Maven: javax.annotation:javax.annotation-api:1.3.2" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-core:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-jcl:5.1.10.RELEASE" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: org.yaml:snakeyaml:1.23" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-web:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-json:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-databind:2.9.9.3" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-core:2.9.9" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.9" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.9" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.9" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-tomcat:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.apache.tomcat.embed:tomcat-embed-core:9.0.26" level="project" />
<orderEntry type="library" name="Maven: org.apache.tomcat.embed:tomcat-embed-el:9.0.26" level="project" />
<orderEntry type="library" name="Maven: org.apache.tomcat.embed:tomcat-embed-websocket:9.0.26" level="project" />
<orderEntry type="library" name="Maven: org.hibernate.validator:hibernate-validator:6.0.17.Final" level="project" />
<orderEntry type="library" name="Maven: javax.validation:validation-api:2.0.1.Final" level="project" />
<orderEntry type="library" name="Maven: org.jboss.logging:jboss-logging:3.3.3.Final" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-web:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-webmvc:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-expression:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.boot:spring-boot-starter-aop:2.1.9.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-aop:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.aspectj:aspectjweaver:1.9.4" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework.boot:spring-boot-starter-test:2.1.9.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework.boot:spring-boot-test:2.1.9.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework.boot:spring-boot-test-autoconfigure:2.1.9.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: com.jayway.jsonpath:json-path:2.4.0" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: net.minidev:json-smart:2.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: net.minidev:accessors-smart:1.2" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.ow2.asm:asm:5.0.4" level="project" />
<orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.assertj:assertj-core:3.11.1" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.mockito:mockito-core:2.23.4" level="project" />
<orderEntry type="library" name="Maven: net.bytebuddy:byte-buddy:1.9.16" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: net.bytebuddy:byte-buddy-agent:1.9.16" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.objenesis:objenesis:2.6" level="project" />
<orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.hamcrest:hamcrest-library:1.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.skyscreamer:jsonassert:1.5.0" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: com.vaadin.external.google:android-json:0.0.20131108.vaadin1" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.springframework:spring-test:5.1.10.RELEASE" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: org.xmlunit:xmlunit-core:2.6.3" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: javax.xml.bind:jaxb-api:2.3.1" level="project" />
<orderEntry type="library" scope="TEST" name="Maven: javax.activation:javax.activation-api:1.2.0" level="project" />
<orderEntry type="library" name="Maven: org.springframework.data:spring-data-commons:2.1.2.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework:spring-beans:5.1.10.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.28" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.8.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.logging.log4j:log4j-slf4j-impl:2.16.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.logging.log4j:log4j-api:2.11.2" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: org.apache.logging.log4j:log4j-core:2.11.2" level="project" />
<orderEntry type="library" name="Maven: com.google.code.gson:gson:2.8.5" level="project" />
<orderEntry type="library" name="Maven: io.milvus:milvus-sdk-java:1.1.0" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven.plugins:maven-gpg-plugin:1.6" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-plugin-api:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-project:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-settings:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-profile:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-artifact-manager:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven.wagon:wagon-provider-api:1.0-beta-6" level="project" />
<orderEntry type="library" name="Maven: backport-util-concurrent:backport-util-concurrent:3.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-plugin-registry:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.plexus:plexus-interpolation:1.11" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.plexus:plexus-container-default:1.0-alpha-9-stable-1" level="project" />
<orderEntry type="library" name="Maven: classworlds:classworlds:1.1-alpha-2" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-artifact:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-repository-metadata:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.apache.maven:maven-model:2.2.1" level="project" />
<orderEntry type="library" name="Maven: org.codehaus.plexus:plexus-utils:3.0.20" level="project" />
<orderEntry type="library" name="Maven: org.sonatype.plexus:plexus-sec-dispatcher:1.4" level="project" />
<orderEntry type="library" name="Maven: org.sonatype.plexus:plexus-cipher:1.4" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: io.grpc:grpc-netty-shaded:1.30.2" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: io.grpc:grpc-core:1.30.2" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: com.google.android:annotations:4.1.1.4" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: io.perfmark:perfmark-api:0.19.0" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-protobuf:1.30.2" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-api:1.30.2" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-context:1.30.2" level="project" />
<orderEntry type="library" name="Maven: com.google.protobuf:protobuf-java:3.12.0" level="project" />
<orderEntry type="library" name="Maven: com.google.api.grpc:proto-google-common-protos:1.17.0" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-protobuf-lite:1.30.2" level="project" />
<orderEntry type="library" scope="RUNTIME" name="Maven: org.codehaus.mojo:animal-sniffer-annotations:1.18" level="project" />
<orderEntry type="library" name="Maven: io.grpc:grpc-stub:1.30.2" level="project" />
<orderEntry type="library" name="Maven: com.google.protobuf:protobuf-java-util:3.11.0" level="project" />
<orderEntry type="library" name="Maven: com.google.errorprone:error_prone_annotations:2.3.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-text:1.6" level="project" />
<orderEntry type="library" name="Maven: org.apache.commons:commons-collections4:4.4" level="project" />
<orderEntry type="library" name="Maven: org.json:json:20190722" level="project" />
<orderEntry type="module-library">
<library name="Maven: rdkit:rdkit:1.0">
<CLASSES>
<root url="jar://$MODULE_DIR$/lib/org.RDKit.jar!/" />
</CLASSES>
<JAVADOC />
<SOURCES />
</library>
</orderEntry>
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-anim:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-shared-resources:1.14" level="project" />
<orderEntry type="library" name="Maven: xml-apis:xml-apis-ext:1.3.04" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-awt-util:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:xmlgraphics-commons:2.6" level="project" />
<orderEntry type="library" name="Maven: commons-io:commons-io:1.3.1" level="project" />
<orderEntry type="library" name="Maven: commons-logging:commons-logging:1.0.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-bridge:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-codec:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-constants:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-css:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-dom:1.14" level="project" />
<orderEntry type="library" name="Maven: xalan:xalan:2.7.2" level="project" />
<orderEntry type="library" name="Maven: xalan:serializer:2.7.2" level="project" />
<orderEntry type="library" name="Maven: xml-apis:xml-apis:1.4.01" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-ext:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-extension:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-gui-util:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-gvt:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-i18n:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-parser:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-rasterizer-ext:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-rasterizer:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-script:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-slideshow:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-squiggle-ext:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-squiggle:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-svg-dom:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-svgbrowser:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-svggen:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-svgpp:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-svgrasterizer:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-swing:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-transcoder:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-util:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-ttf2svg:1.14" level="project" />
<orderEntry type="library" name="Maven: org.apache.xmlgraphics:batik-xml:1.14" level="project" />
<orderEntry type="library" name="Maven: org.projectlombok:lombok:1.18.10" level="project" />
<orderEntry type="library" name="Maven: com.google.http-client:google-http-client:1.19.0" level="project" />
<orderEntry type="library" name="Maven: com.google.code.findbugs:jsr305:1.3.9" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.5.10" level="project" />
<orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4.12" level="project" />
<orderEntry type="library" name="Maven: commons-codec:commons-codec:1.11" level="project" />
<orderEntry type="library" name="Maven: commons-cli:commons-cli:1.4" level="project" />
<orderEntry type="library" name="Maven: com.alibaba:fastjson:1.2.70" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-swagger2:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-spi:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-core:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-schema:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-swagger-common:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-spring-web:2.9.2" level="project" />
<orderEntry type="library" name="Maven: com.google.guava:guava:20.0" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml:classmate:1.4.0" level="project" />
<orderEntry type="library" name="Maven: org.springframework.plugin:spring-plugin-core:1.2.0.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.springframework.plugin:spring-plugin-metadata:1.2.0.RELEASE" level="project" />
<orderEntry type="library" name="Maven: org.mapstruct:mapstruct:1.2.0.Final" level="project" />
<orderEntry type="library" name="Maven: io.springfox:springfox-swagger-ui:2.9.2" level="project" />
<orderEntry type="library" name="Maven: io.swagger:swagger-annotations:1.5.21" level="project" />
<orderEntry type="library" name="Maven: io.swagger:swagger-models:1.5.21" level="project" />
<orderEntry type="library" name="Maven: com.fasterxml.jackson.core:jackson-annotations:2.9.0" level="project" />
<orderEntry type="library" name="Maven: cn.hutool:hutool-all:5.3.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.ant:ant:1.9.4" level="project" />
<orderEntry type="library" name="Maven: org.apache.ant:ant-launcher:1.9.4" level="project" />
<orderEntry type="library" name="Maven: eu.bitwalker:UserAgentUtils:1.21" level="project" />
<orderEntry type="library" name="Maven: de.siegmar:fastcsv:1.0.1" level="project" />
</component>
</module>