mirror of
https://gitee.com/milvus-io/milvus.git
synced 2024-12-02 20:09:57 +08:00
Merge branch '0.5.1' into master
Former-commit-id: 4187de60052a91a0b20b28d362cf6a986572c74a
This commit is contained in:
Jin Hai
GitHub
commit
dd5d4a5e6a
27
CHANGELOG.md
27
CHANGELOG.md
@ -5,10 +5,33 @@ Please mark all change in change log and use the ticket from JIRA.
|
||||
# Milvus 0.5.1 (TODO)
|
||||
|
||||
## Bug
|
||||
## Improvement
|
||||
- \#64 - Improvement dump function in scheduler
|
||||
- \#134 - JFrog cache error
|
||||
- \#161 - Search IVFSQHybrid crash on gpu
|
||||
- \#169 - IVF_FLAT search out of memory
|
||||
|
||||
## Feature
|
||||
- \#90 - The server start error messages could be improved to enhance user experience
|
||||
- \#104 - test_scheduler core dump
|
||||
- \#115 - Using new structure for tasktable
|
||||
- \#139 - New config option use_gpu_threshold
|
||||
- \#146 - Add only GPU and only CPU version for IVF_SQ8 and IVF_FLAT
|
||||
- \#164 - Add CPU version for building index
|
||||
|
||||
## Improvement
|
||||
- \#64 - Improvement dump function in scheduler
|
||||
- \#80 - Print version information into log during server start
|
||||
- \#82 - Move easyloggingpp into "external" directory
|
||||
- \#92 - Speed up CMake build process
|
||||
- \#96 - Remove .a file in milvus/lib for docker-version
|
||||
- \#118 - Using shared_ptr instead of weak_ptr to avoid performance loss
|
||||
- \#122 - Add unique id for Job
|
||||
- \#130 - Set task state MOVED after resource copy it completed
|
||||
- \#149 - Improve large query optimizer pass
|
||||
- \#156 - Not return error when search_resources and index_build_device set cpu
|
||||
- \#159 - Change the configuration name from 'use_gpu_threshold' to 'gpu_search_threshold'
|
||||
- \#168 - Improve result reduce
|
||||
- \#175 - add invalid config unittest
|
||||
|
||||
## Task
|
||||
|
||||
# Milvus 0.5.0 (2019-10-21)
|
||||
|
||||
@ -28,7 +28,7 @@ Contributions to Milvus fall into the following categories.
|
||||
|
||||
If you have improvements to Milvus, send us your pull requests! For those just getting started, see [GitHub workflow](#github-workflow).
|
||||
|
||||
The Milvus team members will review your pull requests, and once it is accepted, it will be given a `ready to merge` label. This means we are working on submitting your pull request to the internal repository. After the change has been submitted internally, your pull request will be merged automatically on GitHub.
|
||||
The Milvus team members will review your pull requests, and once it is accepted, the status of the projects to which it is associated will be changed to **Reviewer approved**. This means we are working on submitting your pull request to the internal repository. After the change has been submitted internally, your pull request will be merged automatically on GitHub.
|
||||
|
||||
### GitHub workflow
|
||||
|
||||
|
||||
@ -25,6 +25,6 @@
|
||||
| libunwind | [MIT](https://github.com/libunwind/libunwind/blob/master/LICENSE) |
|
||||
| gperftools | [BSD 3-Clause](https://github.com/gperftools/gperftools/blob/master/COPYING) |
|
||||
| grpc | [Apache 2.0](https://github.com/grpc/grpc/blob/master/LICENSE) |
|
||||
| EASYLOGGINGPP | [MIT](https://github.com/zuhd-org/easyloggingpp/blob/master/LICENSEhttps://github.com/zuhd-org/easyloggingpp/blob/master/LICENSE) |
|
||||
| EASYLOGGINGPP | [MIT](https://github.com/zuhd-org/easyloggingpp/blob/master/LICENSE) |
|
||||
| Json | [MIT](https://github.com/nlohmann/json/blob/develop/LICENSE.MIT) |
|
||||
|
||||
|
||||
179
README.md
179
README.md
@ -1,158 +1,36 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
[](https://codebeat.co/projects/github-com-jinhai-cn-milvus-master)
|
||||
|
||||
|
||||
|
||||
- [Slack Community](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)
|
||||
- [Twitter](https://twitter.com/milvusio)
|
||||
- [Facebook](https://www.facebook.com/io.milvus.5)
|
||||
- [Blog](https://www.milvus.io/blog/)
|
||||
- [CSDN](https://zilliz.blog.csdn.net/)
|
||||
- [中文官网](https://www.milvus.io/zh-CN/)
|
||||
|
||||
|
||||
# Welcome to Milvus
|
||||
[中文版](README_CN.md)
|
||||
|
||||
## What is Milvus
|
||||
|
||||
Milvus is the world's fastest similarity search engine for massive feature vectors. Designed with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
|
||||
Milvus is the world's fastest similarity search engine for massive feature vectors. Built with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.
|
||||
|
||||
Milvus provides stable Python, Java and C++ APIs.
|
||||
For more detailed introduction of Milvus and its architecture, see [Milvus overview](https://www.milvus.io/docs/en/aboutmilvus/overview/).
|
||||
|
||||
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/releases/v0.5.0/).
|
||||
Milvus provides stable [Python](https://pypi.org/project/pymilvus/), [Java](https://milvus-io.github.io/milvus-sdk-java/javadoc/io/milvus/client/package-summary.html) and C++ APIs.
|
||||
|
||||
- Heterogeneous computing
|
||||
|
||||
Milvus is designed with heterogeneous computing architecture for the best performance and cost efficiency.
|
||||
|
||||
- Multiple indexes
|
||||
|
||||
Milvus supports a variety of indexing types that employs quantization, tree-based, and graph indexing techniques.
|
||||
|
||||
- Intelligent resource management
|
||||
|
||||
Milvus automatically adapts search computation and index building processes based on your datasets and available resources.
|
||||
|
||||
- Horizontal scalability
|
||||
|
||||
Milvus supports online / offline expansion to scale both storage and computation resources with simple commands.
|
||||
|
||||
- High availability
|
||||
|
||||
Milvus is integrated with Kubernetes framework so that all single point of failures could be avoided.
|
||||
|
||||
- High compatibility
|
||||
|
||||
Milvus is compatible with almost all deep learning models and major programming languages such as Python, Java and C++, etc.
|
||||
|
||||
- Ease of use
|
||||
|
||||
Milvus can be easily installed in a few steps and enables you to exclusively focus on feature vectors.
|
||||
|
||||
- Visualized monitor
|
||||
|
||||
You can track system performance on Prometheus-based GUI monitor dashboards.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://www.milvus.io/docs/en/release/v0.5.0/).
|
||||
|
||||
## Get started
|
||||
|
||||
### Hardware Requirements
|
||||
See the [Milvus install guide](https://www.milvus.io/docs/en/userguide/install_milvus/) for using Docker containers. To install Milvus from source code, see [build from source](install.md).
|
||||
|
||||
| Component | Recommended configuration |
|
||||
| --------- | ----------------------------------- |
|
||||
| CPU | Intel CPU Haswell or higher |
|
||||
| GPU | NVIDIA Pascal series or higher |
|
||||
| Memory | 8 GB or more (depends on data size) |
|
||||
| Storage | SATA 3.0 SSD or higher |
|
||||
|
||||
### Install using docker
|
||||
|
||||
Use Docker to install Milvus is a breeze. See the [Milvus install guide](https://milvus.io/docs/en/userguide/install_milvus/) for details.
|
||||
|
||||
### Build from source
|
||||
|
||||
#### Software requirements
|
||||
|
||||
- Ubuntu 18.04 or higher
|
||||
- CMake 3.14 or higher
|
||||
- CUDA 10.0 or higher
|
||||
- NVIDIA driver 418 or higher
|
||||
|
||||
#### Compilation
|
||||
|
||||
##### Step 1 Install dependencies
|
||||
|
||||
```shell
|
||||
$ cd [Milvus sourcecode path]/core
|
||||
./ubuntu_build_deps.sh
|
||||
```
|
||||
|
||||
##### Step 2 Build
|
||||
|
||||
```shell
|
||||
$ cd [Milvus sourcecode path]/core
|
||||
$ ./build.sh -t Debug
|
||||
or
|
||||
$ ./build.sh -t Release
|
||||
```
|
||||
|
||||
When the build is completed, all the stuff that you need in order to run Milvus will be installed under `[Milvus root path]/core/milvus`.
|
||||
|
||||
#### Launch Milvus server
|
||||
|
||||
```shell
|
||||
$ cd [Milvus root path]/core/milvus
|
||||
```
|
||||
|
||||
Add `lib/` directory to `LD_LIBRARY_PATH`
|
||||
|
||||
```
|
||||
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/milvus/lib
|
||||
```
|
||||
|
||||
Then start Milvus server:
|
||||
|
||||
```
|
||||
$ cd scripts
|
||||
$ ./start_server.sh
|
||||
```
|
||||
|
||||
To stop Milvus server, run:
|
||||
|
||||
```shell
|
||||
$ ./stop_server.sh
|
||||
```
|
||||
|
||||
To edit Milvus settings in `conf/server_config.yaml` and `conf/log_config.conf`, please read [Milvus Configuration](https://github.com/milvus-io/docs/blob/master/reference/milvus_config.md).
|
||||
To edit Milvus settings, read [Milvus configuration](https://www.milvus.io/docs/en/reference/milvus_config/).
|
||||
|
||||
### Try your first Milvus program
|
||||
|
||||
#### Run Python example code
|
||||
Try running a program with Milvus using [Python](https://www.milvus.io/docs/en/userguide/example_code/) or [Java example code](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples).
|
||||
|
||||
Make sure [Python 3.5](https://www.python.org/downloads/) or higher is already installed and in use.
|
||||
|
||||
Install Milvus Python SDK.
|
||||
|
||||
```shell
|
||||
# Install Milvus Python SDK
|
||||
$ pip install pymilvus==0.2.3
|
||||
```
|
||||
|
||||
Create a new file `example.py`, and add [Python example code](https://github.com/milvus-io/pymilvus/blob/master/examples/AdvancedExample.py) to it.
|
||||
|
||||
Run the example code.
|
||||
|
||||
```shell
|
||||
# Run Milvus Python example
|
||||
$ python3 example.py
|
||||
```
|
||||
|
||||
#### Run C++ example code
|
||||
To use C++ example code, use below command:
|
||||
|
||||
```shell
|
||||
# Run Milvus C++ example
|
||||
@ -160,41 +38,42 @@ $ python3 example.py
|
||||
$ ./sdk_simple
|
||||
```
|
||||
|
||||
#### Run Java example code
|
||||
Make sure Java 8 or higher is already installed.
|
||||
## Roadmap
|
||||
|
||||
Refer to [this link](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples) for the example code.
|
||||
Please read our [roadmap](https://milvus.io/docs/en/roadmap/) for upcoming features.
|
||||
|
||||
## Contribution guidelines
|
||||
|
||||
Contributions are welcomed and greatly appreciated. If you want to contribute to Milvus, please read our [contribution guidelines](CONTRIBUTING.md). This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.
|
||||
Contributions are welcomed and greatly appreciated. Please read our [contribution guidelines](CONTRIBUTING.md) for detailed contribution workflow. This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.
|
||||
|
||||
We use [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) to track issues and bugs. For general questions and public discussions, please join our community.
|
||||
|
||||
## Join the Milvus community
|
||||
## Join our community
|
||||
|
||||
To connect with other users and contributors, welcome to join our [slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk).
|
||||
To connect with other users and contributors, welcome to join our [Slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk).
|
||||
|
||||
## Milvus Roadmap
|
||||
## Thanks
|
||||
|
||||
Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features.
|
||||
We greatly appreciate the help of the following people.
|
||||
|
||||
- [akihoni](https://github.com/akihoni) found a broken link and a small typo in the README file.
|
||||
|
||||
## Resources
|
||||
|
||||
[Milvus official website](https://www.milvus.io)
|
||||
- [Milvus.io](https://www.milvus.io)
|
||||
|
||||
[Milvus docs](https://www.milvus.io/docs/en/userguide/install_milvus/)
|
||||
- [Milvus bootcamp](https://github.com/milvus-io/bootcamp)
|
||||
|
||||
[Milvus bootcamp](https://github.com/milvus-io/bootcamp)
|
||||
- [Milvus Medium](https://medium.com/@milvusio)
|
||||
|
||||
[Milvus blog](https://www.milvus.io/blog/)
|
||||
- [Milvus CSDN](https://zilliz.blog.csdn.net/)
|
||||
|
||||
[Milvus CSDN](https://zilliz.blog.csdn.net/)
|
||||
|
||||
[Milvus roadmap](https://milvus.io/docs/en/roadmap/)
|
||||
- [Milvus Twitter](https://twitter.com/milvusio)
|
||||
|
||||
- [Milvus Facebook](https://www.facebook.com/io.milvus.5)
|
||||
|
||||
## License
|
||||
|
||||
[Apache 2.0 license](LICENSE)
|
||||
[Apache License 2.0](LICENSE)
|
||||
|
||||
|
||||
|
||||
197
README_CN.md
Normal file
197
README_CN.md
Normal file
@ -0,0 +1,197 @@
|
||||
|
||||
|
||||
|
||||
|
||||
[](https://codebeat.co/projects/github-com-jinhai-cn-milvus-master)
|
||||
|
||||
|
||||
|
||||
|
||||
- [Slack 频道](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)
|
||||
- [Twitter](https://twitter.com/milvusio)
|
||||
- [Facebook](https://www.facebook.com/io.milvus.5)
|
||||
- [博客](https://www.milvus.io/blog/)
|
||||
- [CSDN](https://zilliz.blog.csdn.net/)
|
||||
- [中文官网](https://www.milvus.io/zh-CN/)
|
||||
|
||||
# 欢迎来到 Milvus
|
||||
|
||||
## Milvus 是什么
|
||||
|
||||
Milvus 是一款开源的、针对海量特征向量的相似性搜索引擎。基于异构众核计算框架设计，成本更低，性能更好。在有限的计算资源下，十亿向量搜索仅毫秒响应。
|
||||
|
||||
Milvus 提供稳定的 Python、Java 以及 C++ 的 API 接口。
|
||||
|
||||
通过 [版本发布说明](https://milvus.io/docs/zh-CN/release/v0.5.0/) 获取最新发行版本的 Milvus。
|
||||
|
||||
- 异构众核
|
||||
|
||||
Milvus 基于异构众核计算框架设计，成本更低，性能更好。
|
||||
|
||||
- 多元化索引
|
||||
|
||||
Milvus 支持多种索引方式，使用量化索引、基于树的索引和图索引等算法。
|
||||
|
||||
- 资源智能管理
|
||||
|
||||
Milvus 根据实际数据规模和可利用资源，智能调节优化查询计算和索引构建过程。
|
||||
|
||||
- 水平扩容
|
||||
|
||||
Milvus 支持在线 / 离线扩容，仅需执行简单命令，便可弹性伸缩计算节点和存储节点。
|
||||
|
||||
- 高可用性
|
||||
|
||||
Milvus 集成了 Kubernetes 框架，能有效避免单点障碍情况的发生。
|
||||
|
||||
- 简单易用
|
||||
|
||||
Milvus 安装简单，使用方便，并可使您专注于特征向量。
|
||||
|
||||
- 可视化监控
|
||||
|
||||
您可以使用基于 Prometheus 的图形化监控，以便实时跟踪系统性能。
|
||||
|
||||
## 整体架构
|
||||
|
||||
|
||||
|
||||
## 开始使用 Milvus
|
||||
|
||||
### 硬件要求
|
||||
|
||||
| 硬件设备 | 推荐配置 |
|
||||
| -------- | ------------------------------------- |
|
||||
| CPU | Intel CPU Haswell 及以上 |
|
||||
| GPU | NVIDIA Pascal 系列及以上 |
|
||||
| 内存 | 8 GB 或以上（取决于具体向量数据规模） |
|
||||
| 硬盘 | SATA 3.0 SSD 及以上 |
|
||||
|
||||
### 使用 Docker
|
||||
|
||||
您可以方便地使用 Docker 安装 Milvus。具体请查看 [Milvus 安装指南](https://milvus.io/docs/zh-CN/userguide/install_milvus/)。
|
||||
|
||||
### 从源代码编译
|
||||
|
||||
#### 软件要求
|
||||
|
||||
- Ubuntu 18.04 及以上
|
||||
- CMake 3.14 及以上
|
||||
- CUDA 10.0 及以上
|
||||
- NVIDIA driver 418 及以上
|
||||
|
||||
#### 编译
|
||||
|
||||
##### 第一步 安装依赖项
|
||||
|
||||
```shell
|
||||
$ cd [Milvus sourcecode path]/core
|
||||
$ ./ubuntu_build_deps.sh
|
||||
```
|
||||
|
||||
##### 第二步 编译
|
||||
|
||||
```shell
|
||||
$ cd [Milvus sourcecode path]/core
|
||||
$ ./build.sh -t Debug
|
||||
or
|
||||
$ ./build.sh -t Release
|
||||
```
|
||||
|
||||
当您成功编译后，所有 Milvus 必需组件将安装在`[Milvus root path]/core/milvus`路径下。
|
||||
|
||||
##### 启动 Milvus 服务
|
||||
|
||||
```shell
|
||||
$ cd [Milvus root path]/core/milvus
|
||||
```
|
||||
|
||||
在 `LD_LIBRARY_PATH` 中添加 `lib/` 目录：
|
||||
|
||||
```shell
|
||||
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/milvus/lib
|
||||
```
|
||||
|
||||
启动 Milvus 服务：
|
||||
|
||||
```shell
|
||||
$ cd scripts
|
||||
$ ./start_server.sh
|
||||
```
|
||||
|
||||
若要停止 Milvus 服务，请使用如下命令：
|
||||
|
||||
```shell
|
||||
$ ./stop_server.sh
|
||||
```
|
||||
|
||||
若需要修改 Milvus 配置文件 `conf/server_config.yaml` 和`conf/log_config.conf`，请查看 [Milvus 配置](https://milvus.io/docs/zh-CN/reference/milvus_config/)。
|
||||
|
||||
### 开始您的第一个 Milvus 程序
|
||||
|
||||
#### 运行 Python 示例代码
|
||||
|
||||
请确保系统的 Python 版本为 [Python 3.5](https://www.python.org/downloads/) 或以上。
|
||||
|
||||
安装 Milvus Python SDK。
|
||||
|
||||
```shell
|
||||
# Install Milvus Python SDK
|
||||
$ pip install pymilvus==0.2.3
|
||||
```
|
||||
|
||||
创建 `example.py` 文件，并向文件中加入 [Python 示例代码](https://github.com/milvus-io/pymilvus/blob/master/examples/advanced_example.py)。
|
||||
|
||||
运行示例代码
|
||||
|
||||
```shell
|
||||
# Run Milvus Python example
|
||||
$ python3 example.py
|
||||
```
|
||||
|
||||
#### 运行 C++ 示例代码
|
||||
|
||||
```shell
|
||||
# Run Milvus C++ example
|
||||
$ cd [Milvus root path]/core/milvus/bin
|
||||
$ ./sdk_simple
|
||||
```
|
||||
|
||||
#### 运行 Java 示例代码
|
||||
|
||||
请确保系统的 Java 版本为 Java 8 或以上。
|
||||
|
||||
请从[此处](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples)获取 Java 示例代码。
|
||||
|
||||
## 贡献者指南
|
||||
|
||||
我们由衷欢迎您推送贡献。关于贡献流程的详细信息，请参阅 [贡献者指南](https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md)。本项目遵循 Milvus [行为准则](https://github.com/milvus-io/milvus/blob/master/CODE_OF_CONDUCT.md)。如果您希望参与本项目，请遵守该准则的内容。
|
||||
|
||||
我们使用 [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) 追踪问题和补丁。若您希望提出问题或进行讨论，请加入我们的社区。
|
||||
|
||||
## 加入 Milvus 社区
|
||||
|
||||
欢迎加入我们的 [Slack 频道](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk) 以便与其他用户和贡献者进行交流。
|
||||
|
||||
## Milvus 路线图
|
||||
|
||||
请阅读我们的[路线图](https://milvus.io/docs/zh-CN/roadmap/)以获得更多即将开发的新功能。
|
||||
|
||||
## 相关链接
|
||||
|
||||
[Milvus 官方网站](https://www.milvus.io/)
|
||||
|
||||
[Milvus 文档](https://www.milvus.io/docs/en/userguide/install_milvus/)
|
||||
|
||||
[Milvus 在线训练营](https://github.com/milvus-io/bootcamp)
|
||||
|
||||
[Milvus 博客](https://www.milvus.io/blog/)
|
||||
|
||||
[Milvus CSDN](https://zilliz.blog.csdn.net/)
|
||||
|
||||
[Milvus 路线图](https://milvus.io/docs/en/roadmap/)
|
||||
|
||||
## 许可协议
|
||||
|
||||
[Apache 许可协议2.0版](https://github.com/milvus-io/milvus/blob/master/LICENSE)
|
||||
|
||||
39
ci/jenkins/Jenkinsfile
vendored
39
ci/jenkins/Jenkinsfile
vendored
@ -1,13 +1,23 @@
|
||||
#!/usr/bin/env groovy
|
||||
|
||||
String cron_timezone = "TZ=Asia/Shanghai"
|
||||
String cron_string = BRANCH_NAME == "master" ? "H 0 * * * " : ""
|
||||
cron_string = BRANCH_NAME == "0.5.1" ? "H 1 * * * " : cron_string
|
||||
|
||||
pipeline {
|
||||
agent none
|
||||
|
||||
triggers {
|
||||
cron """${cron_timezone}
|
||||
${cron_string}"""
|
||||
}
|
||||
|
||||
options {
|
||||
timestamps()
|
||||
}
|
||||
|
||||
parameters{
|
||||
choice choices: ['Release', 'Debug'], description: '', name: 'BUILD_TYPE'
|
||||
string defaultValue: 'cf1434e7-5a4b-4d25-82e8-88d667aef9e5', description: 'GIT CREDENTIALS ID', name: 'GIT_CREDENTIALS_ID', trim: true
|
||||
string defaultValue: 'registry.zilliz.com', description: 'DOCKER REGISTRY URL', name: 'DOKCER_REGISTRY_URL', trim: true
|
||||
string defaultValue: 'ba070c98-c8cc-4f7c-b657-897715f359fc', description: 'DOCKER CREDENTIALS ID', name: 'DOCKER_CREDENTIALS_ID', trim: true
|
||||
string defaultValue: 'http://192.168.1.202/artifactory/milvus', description: 'JFROG ARTFACTORY URL', name: 'JFROG_ARTFACTORY_URL', trim: true
|
||||
@ -47,7 +57,7 @@ pipeline {
|
||||
steps {
|
||||
container('milvus-build-env') {
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/build.groovy"
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/build.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -56,7 +66,7 @@ pipeline {
|
||||
steps {
|
||||
container('milvus-build-env') {
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/coverage.groovy"
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/coverage.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -65,7 +75,7 @@ pipeline {
|
||||
steps {
|
||||
container('milvus-build-env') {
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/package.groovy"
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/package.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -87,7 +97,7 @@ pipeline {
|
||||
steps {
|
||||
container('publish-images'){
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/publishImages.groovy"
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/publishImages.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -109,7 +119,7 @@ pipeline {
|
||||
steps {
|
||||
container('milvus-test-env') {
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/deploySingle2Dev.groovy"
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/deploySingle2Dev.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -119,7 +129,12 @@ pipeline {
|
||||
steps {
|
||||
container('milvus-test-env') {
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/singleDevTest.groovy"
|
||||
boolean isNightlyTest = isTimeTriggeredBuild()
|
||||
if (isNightlyTest) {
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/singleDevNightlyTest.groovy"
|
||||
} else {
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/singleDevTest.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -129,7 +144,7 @@ pipeline {
|
||||
steps {
|
||||
container('milvus-test-env') {
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/cleanupSingleDev.groovy"
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/cleanupSingleDev.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -139,7 +154,7 @@ pipeline {
|
||||
unsuccessful {
|
||||
container('milvus-test-env') {
|
||||
script {
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/cleanupSingleDev.groovy"
|
||||
load "${env.WORKSPACE}/ci/jenkins/step/cleanupSingleDev.groovy"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -150,3 +165,9 @@ pipeline {
|
||||
}
|
||||
}
|
||||
|
||||
boolean isTimeTriggeredBuild() {
|
||||
if (currentBuild.getBuildCauses('hudson.triggers.TimerTrigger$TimerTriggerCause').size() != 0) {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
@ -1,14 +0,0 @@
|
||||
try {
|
||||
sh 'helm init --client-only --skip-refresh --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts'
|
||||
sh 'helm repo update'
|
||||
dir ('milvus-helm') {
|
||||
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_CREDENTIALS_ID}", url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
|
||||
dir ("milvus-gpu") {
|
||||
sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
|
||||
}
|
||||
}
|
||||
} catch (exc) {
|
||||
echo 'Helm running failed!'
|
||||
sh "helm del --purge ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu"
|
||||
throw exc
|
||||
}
|
||||
@ -8,7 +8,7 @@ metadata:
|
||||
spec:
|
||||
containers:
|
||||
- name: milvus-build-env
|
||||
image: registry.zilliz.com/milvus/milvus-build-env:v0.5.0-ubuntu18.04
|
||||
image: registry.zilliz.com/milvus/milvus-build-env:v0.5.1-ubuntu18.04
|
||||
env:
|
||||
- name: POD_IP
|
||||
valueFrom:
|
||||
|
||||
@ -109,6 +109,7 @@ for test in `ls ${DIR_UNITTEST}`; do
|
||||
if [ $? -ne 0 ]; then
|
||||
echo ${args}
|
||||
echo ${DIR_UNITTEST}/${test} "run failed"
|
||||
exit -1
|
||||
fi
|
||||
done
|
||||
|
||||
@ -131,8 +132,13 @@ ${LCOV_CMD} -r "${FILE_INFO_OUTPUT}" -o "${FILE_INFO_OUTPUT_NEW}" \
|
||||
"*/src/server/Server.cpp" \
|
||||
"*/src/server/DBWrapper.cpp" \
|
||||
"*/src/server/grpc_impl/GrpcServer.cpp" \
|
||||
"*/src/utils/easylogging++.h" \
|
||||
"*/src/utils/easylogging++.cc"
|
||||
"*/src/external/easyloggingpp/easylogging++.h" \
|
||||
"*/src/external/easyloggingpp/easylogging++.cc"
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "gen ${FILE_INFO_OUTPUT_NEW} failed"
|
||||
exit -2
|
||||
fi
|
||||
|
||||
# gen html report
|
||||
# ${LCOV_GEN_CMD} "${FILE_INFO_OUTPUT_NEW}" --output-directory ${DIR_LCOV_OUTPUT}/
|
||||
|
||||
@ -1,5 +1,8 @@
|
||||
try {
|
||||
sh "helm del --purge ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu"
|
||||
def helmResult = sh script: "helm status ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu", returnStatus: true
|
||||
if (!helmResult) {
|
||||
sh "helm del --purge ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu"
|
||||
}
|
||||
} catch (exc) {
|
||||
def helmResult = sh script: "helm status ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu", returnStatus: true
|
||||
if (!helmResult) {
|
||||
@ -1,4 +1,4 @@
|
||||
timeout(time: 60, unit: 'MINUTES') {
|
||||
timeout(time: 30, unit: 'MINUTES') {
|
||||
dir ("ci/jenkins/scripts") {
|
||||
sh "./coverage.sh -o /opt/milvus -u root -p 123456 -t \$POD_IP"
|
||||
// Set some env variables so codecov detection script works correctly
|
||||
9
ci/jenkins/step/deploySingle2Dev.groovy
Normal file
9
ci/jenkins/step/deploySingle2Dev.groovy
Normal file
@ -0,0 +1,9 @@
|
||||
sh 'helm init --client-only --skip-refresh --stable-repo-url https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts'
|
||||
sh 'helm repo update'
|
||||
dir ('milvus-helm') {
|
||||
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
|
||||
dir ("milvus-gpu") {
|
||||
sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/db_backend/sqlite_values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,14 +1,14 @@
|
||||
timeout(time: 30, unit: 'MINUTES') {
|
||||
timeout(time: 90, unit: 'MINUTES') {
|
||||
dir ("tests/milvus_python_test") {
|
||||
sh 'python3 -m pip install -r requirements.txt'
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --level=1 --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
|
||||
}
|
||||
// mysql database backend test
|
||||
load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/cleanupSingleDev.groovy"
|
||||
|
||||
if (!fileExists('milvus-helm')) {
|
||||
dir ("milvus-helm") {
|
||||
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_CREDENTIALS_ID}", url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
|
||||
checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
|
||||
}
|
||||
}
|
||||
dir ("milvus-helm") {
|
||||
@ -17,6 +17,6 @@ timeout(time: 30, unit: 'MINUTES') {
|
||||
}
|
||||
}
|
||||
dir ("tests/milvus_python_test") {
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --level=1 --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
|
||||
}
|
||||
}
|
||||
24
ci/jenkins/step/singleDevTest.groovy
Normal file
24
ci/jenkins/step/singleDevTest.groovy
Normal file
@ -0,0 +1,24 @@
|
||||
timeout(time: 60, unit: 'MINUTES') {
|
||||
dir ("tests/milvus_python_test") {
|
||||
sh 'python3 -m pip install -r requirements.txt'
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --level=1 --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
|
||||
}
|
||||
// mysql database backend test
|
||||
// load "${env.WORKSPACE}/ci/jenkins/jenkinsfile/cleanupSingleDev.groovy"
|
||||
|
||||
// Remove mysql-version tests: 10-28
|
||||
|
||||
// if (!fileExists('milvus-helm')) {
|
||||
// dir ("milvus-helm") {
|
||||
// checkout([$class: 'GitSCM', branches: [[name: "0.5.0"]], userRemoteConfigs: [[url: "https://github.com/milvus-io/milvus-helm.git", name: 'origin', refspec: "+refs/heads/0.5.0:refs/remotes/origin/0.5.0"]]])
|
||||
// }
|
||||
// }
|
||||
// dir ("milvus-helm") {
|
||||
// dir ("milvus-gpu") {
|
||||
// sh "helm install --wait --timeout 300 --set engine.image.tag=${DOCKER_VERSION} --set expose.type=clusterIP --name ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu -f ci/db_backend/mysql_values.yaml -f ci/filebeat/values.yaml --namespace milvus ."
|
||||
// }
|
||||
// }
|
||||
// dir ("tests/milvus_python_test") {
|
||||
// sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --level=1 --ip ${env.PIPELINE_NAME}-${env.BUILD_NUMBER}-single-gpu-milvus-gpu-engine.milvus.svc.cluster.local"
|
||||
// }
|
||||
}
|
||||
@ -3,7 +3,7 @@ timeout(time: 30, unit: 'MINUTES') {
|
||||
dir ("${PROJECT_NAME}_test") {
|
||||
checkout([$class: 'GitSCM', branches: [[name: "${SEMVER}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_USER}", url: "git@192.168.1.105:Test/milvus_test.git", name: 'origin', refspec: "+refs/heads/${SEMVER}:refs/remotes/origin/${SEMVER}"]]])
|
||||
sh 'python3 -m pip install -r requirements.txt -i http://pypi.douban.com/simple --trusted-host pypi.douban.com'
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --level=1 --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-1.svc.cluster.local"
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --level=1 --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-1.svc.cluster.local --internal=true"
|
||||
}
|
||||
// mysql database backend test
|
||||
load "${env.WORKSPACE}/ci/jenkinsfile/cleanup_dev.groovy"
|
||||
@ -19,7 +19,7 @@ timeout(time: 30, unit: 'MINUTES') {
|
||||
}
|
||||
}
|
||||
dir ("${PROJECT_NAME}_test") {
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --level=1 --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-2.svc.cluster.local"
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --level=1 --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-2.svc.cluster.local --internal=true"
|
||||
}
|
||||
} catch (exc) {
|
||||
echo 'Milvus Test Failed !'
|
||||
|
||||
@ -3,7 +3,7 @@ timeout(time: 60, unit: 'MINUTES') {
|
||||
dir ("${PROJECT_NAME}_test") {
|
||||
checkout([$class: 'GitSCM', branches: [[name: "${SEMVER}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${params.GIT_USER}", url: "git@192.168.1.105:Test/milvus_test.git", name: 'origin', refspec: "+refs/heads/${SEMVER}:refs/remotes/origin/${SEMVER}"]]])
|
||||
sh 'python3 -m pip install -r requirements.txt -i http://pypi.douban.com/simple --trusted-host pypi.douban.com'
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-1.svc.cluster.local"
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/sqlite\" --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-1.svc.cluster.local --internal=true"
|
||||
}
|
||||
|
||||
// mysql database backend test
|
||||
@ -20,7 +20,7 @@ timeout(time: 60, unit: 'MINUTES') {
|
||||
}
|
||||
}
|
||||
dir ("${PROJECT_NAME}_test") {
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-2.svc.cluster.local"
|
||||
sh "pytest . --alluredir=\"test_out/dev/single/mysql\" --ip ${env.JOB_NAME}-${env.BUILD_NUMBER}-milvus-gpu-engine.milvus-2.svc.cluster.local --internal=true"
|
||||
}
|
||||
} catch (exc) {
|
||||
echo 'Milvus Test Failed !'
|
||||
|
||||
@ -14,7 +14,7 @@ container('milvus-build-env') {
|
||||
sh "export JFROG_ARTFACTORY_URL='${params.JFROG_ARTFACTORY_URL}' \
|
||||
&& export JFROG_USER_NAME='${USERNAME}' \
|
||||
&& export JFROG_PASSWORD='${PASSWORD}' \
|
||||
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.2.1/faiss-branch-0.2.1.tar.gz' \
|
||||
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.3.0/faiss-branch-0.3.0.tar.gz' \
|
||||
&& ./build.sh -t ${params.BUILD_TYPE} -d /opt/milvus -j -u -c"
|
||||
|
||||
sh "./coverage.sh -u root -p 123456 -t \$POD_IP"
|
||||
|
||||
@ -14,7 +14,7 @@ container('milvus-build-env') {
|
||||
sh "export JFROG_ARTFACTORY_URL='${params.JFROG_ARTFACTORY_URL}' \
|
||||
&& export JFROG_USER_NAME='${USERNAME}' \
|
||||
&& export JFROG_PASSWORD='${PASSWORD}' \
|
||||
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.2.1/faiss-branch-0.2.1.tar.gz' \
|
||||
&& export FAISS_URL='http://192.168.1.105:6060/jinhai/faiss/-/archive/branch-0.3.0/faiss-branch-0.3.0.tar.gz' \
|
||||
&& ./build.sh -t ${params.BUILD_TYPE} -j -d /opt/milvus"
|
||||
}
|
||||
}
|
||||
|
||||
@ -32,15 +32,19 @@ string(REGEX REPLACE "\n" "" BUILD_TIME ${BUILD_TIME})
|
||||
message(STATUS "Build time = ${BUILD_TIME}")
|
||||
|
||||
MACRO (GET_GIT_BRANCH_NAME GIT_BRANCH_NAME)
|
||||
execute_process(COMMAND "git" symbolic-ref --short HEAD OUTPUT_VARIABLE ${GIT_BRANCH_NAME})
|
||||
execute_process(COMMAND "git" rev-parse --abbrev-ref HEAD OUTPUT_VARIABLE ${GIT_BRANCH_NAME})
|
||||
if(GIT_BRANCH_NAME STREQUAL "")
|
||||
execute_process(COMMAND "git" symbolic-ref --short -q HEAD OUTPUT_VARIABLE ${GIT_BRANCH_NAME})
|
||||
endif()
|
||||
ENDMACRO (GET_GIT_BRANCH_NAME)
|
||||
|
||||
GET_GIT_BRANCH_NAME(GIT_BRANCH_NAME)
|
||||
message(STATUS "GIT_BRANCH_NAME = ${GIT_BRANCH_NAME}")
|
||||
if(NOT GIT_BRANCH_NAME STREQUAL "")
|
||||
string(REGEX REPLACE "\n" "" GIT_BRANCH_NAME ${GIT_BRANCH_NAME})
|
||||
endif()
|
||||
|
||||
set(MILVUS_VERSION "${GIT_BRANCH_NAME}")
|
||||
set(MILVUS_VERSION "0.5.1")
|
||||
string(REGEX MATCH "[0-9]+\\.[0-9]+\\.[0-9]" MILVUS_VERSION "${MILVUS_VERSION}")
|
||||
|
||||
find_package(ClangTools)
|
||||
@ -71,7 +75,7 @@ if(MILVUS_VERSION_MAJOR STREQUAL ""
|
||||
endif()
|
||||
|
||||
message(STATUS "Build version = ${MILVUS_VERSION}")
|
||||
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/version.h.macro ${CMAKE_CURRENT_SOURCE_DIR}/version.h)
|
||||
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/src/version.h.macro ${CMAKE_CURRENT_SOURCE_DIR}/src/version.h)
|
||||
|
||||
message(STATUS "Milvus version: "
|
||||
"${MILVUS_VERSION_MAJOR}.${MILVUS_VERSION_MINOR}.${MILVUS_VERSION_PATCH} "
|
||||
|
||||
@ -6,4 +6,5 @@
|
||||
*easylogging++*
|
||||
*SqliteMetaImpl.cpp
|
||||
*src/grpc*
|
||||
*src/external*
|
||||
*milvus/include*
|
||||
@ -55,21 +55,10 @@ define_option_string(MILVUS_DEPENDENCY_SOURCE
|
||||
define_option(MILVUS_VERBOSE_THIRDPARTY_BUILD
|
||||
"Show output from ExternalProjects rather than just logging to files" ON)
|
||||
|
||||
define_option(MILVUS_BOOST_VENDORED "Use vendored Boost instead of existing Boost. \
|
||||
Note that this requires linking Boost statically" OFF)
|
||||
|
||||
define_option(MILVUS_BOOST_HEADER_ONLY "Use only BOOST headers" OFF)
|
||||
|
||||
define_option(MILVUS_WITH_BZ2 "Build with BZ2 compression" ON)
|
||||
|
||||
define_option(MILVUS_WITH_EASYLOGGINGPP "Build with Easylogging++ library" ON)
|
||||
|
||||
define_option(MILVUS_WITH_LZ4 "Build with lz4 compression" ON)
|
||||
|
||||
define_option(MILVUS_WITH_PROMETHEUS "Build with PROMETHEUS library" ON)
|
||||
|
||||
define_option(MILVUS_WITH_SNAPPY "Build with Snappy compression" ON)
|
||||
|
||||
define_option(MILVUS_WITH_SQLITE "Build with SQLite library" ON)
|
||||
|
||||
define_option(MILVUS_WITH_SQLITE_ORM "Build with SQLite ORM library" ON)
|
||||
@ -78,16 +67,6 @@ define_option(MILVUS_WITH_MYSQLPP "Build with MySQL++" ON)
|
||||
|
||||
define_option(MILVUS_WITH_YAMLCPP "Build with yaml-cpp library" ON)
|
||||
|
||||
define_option(MILVUS_WITH_ZLIB "Build with zlib compression" ON)
|
||||
|
||||
if(CMAKE_VERSION VERSION_LESS 3.7)
|
||||
set(MILVUS_WITH_ZSTD_DEFAULT OFF)
|
||||
else()
|
||||
# ExternalProject_Add(SOURCE_SUBDIR) is available since CMake 3.7.
|
||||
set(MILVUS_WITH_ZSTD_DEFAULT ON)
|
||||
endif()
|
||||
define_option(MILVUS_WITH_ZSTD "Build with zstd compression" ${MILVUS_WITH_ZSTD_DEFAULT})
|
||||
|
||||
if (MILVUS_ENABLE_PROFILING STREQUAL "ON")
|
||||
define_option(MILVUS_WITH_LIBUNWIND "Build with libunwind" ON)
|
||||
define_option(MILVUS_WITH_GPERFTOOLS "Build with gperftools" ON)
|
||||
@ -95,6 +74,8 @@ endif()
|
||||
|
||||
define_option(MILVUS_WITH_GRPC "Build with GRPC" ON)
|
||||
|
||||
define_option(MILVUS_WITH_ZLIB "Build with zlib compression" ON)
|
||||
|
||||
#----------------------------------------------------------------------
|
||||
if(MSVC)
|
||||
set_option_category("MSVC")
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -2,9 +2,9 @@
|
||||
|
||||
server_config:
|
||||
address: 0.0.0.0 # milvus server ip address (IPv4)
|
||||
port: 19530 # port range: 1025 ~ 65534
|
||||
port: 19530 # milvus server port, must in range [1025, 65534]
|
||||
deploy_mode: single # deployment type: single, cluster_readonly, cluster_writable
|
||||
time_zone: UTC+8
|
||||
time_zone: UTC+8 # time zone, must be in format: UTC+X
|
||||
|
||||
db_config:
|
||||
primary_path: @MILVUS_DB_PATH@ # path used to store data and meta
|
||||
@ -14,30 +14,31 @@ db_config:
|
||||
# Keep 'dialect://:@:/', and replace other texts with real values
|
||||
# Replace 'dialect' with 'mysql' or 'sqlite'
|
||||
|
||||
insert_buffer_size: 4 # GB, maximum insert buffer size allowed
|
||||
insert_buffer_size: 4 # GB, maximum insert buffer size allowed, must be a positive integer
|
||||
# sum of insert_buffer_size and cpu_cache_capacity cannot exceed total memory
|
||||
|
||||
preload_table: # preload data at startup, '*' means load all tables, empty value means no preload
|
||||
# you can specify preload tables like this: table1,table2,table3
|
||||
|
||||
metric_config:
|
||||
enable_monitor: false # enable monitoring or not
|
||||
enable_monitor: false # enable monitoring or not, must be a boolean
|
||||
collector: prometheus # prometheus
|
||||
prometheus_config:
|
||||
port: 8080 # port prometheus uses to fetch metrics
|
||||
port: 8080 # port prometheus uses to fetch metrics, must in range [1025, 65534]
|
||||
|
||||
cache_config:
|
||||
cpu_cache_capacity: 16 # GB, CPU memory used for cache
|
||||
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
|
||||
gpu_cache_capacity: 4 # GB, GPU memory used for cache
|
||||
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered
|
||||
cache_insert_data: false # whether to load inserted data into cache
|
||||
cpu_cache_capacity: 16 # GB, CPU memory used for cache, must be a positive integer
|
||||
cpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
|
||||
gpu_cache_capacity: 4 # GB, GPU memory used for cache, must be a positive integer
|
||||
gpu_cache_threshold: 0.85 # percentage of data that will be kept when cache cleanup is triggered, must be in range (0.0, 1.0]
|
||||
cache_insert_data: false # whether to load inserted data into cache, must be a boolean
|
||||
|
||||
engine_config:
|
||||
use_blas_threshold: 20 # if nq < use_blas_threshold, use SSE, faster with fluctuated response times
|
||||
# if nq >= use_blas_threshold, use OpenBlas, slower with stable response times
|
||||
gpu_search_threshold: 1000 # threshold beyond which the search computation is executed on GPUs only
|
||||
|
||||
resource_config:
|
||||
search_resources: # define the GPUs used for search computation, valid value: gpux
|
||||
search_resources: # define the GPUs used for search computation, must be in format: gpux
|
||||
- gpu0
|
||||
index_build_device: gpu0 # GPU used for building index
|
||||
index_build_device: gpu0 # GPU used for building index, must be in format: gpux
|
||||
@ -99,6 +99,7 @@ for test in `ls ${DIR_UNITTEST}`; do
|
||||
if [ $? -ne 0 ]; then
|
||||
echo ${args}
|
||||
echo ${DIR_UNITTEST}/${test} "run failed"
|
||||
exit -1
|
||||
fi
|
||||
done
|
||||
|
||||
@ -121,8 +122,13 @@ ${LCOV_CMD} -r "${FILE_INFO_OUTPUT}" -o "${FILE_INFO_OUTPUT_NEW}" \
|
||||
"*/src/server/Server.cpp" \
|
||||
"*/src/server/DBWrapper.cpp" \
|
||||
"*/src/server/grpc_impl/GrpcServer.cpp" \
|
||||
"*/src/utils/easylogging++.h" \
|
||||
"*/src/utils/easylogging++.cc"
|
||||
"*/easylogging++.h" \
|
||||
"*/easylogging++.cc" \
|
||||
"*/src/external/*"
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "generate ${FILE_INFO_OUTPUT_NEW} failed"
|
||||
exit -2
|
||||
fi
|
||||
# gen html report
|
||||
${LCOV_GEN_CMD} "${FILE_INFO_OUTPUT_NEW}" --output-directory ${DIR_LCOV_OUTPUT}/
|
||||
|
||||
@ -64,6 +64,13 @@ set(scheduler_files
|
||||
${scheduler_task_files}
|
||||
)
|
||||
|
||||
aux_source_directory(${MILVUS_ENGINE_SRC}/external/easyloggingpp external_easyloggingpp_files)
|
||||
aux_source_directory(${MILVUS_ENGINE_SRC}/external/nlohmann external_nlohmann_files)
|
||||
set(external_files
|
||||
${external_easyloggingpp_files}
|
||||
${external_nlohmann_files}
|
||||
)
|
||||
|
||||
aux_source_directory(${MILVUS_ENGINE_SRC}/server server_files)
|
||||
aux_source_directory(${MILVUS_ENGINE_SRC}/server/grpc_impl grpc_server_files)
|
||||
aux_source_directory(${MILVUS_ENGINE_SRC}/utils utils_files)
|
||||
@ -77,6 +84,7 @@ set(engine_files
|
||||
${db_insert_files}
|
||||
${db_meta_files}
|
||||
${metrics_files}
|
||||
${external_files}
|
||||
${utils_files}
|
||||
${wrapper_files}
|
||||
)
|
||||
@ -112,14 +120,10 @@ set(third_party_libs
|
||||
${client_grpc_lib}
|
||||
yaml-cpp
|
||||
${prometheus_lib}
|
||||
${boost_lib}
|
||||
bzip2
|
||||
lz4
|
||||
snappy
|
||||
zlib
|
||||
zstd
|
||||
${cuda_lib}
|
||||
mysqlpp
|
||||
zlib
|
||||
${boost_lib}
|
||||
)
|
||||
|
||||
if (MILVUS_ENABLE_PROFILING STREQUAL "ON")
|
||||
|
||||
@ -67,15 +67,16 @@ class DB {
|
||||
|
||||
virtual Status
|
||||
Query(const std::string& table_id, uint64_t k, uint64_t nq, uint64_t nprobe, const float* vectors,
|
||||
QueryResults& results) = 0;
|
||||
ResultIds& result_ids, ResultDistances& result_distances) = 0;
|
||||
|
||||
virtual Status
|
||||
Query(const std::string& table_id, uint64_t k, uint64_t nq, uint64_t nprobe, const float* vectors,
|
||||
const meta::DatesT& dates, QueryResults& results) = 0;
|
||||
const meta::DatesT& dates, ResultIds& result_ids, ResultDistances& result_distances) = 0;
|
||||
|
||||
virtual Status
|
||||
Query(const std::string& table_id, const std::vector<std::string>& file_ids, uint64_t k, uint64_t nq,
|
||||
uint64_t nprobe, const float* vectors, const meta::DatesT& dates, QueryResults& results) = 0;
|
||||
uint64_t nprobe, const float* vectors, const meta::DatesT& dates, ResultIds& result_ids,
|
||||
ResultDistances& result_distances) = 0;
|
||||
|
||||
virtual Status
|
||||
Size(uint64_t& result) = 0;
|
||||
|
||||
@ -136,7 +136,7 @@ DBImpl::DeleteTable(const std::string& table_id, const meta::DatesT& dates) {
|
||||
|
||||
// scheduler will determine when to delete table files
|
||||
auto nres = scheduler::ResMgrInst::GetInstance()->GetNumOfComputeResource();
|
||||
scheduler::DeleteJobPtr job = std::make_shared<scheduler::DeleteJob>(0, table_id, meta_ptr_, nres);
|
||||
scheduler::DeleteJobPtr job = std::make_shared<scheduler::DeleteJob>(table_id, meta_ptr_, nres);
|
||||
scheduler::JobMgrInst::GetInstance()->Put(job);
|
||||
job->WaitAndDelete();
|
||||
} else {
|
||||
@ -336,20 +336,20 @@ DBImpl::DropIndex(const std::string& table_id) {
|
||||
|
||||
Status
|
||||
DBImpl::Query(const std::string& table_id, uint64_t k, uint64_t nq, uint64_t nprobe, const float* vectors,
|
||||
QueryResults& results) {
|
||||
ResultIds& result_ids, ResultDistances& result_distances) {
|
||||
if (shutting_down_.load(std::memory_order_acquire)) {
|
||||
return Status(DB_ERROR, "Milsvus server is shutdown!");
|
||||
}
|
||||
|
||||
meta::DatesT dates = {utils::GetDate()};
|
||||
Status result = Query(table_id, k, nq, nprobe, vectors, dates, results);
|
||||
Status result = Query(table_id, k, nq, nprobe, vectors, dates, result_ids, result_distances);
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
Status
|
||||
DBImpl::Query(const std::string& table_id, uint64_t k, uint64_t nq, uint64_t nprobe, const float* vectors,
|
||||
const meta::DatesT& dates, QueryResults& results) {
|
||||
const meta::DatesT& dates, ResultIds& result_ids, ResultDistances& result_distances) {
|
||||
if (shutting_down_.load(std::memory_order_acquire)) {
|
||||
return Status(DB_ERROR, "Milsvus server is shutdown!");
|
||||
}
|
||||
@ -372,14 +372,15 @@ DBImpl::Query(const std::string& table_id, uint64_t k, uint64_t nq, uint64_t npr
|
||||
}
|
||||
|
||||
cache::CpuCacheMgr::GetInstance()->PrintInfo(); // print cache info before query
|
||||
status = QueryAsync(table_id, file_id_array, k, nq, nprobe, vectors, results);
|
||||
status = QueryAsync(table_id, file_id_array, k, nq, nprobe, vectors, result_ids, result_distances);
|
||||
cache::CpuCacheMgr::GetInstance()->PrintInfo(); // print cache info after query
|
||||
return status;
|
||||
}
|
||||
|
||||
Status
|
||||
DBImpl::Query(const std::string& table_id, const std::vector<std::string>& file_ids, uint64_t k, uint64_t nq,
|
||||
uint64_t nprobe, const float* vectors, const meta::DatesT& dates, QueryResults& results) {
|
||||
uint64_t nprobe, const float* vectors, const meta::DatesT& dates, ResultIds& result_ids,
|
||||
ResultDistances& result_distances) {
|
||||
if (shutting_down_.load(std::memory_order_acquire)) {
|
||||
return Status(DB_ERROR, "Milsvus server is shutdown!");
|
||||
}
|
||||
@ -413,7 +414,7 @@ DBImpl::Query(const std::string& table_id, const std::vector<std::string>& file_
|
||||
}
|
||||
|
||||
cache::CpuCacheMgr::GetInstance()->PrintInfo(); // print cache info before query
|
||||
status = QueryAsync(table_id, file_id_array, k, nq, nprobe, vectors, results);
|
||||
status = QueryAsync(table_id, file_id_array, k, nq, nprobe, vectors, result_ids, result_distances);
|
||||
cache::CpuCacheMgr::GetInstance()->PrintInfo(); // print cache info after query
|
||||
return status;
|
||||
}
|
||||
@ -432,14 +433,14 @@ DBImpl::Size(uint64_t& result) {
|
||||
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
|
||||
Status
|
||||
DBImpl::QueryAsync(const std::string& table_id, const meta::TableFilesSchema& files, uint64_t k, uint64_t nq,
|
||||
uint64_t nprobe, const float* vectors, QueryResults& results) {
|
||||
uint64_t nprobe, const float* vectors, ResultIds& result_ids, ResultDistances& result_distances) {
|
||||
server::CollectQueryMetrics metrics(nq);
|
||||
|
||||
TimeRecorder rc("");
|
||||
|
||||
// step 1: get files to search
|
||||
ENGINE_LOG_DEBUG << "Engine query begin, index file count: " << files.size();
|
||||
scheduler::SearchJobPtr job = std::make_shared<scheduler::SearchJob>(0, k, nq, nprobe, vectors);
|
||||
scheduler::SearchJobPtr job = std::make_shared<scheduler::SearchJob>(k, nq, nprobe, vectors);
|
||||
for (auto& file : files) {
|
||||
scheduler::TableFileSchemaPtr file_ptr = std::make_shared<meta::TableFileSchema>(file);
|
||||
job->AddIndexFile(file_ptr);
|
||||
@ -453,7 +454,8 @@ DBImpl::QueryAsync(const std::string& table_id, const meta::TableFilesSchema& fi
|
||||
}
|
||||
|
||||
// step 3: construct results
|
||||
results = job->GetResult();
|
||||
result_ids = job->GetResultIds();
|
||||
result_distances = job->GetResultDistances();
|
||||
rc.ElapseFromBegin("Engine query totally cost");
|
||||
|
||||
return Status::OK();
|
||||
@ -754,7 +756,7 @@ DBImpl::BackgroundBuildIndex() {
|
||||
Status status;
|
||||
|
||||
if (!to_index_files.empty()) {
|
||||
scheduler::BuildIndexJobPtr job = std::make_shared<scheduler::BuildIndexJob>(0, meta_ptr_, options_);
|
||||
scheduler::BuildIndexJobPtr job = std::make_shared<scheduler::BuildIndexJob>(meta_ptr_, options_);
|
||||
|
||||
// step 2: put build index task to scheduler
|
||||
for (auto& file : to_index_files) {
|
||||
|
||||
@ -91,15 +91,16 @@ class DBImpl : public DB {
|
||||
|
||||
Status
|
||||
Query(const std::string& table_id, uint64_t k, uint64_t nq, uint64_t nprobe, const float* vectors,
|
||||
QueryResults& results) override;
|
||||
ResultIds& result_ids, ResultDistances& result_distances) override;
|
||||
|
||||
Status
|
||||
Query(const std::string& table_id, uint64_t k, uint64_t nq, uint64_t nprobe, const float* vectors,
|
||||
const meta::DatesT& dates, QueryResults& results) override;
|
||||
const meta::DatesT& dates, ResultIds& result_ids, ResultDistances& result_distances) override;
|
||||
|
||||
Status
|
||||
Query(const std::string& table_id, const std::vector<std::string>& file_ids, uint64_t k, uint64_t nq,
|
||||
uint64_t nprobe, const float* vectors, const meta::DatesT& dates, QueryResults& results) override;
|
||||
uint64_t nprobe, const float* vectors, const meta::DatesT& dates, ResultIds& result_ids,
|
||||
ResultDistances& result_distances) override;
|
||||
|
||||
Status
|
||||
Size(uint64_t& result) override;
|
||||
@ -107,7 +108,7 @@ class DBImpl : public DB {
|
||||
private:
|
||||
Status
|
||||
QueryAsync(const std::string& table_id, const meta::TableFilesSchema& files, uint64_t k, uint64_t nq,
|
||||
uint64_t nprobe, const float* vectors, QueryResults& results);
|
||||
uint64_t nprobe, const float* vectors, ResultIds& result_ids, ResultDistances& result_distances);
|
||||
|
||||
void
|
||||
BackgroundTimerTask();
|
||||
|
||||
@ -19,6 +19,7 @@
|
||||
|
||||
#include "db/engine/ExecutionEngine.h"
|
||||
|
||||
#include <faiss/Index.h>
|
||||
#include <stdint.h>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
@ -26,12 +27,13 @@
|
||||
namespace milvus {
|
||||
namespace engine {
|
||||
|
||||
typedef int64_t IDNumber;
|
||||
using IDNumber = faiss::Index::idx_t;
|
||||
|
||||
typedef IDNumber* IDNumberPtr;
|
||||
typedef std::vector<IDNumber> IDNumbers;
|
||||
|
||||
typedef std::vector<std::pair<IDNumber, double>> QueryResult;
|
||||
typedef std::vector<QueryResult> QueryResults;
|
||||
typedef std::vector<faiss::Index::idx_t> ResultIds;
|
||||
typedef std::vector<faiss::Index::distance_t> ResultDistances;
|
||||
|
||||
struct TableIndex {
|
||||
int32_t engine_type_ = (int)EngineType::FAISS_IDMAP;
|
||||
|
||||
@ -18,16 +18,13 @@
|
||||
#include "db/engine/ExecutionEngineImpl.h"
|
||||
#include "cache/CpuCacheMgr.h"
|
||||
#include "cache/GpuCacheMgr.h"
|
||||
#include "knowhere/common/Config.h"
|
||||
#include "metrics/Metrics.h"
|
||||
#include "scheduler/Utils.h"
|
||||
#include "server/Config.h"
|
||||
#include "utils/CommonUtil.h"
|
||||
#include "utils/Exception.h"
|
||||
#include "utils/Log.h"
|
||||
|
||||
#include "knowhere/common/Config.h"
|
||||
#include "knowhere/common/Exception.h"
|
||||
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
|
||||
#include "scheduler/Utils.h"
|
||||
#include "server/Config.h"
|
||||
#include "wrapper/ConfAdapter.h"
|
||||
#include "wrapper/ConfAdapterMgr.h"
|
||||
#include "wrapper/VecImpl.h"
|
||||
@ -259,33 +256,66 @@ ExecutionEngineImpl::Load(bool to_cache) {
|
||||
|
||||
Status
|
||||
ExecutionEngineImpl::CopyToGpu(uint64_t device_id, bool hybrid) {
|
||||
#if 0
|
||||
if (hybrid) {
|
||||
const std::string key = location_ + ".quantizer";
|
||||
std::vector<uint64_t> gpus{device_id};
|
||||
|
||||
const int64_t NOT_FOUND = -1;
|
||||
int64_t device_id = NOT_FOUND;
|
||||
|
||||
// cache hit
|
||||
{
|
||||
knowhere::QuantizerPtr quantizer = nullptr;
|
||||
|
||||
for (auto& gpu : gpus) {
|
||||
auto cache = cache::GpuCacheMgr::GetInstance(gpu);
|
||||
if (auto cached_quantizer = cache->GetIndex(key)) {
|
||||
device_id = gpu;
|
||||
quantizer = std::static_pointer_cast<CachedQuantizer>(cached_quantizer)->Data();
|
||||
}
|
||||
}
|
||||
|
||||
if (device_id != NOT_FOUND) {
|
||||
// cache hit
|
||||
auto config = std::make_shared<knowhere::QuantizerCfg>();
|
||||
config->gpu_id = device_id;
|
||||
config->mode = 2;
|
||||
auto new_index = index_->LoadData(quantizer, config);
|
||||
index_ = new_index;
|
||||
}
|
||||
}
|
||||
|
||||
if (device_id == NOT_FOUND) {
|
||||
// cache miss
|
||||
std::vector<int64_t> all_free_mem;
|
||||
for (auto& gpu : gpus) {
|
||||
auto cache = cache::GpuCacheMgr::GetInstance(gpu);
|
||||
auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
|
||||
all_free_mem.push_back(free_mem);
|
||||
}
|
||||
|
||||
auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
|
||||
auto best_index = std::distance(all_free_mem.begin(), max_e);
|
||||
device_id = gpus[best_index];
|
||||
|
||||
auto pair = index_->CopyToGpuWithQuantizer(device_id);
|
||||
index_ = pair.first;
|
||||
|
||||
// cache
|
||||
auto cached_quantizer = std::make_shared<CachedQuantizer>(pair.second);
|
||||
cache::GpuCacheMgr::GetInstance(device_id)->InsertItem(key, cached_quantizer);
|
||||
}
|
||||
return Status::OK();
|
||||
}
|
||||
|
||||
auto index = std::static_pointer_cast<VecIndex>(cache::GpuCacheMgr::GetInstance(device_id)->GetIndex(location_));
|
||||
bool already_in_cache = (index != nullptr);
|
||||
if (already_in_cache) {
|
||||
index_ = index;
|
||||
} else {
|
||||
if (index_ == nullptr) {
|
||||
ENGINE_LOG_ERROR << "ExecutionEngineImpl: index is null, failed to copy to gpu";
|
||||
return Status(DB_ERROR, "index is null");
|
||||
}
|
||||
|
||||
try {
|
||||
index_ = index_->CopyToGpu(device_id);
|
||||
ENGINE_LOG_DEBUG << "CPU to GPU" << device_id;
|
||||
} catch (std::exception& e) {
|
||||
ENGINE_LOG_ERROR << e.what();
|
||||
return Status(DB_ERROR, e.what());
|
||||
}
|
||||
#endif
|
||||
try {
|
||||
index_ = index_->CopyToGpu(device_id);
|
||||
ENGINE_LOG_DEBUG << "CPU to GPU" << device_id;
|
||||
} catch (std::exception& e) {
|
||||
ENGINE_LOG_ERROR << e.what();
|
||||
return Status(DB_ERROR, e.what());
|
||||
}
|
||||
|
||||
if (!already_in_cache) {
|
||||
GpuCache(device_id);
|
||||
}
|
||||
|
||||
return Status::OK();
|
||||
}
|
||||
|
||||
|
||||
@ -244,11 +244,14 @@ if(CUSTOMIZATION)
|
||||
# set(FAISS_MD5 "21deb1c708490ca40ecb899122c01403") # commit-id 643e48f479637fd947e7b93fa4ca72b38ecc9a39 branch-0.2.0
|
||||
# set(FAISS_MD5 "072db398351cca6e88f52d743bbb9fa0") # commit-id 3a2344d04744166af41ef1a74449d68a315bfe17 branch-0.2.1
|
||||
# set(FAISS_MD5 "c89ea8e655f5cdf58f42486f13614714") # commit-id 9c28a1cbb88f41fa03b03d7204106201ad33276b branch-0.2.1
|
||||
set(FAISS_MD5 "87fdd86351ffcaf3f80dc26ade63c44b") # commit-id 841a156e67e8e22cd8088e1b58c00afbf2efc30b branch-0.2.1
|
||||
# set(FAISS_MD5 "87fdd86351ffcaf3f80dc26ade63c44b") # commit-id 841a156e67e8e22cd8088e1b58c00afbf2efc30b branch-0.2.1
|
||||
# set(FAISS_MD5 "f3b2ce3364c3fa7febd3aa7fdd0fe380") # commit-id 694e03458e6b69ce8a62502f71f69a614af5af8f branch-0.3.0
|
||||
# set(FAISS_MD5 "bb30722c22390ce5f6759ccb216c1b2a") # commit-id d324db297475286afe107847c7fb7a0f9dc7e90e branch-0.3.0
|
||||
set(FAISS_MD5 "2293cdb209c3718e3b19f3edae8b32b3") # commit-id a13c1205dc52977a9ad3b33a14efa958604a8bff branch-0.3.0
|
||||
endif()
|
||||
else()
|
||||
set(FAISS_SOURCE_URL "https://github.com/facebookresearch/faiss/archive/v1.5.3.tar.gz")
|
||||
set(FAISS_MD5 "0bc12737b23def156f6a1eb782050135")
|
||||
set(FAISS_SOURCE_URL "https://github.com/milvus-io/faiss/archive/1.6.0.tar.gz")
|
||||
set(FAISS_MD5 "eb96d84f98b078a9eec04a796f5c792e")
|
||||
endif()
|
||||
message(STATUS "FAISS URL = ${FAISS_SOURCE_URL}")
|
||||
|
||||
@ -299,12 +302,29 @@ macro(build_arrow)
|
||||
${EP_COMMON_CMAKE_ARGS}
|
||||
-DARROW_BUILD_STATIC=ON
|
||||
-DARROW_BUILD_SHARED=OFF
|
||||
-DARROW_PARQUET=OFF
|
||||
-DARROW_USE_GLOG=OFF
|
||||
-DCMAKE_INSTALL_PREFIX=${ARROW_PREFIX}
|
||||
"-DCMAKE_LIBRARY_PATH=${CUDA_TOOLKIT_ROOT_DIR}/lib64/stubs"
|
||||
-DARROW_CUDA=OFF
|
||||
-DARROW_FLIGHT=OFF
|
||||
-DARROW_GANDIVA=OFF
|
||||
-DARROW_GANDIVA_JAVA=OFF
|
||||
-DARROW_HDFS=OFF
|
||||
-DARROW_HIVESERVER2=OFF
|
||||
-DARROW_ORC=OFF
|
||||
-DARROW_PARQUET=OFF
|
||||
-DARROW_PLASMA=OFF
|
||||
-DARROW_PLASMA_JAVA_CLIENT=OFF
|
||||
-DARROW_PYTHON=OFF
|
||||
-DARROW_WITH_BZ2=OFF
|
||||
-DARROW_WITH_ZLIB=OFF
|
||||
-DARROW_WITH_LZ4=OFF
|
||||
-DARROW_WITH_SNAPPY=OFF
|
||||
-DARROW_WITH_ZSTD=OFF
|
||||
-DARROW_WITH_BROTLI=OFF
|
||||
-DCMAKE_BUILD_TYPE=Release
|
||||
-DARROW_DEPENDENCY_SOURCE=BUNDLED) #Build all arrow dependencies from source instead of calling find_package first
|
||||
-DARROW_DEPENDENCY_SOURCE=BUNDLED #Build all arrow dependencies from source instead of calling find_package first
|
||||
-DBOOST_SOURCE=AUTO #try to find BOOST in the system default locations and build from source if not found
|
||||
)
|
||||
|
||||
|
||||
if(USE_JFROG_CACHE STREQUAL "ON")
|
||||
|
||||
@ -81,27 +81,6 @@ target_link_libraries(
|
||||
${depend_libs}
|
||||
)
|
||||
|
||||
INSTALL(TARGETS
|
||||
knowhere
|
||||
SPTAGLibStatic
|
||||
DESTINATION
|
||||
lib)
|
||||
|
||||
INSTALL(FILES
|
||||
${ARROW_STATIC_LIB}
|
||||
${ARROW_PREFIX}/lib/libjemalloc_pic.a
|
||||
${FAISS_STATIC_LIB}
|
||||
${LAPACK_STATIC_LIB}
|
||||
${BLAS_STATIC_LIB}
|
||||
DESTINATION
|
||||
lib
|
||||
)
|
||||
|
||||
INSTALL(FILES ${OPENBLAS_REAL_STATIC_LIB}
|
||||
RENAME "libopenblas.a"
|
||||
DESTINATION lib
|
||||
)
|
||||
|
||||
set(INDEX_INCLUDE_DIRS
|
||||
${INDEX_SOURCE_DIR}/knowhere
|
||||
${INDEX_SOURCE_DIR}/thirdparty
|
||||
@ -112,17 +91,4 @@ set(INDEX_INCLUDE_DIRS
|
||||
${LAPACK_INCLUDE_DIR}
|
||||
)
|
||||
|
||||
set(INDEX_INCLUDE_DIRS ${INDEX_INCLUDE_DIRS} PARENT_SCOPE)
|
||||
|
||||
#INSTALL(DIRECTORY
|
||||
# ${INDEX_SOURCE_DIR}/include/knowhere
|
||||
# ${ARROW_INCLUDE_DIR}/arrow
|
||||
# ${FAISS_PREFIX}/include/faiss
|
||||
# ${OPENBLAS_INCLUDE_DIR}/
|
||||
# DESTINATION
|
||||
# include)
|
||||
#
|
||||
#INSTALL(DIRECTORY
|
||||
# ${SPTAG_SOURCE_DIR}/AnnService/inc/
|
||||
# DESTINATION
|
||||
# include/SPTAG/AnnService/inc)
|
||||
set(INDEX_INCLUDE_DIRS ${INDEX_INCLUDE_DIRS} PARENT_SCOPE)
|
||||
@ -17,7 +17,7 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include "utils/easylogging++.h"
|
||||
#include "external/easyloggingpp/easylogging++.h"
|
||||
|
||||
namespace knowhere {
|
||||
|
||||
|
||||
@ -38,7 +38,7 @@ class FaissBaseIndex {
|
||||
virtual void
|
||||
SealImpl();
|
||||
|
||||
protected:
|
||||
public:
|
||||
std::shared_ptr<faiss::Index> index_ = nullptr;
|
||||
};
|
||||
|
||||
|
||||
@ -15,12 +15,12 @@
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/gpu/GpuIndexFlat.h>
|
||||
#include <memory>
|
||||
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
#include <faiss/gpu/GpuIndexIVF.h>
|
||||
#include <faiss/gpu/GpuIndexIVFFlat.h>
|
||||
#include <faiss/index_io.h>
|
||||
#include <memory>
|
||||
|
||||
#include "knowhere/adapter/VectorAdapter.h"
|
||||
#include "knowhere/common/Exception.h"
|
||||
@ -86,7 +86,8 @@ GPUIVF::SerializeImpl() {
|
||||
faiss::Index* index = index_.get();
|
||||
faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(index);
|
||||
|
||||
SealImpl();
|
||||
// TODO(linxj): support seal
|
||||
// SealImpl();
|
||||
|
||||
faiss::write_index(host_index, &writer);
|
||||
delete host_index;
|
||||
@ -130,13 +131,12 @@ void
|
||||
GPUIVF::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) {
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
|
||||
// TODO(linxj): gpu index support GenParams
|
||||
if (auto device_index = std::dynamic_pointer_cast<faiss::gpu::GpuIndexIVF>(index_)) {
|
||||
auto search_cfg = std::dynamic_pointer_cast<IVFCfg>(cfg);
|
||||
device_index->setNumProbes(search_cfg->nprobe);
|
||||
device_index->nprobe = search_cfg->nprobe;
|
||||
// assert(device_index->getNumProbes() == search_cfg->nprobe);
|
||||
|
||||
{
|
||||
// TODO(linxj): allocate gpu mem
|
||||
ResScope rs(res_, gpu_id_);
|
||||
device_index->search(n, (float*)data, k, distances, labels);
|
||||
}
|
||||
|
||||
@ -16,8 +16,10 @@
|
||||
// under the License.
|
||||
|
||||
#include <faiss/IndexIVFPQ.h>
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
#include <faiss/gpu/GpuIndexIVFPQ.h>
|
||||
#include <faiss/index_factory.h>
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "knowhere/adapter/VectorAdapter.h"
|
||||
|
||||
@ -15,9 +15,10 @@
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
#include <faiss/index_factory.h>
|
||||
|
||||
#include <memory>
|
||||
#include <utility>
|
||||
|
||||
#include "knowhere/adapter/VectorAdapter.h"
|
||||
#include "knowhere/common/Exception.h"
|
||||
@ -71,13 +72,4 @@ GPUIVFSQ::CopyGpuToCpu(const Config& config) {
|
||||
return std::make_shared<IVFSQ>(new_index);
|
||||
}
|
||||
|
||||
void
|
||||
GPUIVFSQ::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) {
|
||||
#ifdef CUSTOMIZATION
|
||||
GPUIVF::search_impl(n, data, k, distances, labels, cfg);
|
||||
#else
|
||||
IVF::search_impl(n, data, k, distances, labels, cfg);
|
||||
#endif
|
||||
}
|
||||
|
||||
} // namespace knowhere
|
||||
|
||||
@ -38,10 +38,6 @@ class GPUIVFSQ : public GPUIVF {
|
||||
|
||||
VectorIndexPtr
|
||||
CopyGpuToCpu(const Config& config) override;
|
||||
|
||||
protected:
|
||||
void
|
||||
search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels, const Config& cfg) override;
|
||||
};
|
||||
|
||||
} // namespace knowhere
|
||||
|
||||
@ -15,11 +15,12 @@
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include <faiss/AutoTune.h>
|
||||
#include <faiss/IndexFlat.h>
|
||||
#include <faiss/MetaIndexes.h>
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
#include <faiss/index_factory.h>
|
||||
#include <faiss/index_io.h>
|
||||
|
||||
#include <vector>
|
||||
|
||||
#include "knowhere/adapter/VectorAdapter.h"
|
||||
|
||||
@ -15,15 +15,12 @@
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include <faiss/AutoTune.h>
|
||||
#include <faiss/AuxIndexStructures.h>
|
||||
#include <faiss/IVFlib.h>
|
||||
#include <faiss/IndexFlat.h>
|
||||
#include <faiss/IndexIVF.h>
|
||||
#include <faiss/IndexIVFFlat.h>
|
||||
#include <faiss/IndexIVFPQ.h>
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/index_io.h>
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
|
||||
#include <chrono>
|
||||
#include <memory>
|
||||
#include <utility>
|
||||
@ -224,9 +221,10 @@ IVF::search_impl(int64_t n, const float* data, int64_t k, float* distances, int6
|
||||
faiss::ivflib::search_with_parameters(index_.get(), n, (float*)data, k, distances, labels, params.get());
|
||||
stdclock::time_point after = stdclock::now();
|
||||
double search_cost = (std::chrono::duration<double, std::micro>(after - before)).count();
|
||||
KNOWHERE_LOG_DEBUG << "IVF search cost: " << search_cost
|
||||
<< ", quantization cost: " << faiss::indexIVF_stats.quantization_time
|
||||
<< ", data search cost: " << faiss::indexIVF_stats.search_time;
|
||||
KNOWHERE_LOG_DEBUG << "K=" << k << " NQ=" << n << " NL=" << faiss::indexIVF_stats.nlist
|
||||
<< " ND=" << faiss::indexIVF_stats.ndis << " NH=" << faiss::indexIVF_stats.nheap_updates
|
||||
<< " Q=" << faiss::indexIVF_stats.quantization_time
|
||||
<< " S=" << faiss::indexIVF_stats.search_time;
|
||||
faiss::indexIVF_stats.quantization_time = 0;
|
||||
faiss::indexIVF_stats.search_time = 0;
|
||||
}
|
||||
|
||||
@ -30,7 +30,7 @@ namespace knowhere {
|
||||
|
||||
using Graph = std::vector<std::vector<int64_t>>;
|
||||
|
||||
class IVF : public VectorIndex, protected FaissBaseIndex {
|
||||
class IVF : public VectorIndex, public FaissBaseIndex {
|
||||
public:
|
||||
IVF() : FaissBaseIndex(nullptr) {
|
||||
}
|
||||
|
||||
@ -15,7 +15,8 @@
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
#include <faiss/index_factory.h>
|
||||
#include <memory>
|
||||
|
||||
#include "knowhere/adapter/VectorAdapter.h"
|
||||
@ -56,14 +57,7 @@ IVFSQ::CopyCpuToGpu(const int64_t& device_id, const Config& config) {
|
||||
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
|
||||
ResScope rs(res, device_id, false);
|
||||
|
||||
#ifdef CUSTOMIZATION
|
||||
faiss::gpu::GpuClonerOptions option;
|
||||
option.allInGpu = true;
|
||||
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, index_.get(), &option);
|
||||
#else
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, index_.get());
|
||||
#endif
|
||||
|
||||
std::shared_ptr<faiss::Index> device_index;
|
||||
device_index.reset(gpu_index);
|
||||
|
||||
@ -17,19 +17,25 @@
|
||||
// under the License.
|
||||
|
||||
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
|
||||
#include <utility>
|
||||
#include "faiss/AutoTune.h"
|
||||
#include "faiss/gpu/GpuAutoTune.h"
|
||||
#include "faiss/gpu/GpuIndexIVF.h"
|
||||
#include "knowhere/adapter/VectorAdapter.h"
|
||||
#include "knowhere/common/Exception.h"
|
||||
|
||||
#include <utility>
|
||||
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
#include <faiss/gpu/GpuIndexIVF.h>
|
||||
#include <faiss/index_factory.h>
|
||||
|
||||
namespace knowhere {
|
||||
|
||||
#ifdef CUSTOMIZATION
|
||||
|
||||
// std::mutex g_mutex;
|
||||
|
||||
IndexModelPtr
|
||||
IVFSQHybrid::Train(const DatasetPtr& dataset, const Config& config) {
|
||||
// std::lock_guard<std::mutex> lk(g_mutex);
|
||||
|
||||
auto build_cfg = std::dynamic_pointer_cast<IVFSQCfg>(config);
|
||||
if (build_cfg != nullptr) {
|
||||
build_cfg->CheckValid(); // throw exception
|
||||
@ -63,19 +69,17 @@ IVFSQHybrid::Train(const DatasetPtr& dataset, const Config& config) {
|
||||
|
||||
VectorIndexPtr
|
||||
IVFSQHybrid::CopyGpuToCpu(const Config& config) {
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
|
||||
if (auto device_idx = std::dynamic_pointer_cast<faiss::IndexIVF>(index_)) {
|
||||
faiss::Index* device_index = index_.get();
|
||||
faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(device_index);
|
||||
|
||||
std::shared_ptr<faiss::Index> new_index;
|
||||
new_index.reset(host_index);
|
||||
return std::make_shared<IVFSQHybrid>(new_index);
|
||||
} else {
|
||||
// TODO(linxj): why? jinhai
|
||||
if (gpu_mode == 0) {
|
||||
return std::make_shared<IVFSQHybrid>(index_);
|
||||
}
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
|
||||
faiss::Index* device_index = index_.get();
|
||||
faiss::Index* host_index = faiss::gpu::index_gpu_to_cpu(device_index);
|
||||
|
||||
std::shared_ptr<faiss::Index> new_index;
|
||||
new_index.reset(host_index);
|
||||
return std::make_shared<IVFSQHybrid>(new_index);
|
||||
}
|
||||
|
||||
VectorIndexPtr
|
||||
@ -85,13 +89,9 @@ IVFSQHybrid::CopyCpuToGpu(const int64_t& device_id, const Config& config) {
|
||||
faiss::gpu::GpuClonerOptions option;
|
||||
option.allInGpu = true;
|
||||
|
||||
faiss::IndexComposition index_composition;
|
||||
index_composition.index = index_.get();
|
||||
index_composition.quantizer = nullptr;
|
||||
index_composition.mode = 0; // copy all
|
||||
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, &index_composition, &option);
|
||||
|
||||
auto idx = dynamic_cast<faiss::IndexIVF*>(index_.get());
|
||||
idx->restore_quantizer();
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), device_id, index_.get(), &option);
|
||||
std::shared_ptr<faiss::Index> device_index = std::shared_ptr<faiss::Index>(gpu_index);
|
||||
auto new_idx = std::make_shared<IVFSQHybrid>(device_index, device_id, res);
|
||||
return new_idx;
|
||||
@ -105,16 +105,26 @@ IVFSQHybrid::LoadImpl(const BinarySet& index_binary) {
|
||||
FaissBaseIndex::LoadImpl(index_binary); // load on cpu
|
||||
auto* ivf_index = dynamic_cast<faiss::IndexIVF*>(index_.get());
|
||||
ivf_index->backup_quantizer();
|
||||
gpu_mode = 0;
|
||||
}
|
||||
|
||||
void
|
||||
IVFSQHybrid::search_impl(int64_t n, const float* data, int64_t k, float* distances, int64_t* labels,
|
||||
const Config& cfg) {
|
||||
// std::lock_guard<std::mutex> lk(g_mutex);
|
||||
// static int64_t search_count;
|
||||
// ++search_count;
|
||||
|
||||
if (gpu_mode == 2) {
|
||||
GPUIVF::search_impl(n, data, k, distances, labels, cfg);
|
||||
} else if (gpu_mode == 1) {
|
||||
ResScope rs(res_, gpu_id_);
|
||||
IVF::search_impl(n, data, k, distances, labels, cfg);
|
||||
// index_->search(n, (float*)data, k, distances, labels);
|
||||
} else if (gpu_mode == 1) { // hybrid
|
||||
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(quantizer_gpu_id_)) {
|
||||
ResScope rs(res, quantizer_gpu_id_, true);
|
||||
IVF::search_impl(n, data, k, distances, labels, cfg);
|
||||
} else {
|
||||
KNOWHERE_THROW_MSG("Hybrid Search Error, can't get gpu: " + std::to_string(quantizer_gpu_id_) + "resource");
|
||||
}
|
||||
} else if (gpu_mode == 0) {
|
||||
IVF::search_impl(n, data, k, distances, labels, cfg);
|
||||
}
|
||||
@ -122,16 +132,18 @@ IVFSQHybrid::search_impl(int64_t n, const float* data, int64_t k, float* distanc
|
||||
|
||||
QuantizerPtr
|
||||
IVFSQHybrid::LoadQuantizer(const Config& conf) {
|
||||
// std::lock_guard<std::mutex> lk(g_mutex);
|
||||
|
||||
auto quantizer_conf = std::dynamic_pointer_cast<QuantizerCfg>(conf);
|
||||
if (quantizer_conf != nullptr) {
|
||||
if (quantizer_conf->mode != 1) {
|
||||
KNOWHERE_THROW_MSG("mode only support 1 in this func");
|
||||
}
|
||||
}
|
||||
gpu_id_ = quantizer_conf->gpu_id;
|
||||
auto gpu_id = quantizer_conf->gpu_id;
|
||||
|
||||
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
|
||||
ResScope rs(res, gpu_id_, false);
|
||||
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id)) {
|
||||
ResScope rs(res, gpu_id, false);
|
||||
faiss::gpu::GpuClonerOptions option;
|
||||
option.allInGpu = true;
|
||||
|
||||
@ -140,7 +152,7 @@ IVFSQHybrid::LoadQuantizer(const Config& conf) {
|
||||
index_composition->quantizer = nullptr;
|
||||
index_composition->mode = quantizer_conf->mode; // only 1
|
||||
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id_, index_composition, &option);
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id, index_composition, &option);
|
||||
delete gpu_index;
|
||||
|
||||
auto q = std::make_shared<FaissIVFQuantizer>();
|
||||
@ -148,16 +160,19 @@ IVFSQHybrid::LoadQuantizer(const Config& conf) {
|
||||
auto& q_ptr = index_composition->quantizer;
|
||||
q->size = q_ptr->d * q_ptr->getNumVecs() * sizeof(float);
|
||||
q->quantizer = q_ptr;
|
||||
q->gpu_id = gpu_id;
|
||||
res_ = res;
|
||||
gpu_mode = 1;
|
||||
return q;
|
||||
} else {
|
||||
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
|
||||
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id) + "resource");
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
IVFSQHybrid::SetQuantizer(const QuantizerPtr& q) {
|
||||
// std::lock_guard<std::mutex> lk(g_mutex);
|
||||
|
||||
auto ivf_quantizer = std::dynamic_pointer_cast<FaissIVFQuantizer>(q);
|
||||
if (ivf_quantizer == nullptr) {
|
||||
KNOWHERE_THROW_MSG("Quantizer type error");
|
||||
@ -170,20 +185,27 @@ IVFSQHybrid::SetQuantizer(const QuantizerPtr& q) {
|
||||
// delete ivf_index->quantizer;
|
||||
ivf_index->quantizer = ivf_quantizer->quantizer;
|
||||
}
|
||||
quantizer_gpu_id_ = ivf_quantizer->gpu_id;
|
||||
gpu_mode = 1;
|
||||
}
|
||||
|
||||
void
|
||||
IVFSQHybrid::UnsetQuantizer() {
|
||||
// std::lock_guard<std::mutex> lk(g_mutex);
|
||||
|
||||
auto* ivf_index = dynamic_cast<faiss::IndexIVF*>(index_.get());
|
||||
if (ivf_index == nullptr) {
|
||||
KNOWHERE_THROW_MSG("Index type error");
|
||||
}
|
||||
|
||||
ivf_index->quantizer = nullptr;
|
||||
quantizer_gpu_id_ = -1;
|
||||
}
|
||||
|
||||
VectorIndexPtr
|
||||
IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
|
||||
// std::lock_guard<std::mutex> lk(g_mutex);
|
||||
|
||||
auto quantizer_conf = std::dynamic_pointer_cast<QuantizerCfg>(conf);
|
||||
if (quantizer_conf != nullptr) {
|
||||
if (quantizer_conf->mode != 2) {
|
||||
@ -192,13 +214,11 @@ IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
|
||||
} else {
|
||||
KNOWHERE_THROW_MSG("conf error");
|
||||
}
|
||||
// if (quantizer_conf->gpu_id != gpu_id_) {
|
||||
// KNOWHERE_THROW_MSG("quantizer and data must on the same gpu card");
|
||||
// }
|
||||
gpu_id_ = quantizer_conf->gpu_id;
|
||||
|
||||
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
|
||||
ResScope rs(res, gpu_id_, false);
|
||||
auto gpu_id = quantizer_conf->gpu_id;
|
||||
|
||||
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id)) {
|
||||
ResScope rs(res, gpu_id, false);
|
||||
faiss::gpu::GpuClonerOptions option;
|
||||
option.allInGpu = true;
|
||||
|
||||
@ -211,18 +231,20 @@ IVFSQHybrid::LoadData(const knowhere::QuantizerPtr& q, const Config& conf) {
|
||||
index_composition->quantizer = ivf_quantizer->quantizer;
|
||||
index_composition->mode = quantizer_conf->mode; // only 2
|
||||
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id_, index_composition, &option);
|
||||
auto gpu_index = faiss::gpu::index_cpu_to_gpu(res->faiss_res.get(), gpu_id, index_composition, &option);
|
||||
std::shared_ptr<faiss::Index> new_idx;
|
||||
new_idx.reset(gpu_index);
|
||||
auto sq_idx = std::make_shared<IVFSQHybrid>(new_idx, gpu_id_, res);
|
||||
auto sq_idx = std::make_shared<IVFSQHybrid>(new_idx, gpu_id, res);
|
||||
return sq_idx;
|
||||
} else {
|
||||
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
|
||||
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id) + "resource");
|
||||
}
|
||||
}
|
||||
|
||||
std::pair<VectorIndexPtr, QuantizerPtr>
|
||||
IVFSQHybrid::CopyCpuToGpuWithQuantizer(const int64_t& device_id, const Config& config) {
|
||||
// std::lock_guard<std::mutex> lk(g_mutex);
|
||||
|
||||
if (auto res = FaissGpuResourceMgr::GetInstance().GetRes(device_id)) {
|
||||
ResScope rs(res, device_id, false);
|
||||
faiss::gpu::GpuClonerOptions option;
|
||||
@ -242,12 +264,29 @@ IVFSQHybrid::CopyCpuToGpuWithQuantizer(const int64_t& device_id, const Config& c
|
||||
auto q = std::make_shared<FaissIVFQuantizer>();
|
||||
q->quantizer = index_composition.quantizer;
|
||||
q->size = index_composition.quantizer->d * index_composition.quantizer->getNumVecs() * sizeof(float);
|
||||
q->gpu_id = device_id;
|
||||
return std::make_pair(new_idx, q);
|
||||
} else {
|
||||
KNOWHERE_THROW_MSG("CopyCpuToGpu Error, can't get gpu: " + std::to_string(gpu_id_) + "resource");
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
IVFSQHybrid::set_index_model(IndexModelPtr model) {
|
||||
std::lock_guard<std::mutex> lk(mutex_);
|
||||
|
||||
auto host_index = std::static_pointer_cast<IVFIndexModel>(model);
|
||||
if (auto gpures = FaissGpuResourceMgr::GetInstance().GetRes(gpu_id_)) {
|
||||
ResScope rs(gpures, gpu_id_, false);
|
||||
auto device_index = faiss::gpu::index_cpu_to_gpu(gpures->faiss_res.get(), gpu_id_, host_index->index_.get());
|
||||
index_.reset(device_index);
|
||||
res_ = gpures;
|
||||
gpu_mode = 2;
|
||||
} else {
|
||||
KNOWHERE_THROW_MSG("load index model error, can't get gpu_resource");
|
||||
}
|
||||
}
|
||||
|
||||
FaissIVFQuantizer::~FaissIVFQuantizer() {
|
||||
if (quantizer != nullptr) {
|
||||
delete quantizer;
|
||||
@ -307,5 +346,10 @@ IVFSQHybrid::LoadImpl(const BinarySet& index_binary) {
|
||||
GPUIVF::LoadImpl(index_binary);
|
||||
}
|
||||
|
||||
void
|
||||
IVFSQHybrid::set_index_model(IndexModelPtr model) {
|
||||
GPUIVF::set_index_model(model);
|
||||
}
|
||||
|
||||
#endif
|
||||
} // namespace knowhere
|
||||
|
||||
@ -17,7 +17,9 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <faiss/gpu/GpuIndexFlat.h>
|
||||
#include <faiss/index_io.h>
|
||||
|
||||
#include <memory>
|
||||
#include <utility>
|
||||
|
||||
@ -29,6 +31,7 @@ namespace knowhere {
|
||||
#ifdef CUSTOMIZATION
|
||||
struct FaissIVFQuantizer : public Quantizer {
|
||||
faiss::gpu::GpuIndexFlat* quantizer = nullptr;
|
||||
int64_t gpu_id;
|
||||
|
||||
~FaissIVFQuantizer() override;
|
||||
};
|
||||
@ -52,6 +55,9 @@ class IVFSQHybrid : public GPUIVFSQ {
|
||||
}
|
||||
|
||||
public:
|
||||
void
|
||||
set_index_model(IndexModelPtr model) override;
|
||||
|
||||
QuantizerPtr
|
||||
LoadQuantizer(const Config& conf);
|
||||
|
||||
@ -85,6 +91,7 @@ class IVFSQHybrid : public GPUIVFSQ {
|
||||
|
||||
protected:
|
||||
int64_t gpu_mode = 0; // 0,1,2
|
||||
int64_t quantizer_gpu_id_ = -1;
|
||||
};
|
||||
|
||||
} // namespace knowhere
|
||||
|
||||
@ -48,6 +48,7 @@ class VectorIndex : public Index {
|
||||
virtual void
|
||||
Seal() = 0;
|
||||
|
||||
// TODO(linxj): Deprecated
|
||||
virtual VectorIndexPtr
|
||||
Clone() = 0;
|
||||
|
||||
|
||||
@ -17,7 +17,7 @@
|
||||
|
||||
#pragma once
|
||||
|
||||
#include <faiss/AuxIndexStructures.h>
|
||||
#include <faiss/impl/io.h>
|
||||
|
||||
namespace knowhere {
|
||||
|
||||
|
||||
2
core/src/index/thirdparty/versions.txt
vendored
2
core/src/index/thirdparty/versions.txt
vendored
@ -3,4 +3,4 @@ BOOST_VERSION=1.70.0
|
||||
GTEST_VERSION=1.8.1
|
||||
LAPACK_VERSION=v3.8.0
|
||||
OPENBLAS_VERSION=v0.3.6
|
||||
FAISS_VERSION=branch-0.2.1
|
||||
FAISS_VERSION=branch-0.3.0
|
||||
@ -20,7 +20,7 @@ set(basic_libs
|
||||
)
|
||||
|
||||
set(util_srcs
|
||||
${MILVUS_ENGINE_SRC}/utils/easylogging++.cc
|
||||
${MILVUS_ENGINE_SRC}/external/easyloggingpp/easylogging++.cc
|
||||
${INDEX_SOURCE_DIR}/knowhere/knowhere/index/vector_index/helpers/FaissGpuResourceMgr.cpp
|
||||
${INDEX_SOURCE_DIR}/knowhere/knowhere/index/vector_index/helpers/FaissIO.cpp
|
||||
${INDEX_SOURCE_DIR}/knowhere/knowhere/index/vector_index/helpers/IndexParameter.cpp
|
||||
@ -86,5 +86,6 @@ install(TARGETS test_gpuresource DESTINATION unittest)
|
||||
install(TARGETS test_customized_index DESTINATION unittest)
|
||||
|
||||
#add_subdirectory(faiss_ori)
|
||||
#add_subdirectory(faiss_benchmark)
|
||||
add_subdirectory(test_nsg)
|
||||
|
||||
|
||||
@ -26,7 +26,7 @@
|
||||
#include "knowhere/index/vector_index/IndexIVFSQ.h"
|
||||
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
|
||||
|
||||
constexpr int DEVICEID = 0;
|
||||
int DEVICEID = 0;
|
||||
constexpr int64_t DIM = 128;
|
||||
constexpr int64_t NB = 10000;
|
||||
constexpr int64_t NQ = 10;
|
||||
|
||||
24
core/src/index/unittest/faiss_benchmark/CMakeLists.txt
Normal file
24
core/src/index/unittest/faiss_benchmark/CMakeLists.txt
Normal file
@ -0,0 +1,24 @@
|
||||
include_directories(${INDEX_SOURCE_DIR}/thirdparty)
|
||||
include_directories(${INDEX_SOURCE_DIR}/include)
|
||||
include_directories(/usr/local/cuda/include)
|
||||
include_directories(/usr/local/hdf5/include)
|
||||
|
||||
link_directories(/usr/local/cuda/lib64)
|
||||
link_directories(/usr/local/hdf5/lib)
|
||||
|
||||
set(unittest_libs
|
||||
gtest gmock gtest_main gmock_main)
|
||||
|
||||
set(depend_libs
|
||||
faiss openblas lapack hdf5
|
||||
arrow ${ARROW_PREFIX}/lib/libjemalloc_pic.a
|
||||
)
|
||||
|
||||
set(basic_libs
|
||||
cudart cublas
|
||||
gomp gfortran pthread
|
||||
)
|
||||
|
||||
add_executable(test_faiss_benchmark faiss_benchmark_test.cpp)
|
||||
target_link_libraries(test_faiss_benchmark ${depend_libs} ${unittest_libs} ${basic_libs})
|
||||
install(TARGETS test_faiss_benchmark DESTINATION unittest)
|
||||
25
core/src/index/unittest/faiss_benchmark/README.md
Normal file
25
core/src/index/unittest/faiss_benchmark/README.md
Normal file
@ -0,0 +1,25 @@
|
||||
### To run this FAISS benchmark, please follow these steps:
|
||||
|
||||
#### Step 1:
|
||||
Download the HDF5 source from:
|
||||
https://support.hdfgroup.org/ftp/HDF5/releases/
|
||||
and build/install to "/usr/local/hdf5".
|
||||
|
||||
#### Step 2:
|
||||
Download HDF5 data files from:
|
||||
https://github.com/erikbern/ann-benchmarks
|
||||
|
||||
#### Step 3:
|
||||
Update 'milvus/core/src/index/unittest/CMakeLists.txt',
|
||||
uncomment "#add_subdirectory(faiss_benchmark)".
|
||||
|
||||
#### Step 4:
|
||||
Build Milvus with unittest enabled: "./build.sh -t Release -u",
|
||||
binary 'test_faiss_benchmark' will be generated.
|
||||
|
||||
#### Step 5:
|
||||
Put HDF5 data files into the same directory with binary 'test_faiss_benchmark'.
|
||||
|
||||
#### Step 6:
|
||||
Run test binary 'test_faiss_benchmark'.
|
||||
|
||||
571
core/src/index/unittest/faiss_benchmark/faiss_benchmark_test.cpp
Normal file
571
core/src/index/unittest/faiss_benchmark/faiss_benchmark_test.cpp
Normal file
@ -0,0 +1,571 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <hdf5.h>
|
||||
#include <sys/time.h>
|
||||
#include <sys/types.h>
|
||||
#include <unistd.h>
|
||||
#include <cassert>
|
||||
#include <cmath>
|
||||
#include <cstdio>
|
||||
#include <vector>
|
||||
|
||||
#define USE_FAISS_V1_5_3 0
|
||||
|
||||
#if USE_FAISS_V1_5_3
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/utils.h>
|
||||
#include <sys/stat.h>
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
|
||||
#else
|
||||
#include <faiss/gpu/GpuCloner.h>
|
||||
#include <faiss/index_factory.h>
|
||||
#include <faiss/utils/distances.h>
|
||||
#endif
|
||||
|
||||
#include <faiss/AutoTune.h>
|
||||
#include <faiss/Index.h>
|
||||
#include <faiss/IndexIVF.h>
|
||||
#include <faiss/gpu/GpuIndexFlat.h>
|
||||
#include <faiss/gpu/StandardGpuResources.h>
|
||||
#include <faiss/index_io.h>
|
||||
|
||||
#ifdef CUSTOMIZATION
|
||||
#include <faiss/gpu/GpuIndexIVFSQHybrid.h>
|
||||
#endif
|
||||
|
||||
/*****************************************************
|
||||
* To run this test, please download the HDF5 from
|
||||
* https://support.hdfgroup.org/ftp/HDF5/releases/
|
||||
* and install it to /usr/local/hdf5 .
|
||||
*****************************************************/
|
||||
#define DEBUG_VERBOSE 0
|
||||
|
||||
const char HDF5_POSTFIX[] = ".hdf5";
|
||||
const char HDF5_DATASET_TRAIN[] = "train";
|
||||
const char HDF5_DATASET_TEST[] = "test";
|
||||
const char HDF5_DATASET_NEIGHBORS[] = "neighbors";
|
||||
const char HDF5_DATASET_DISTANCES[] = "distances";
|
||||
|
||||
const int32_t GPU_DEVICE_IDX = 0;
|
||||
|
||||
enum QueryMode { MODE_CPU = 0, MODE_MIX, MODE_GPU };
|
||||
|
||||
double
|
||||
elapsed() {
|
||||
struct timeval tv;
|
||||
gettimeofday(&tv, nullptr);
|
||||
return tv.tv_sec + tv.tv_usec * 1e-6;
|
||||
}
|
||||
|
||||
void
|
||||
normalize(float* arr, size_t nq, size_t dim) {
|
||||
for (size_t i = 0; i < nq; i++) {
|
||||
double vecLen = 0.0, inv_vecLen = 0.0;
|
||||
for (size_t j = 0; j < dim; j++) {
|
||||
double val = arr[i * dim + j];
|
||||
vecLen += val * val;
|
||||
}
|
||||
inv_vecLen = 1.0 / std::sqrt(vecLen);
|
||||
for (size_t j = 0; j < dim; j++) {
|
||||
arr[i * dim + j] = (float)(arr[i * dim + j] * inv_vecLen);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void*
|
||||
hdf5_read(const std::string& file_name, const std::string& dataset_name, H5T_class_t dataset_class, size_t& d_out,
|
||||
size_t& n_out) {
|
||||
hid_t file, dataset, datatype, dataspace, memspace;
|
||||
H5T_class_t t_class; /* data type class */
|
||||
hsize_t dimsm[3]; /* memory space dimensions */
|
||||
hsize_t dims_out[2]; /* dataset dimensions */
|
||||
hsize_t count[2]; /* size of the hyperslab in the file */
|
||||
hsize_t offset[2]; /* hyperslab offset in the file */
|
||||
hsize_t count_out[3]; /* size of the hyperslab in memory */
|
||||
hsize_t offset_out[3]; /* hyperslab offset in memory */
|
||||
void* data_out; /* output buffer */
|
||||
|
||||
/* Open the file and the dataset. */
|
||||
file = H5Fopen(file_name.c_str(), H5F_ACC_RDONLY, H5P_DEFAULT);
|
||||
dataset = H5Dopen2(file, dataset_name.c_str(), H5P_DEFAULT);
|
||||
|
||||
/* Get datatype and dataspace handles and then query
|
||||
* dataset class, order, size, rank and dimensions. */
|
||||
datatype = H5Dget_type(dataset); /* datatype handle */
|
||||
t_class = H5Tget_class(datatype);
|
||||
assert(t_class == dataset_class || !"Illegal dataset class type");
|
||||
|
||||
dataspace = H5Dget_space(dataset); /* dataspace handle */
|
||||
H5Sget_simple_extent_dims(dataspace, dims_out, NULL);
|
||||
n_out = dims_out[0];
|
||||
d_out = dims_out[1];
|
||||
|
||||
/* Define hyperslab in the dataset. */
|
||||
offset[0] = offset[1] = 0;
|
||||
count[0] = dims_out[0];
|
||||
count[1] = dims_out[1];
|
||||
H5Sselect_hyperslab(dataspace, H5S_SELECT_SET, offset, NULL, count, NULL);
|
||||
|
||||
/* Define the memory dataspace. */
|
||||
dimsm[0] = dims_out[0];
|
||||
dimsm[1] = dims_out[1];
|
||||
dimsm[2] = 1;
|
||||
memspace = H5Screate_simple(3, dimsm, NULL);
|
||||
|
||||
/* Define memory hyperslab. */
|
||||
offset_out[0] = offset_out[1] = offset_out[2] = 0;
|
||||
count_out[0] = dims_out[0];
|
||||
count_out[1] = dims_out[1];
|
||||
count_out[2] = 1;
|
||||
H5Sselect_hyperslab(memspace, H5S_SELECT_SET, offset_out, NULL, count_out, NULL);
|
||||
|
||||
/* Read data from hyperslab in the file into the hyperslab in memory and display. */
|
||||
switch (t_class) {
|
||||
case H5T_INTEGER:
|
||||
data_out = new int[dims_out[0] * dims_out[1]];
|
||||
H5Dread(dataset, H5T_NATIVE_INT, memspace, dataspace, H5P_DEFAULT, data_out);
|
||||
break;
|
||||
case H5T_FLOAT:
|
||||
data_out = new float[dims_out[0] * dims_out[1]];
|
||||
H5Dread(dataset, H5T_NATIVE_FLOAT, memspace, dataspace, H5P_DEFAULT, data_out);
|
||||
break;
|
||||
default:
|
||||
printf("Illegal dataset class type\n");
|
||||
break;
|
||||
}
|
||||
|
||||
/* Close/release resources. */
|
||||
H5Tclose(datatype);
|
||||
H5Dclose(dataset);
|
||||
H5Sclose(dataspace);
|
||||
H5Sclose(memspace);
|
||||
H5Fclose(file);
|
||||
|
||||
return data_out;
|
||||
}
|
||||
|
||||
std::string
|
||||
get_index_file_name(const std::string& ann_test_name, const std::string& index_key, int32_t data_loops) {
|
||||
size_t pos = index_key.find_first_of(',', 0);
|
||||
std::string file_name = ann_test_name;
|
||||
file_name = file_name + "_" + index_key.substr(0, pos) + "_" + index_key.substr(pos + 1);
|
||||
file_name = file_name + "_" + std::to_string(data_loops) + ".index";
|
||||
return file_name;
|
||||
}
|
||||
|
||||
bool
|
||||
parse_ann_test_name(const std::string& ann_test_name, size_t& dim, faiss::MetricType& metric_type) {
|
||||
size_t pos1, pos2;
|
||||
|
||||
if (ann_test_name.empty())
|
||||
return false;
|
||||
|
||||
pos1 = ann_test_name.find_first_of('-', 0);
|
||||
if (pos1 == std::string::npos)
|
||||
return false;
|
||||
pos2 = ann_test_name.find_first_of('-', pos1 + 1);
|
||||
if (pos2 == std::string::npos)
|
||||
return false;
|
||||
|
||||
dim = std::stoi(ann_test_name.substr(pos1 + 1, pos2 - pos1 - 1));
|
||||
std::string metric_str = ann_test_name.substr(pos2 + 1);
|
||||
if (metric_str == "angular") {
|
||||
metric_type = faiss::METRIC_INNER_PRODUCT;
|
||||
} else if (metric_str == "euclidean") {
|
||||
metric_type = faiss::METRIC_L2;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
int32_t
|
||||
GetResultHitCount(const faiss::Index::idx_t* ground_index, const faiss::Index::idx_t* index, size_t ground_k, size_t k,
|
||||
size_t nq, int32_t index_add_loops) {
|
||||
assert(ground_k <= k);
|
||||
int hit = 0;
|
||||
for (int i = 0; i < nq; i++) {
|
||||
// count the num of results exist in ground truth result set
|
||||
// each result replicates INDEX_ADD_LOOPS times
|
||||
for (int j_c = 0; j_c < k; j_c++) {
|
||||
int r_c = index[i * k + j_c];
|
||||
for (int j_g = 0; j_g < ground_k / index_add_loops; j_g++) {
|
||||
if (ground_index[i * ground_k + j_g] == r_c) {
|
||||
hit++;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return hit;
|
||||
}
|
||||
|
||||
#if DEBUG_VERBOSE
|
||||
void
|
||||
print_array(const char* header, bool is_integer, const void* arr, size_t nq, size_t k) {
|
||||
const int ROW = 10;
|
||||
const int COL = 10;
|
||||
assert(ROW <= nq);
|
||||
assert(COL <= k);
|
||||
printf("%s\n", header);
|
||||
printf("==============================================\n");
|
||||
for (int i = 0; i < 10; i++) {
|
||||
for (int j = 0; j < 10; j++) {
|
||||
if (is_integer) {
|
||||
printf("%7ld ", ((int64_t*)arr)[i * k + j]);
|
||||
} else {
|
||||
printf("%.6f ", ((float*)arr)[i * k + j]);
|
||||
}
|
||||
}
|
||||
printf("\n");
|
||||
}
|
||||
printf("\n");
|
||||
}
|
||||
#endif
|
||||
|
||||
void
|
||||
load_base_data(faiss::Index*& index, const std::string& ann_test_name, const std::string& index_key,
|
||||
faiss::gpu::StandardGpuResources& res, const faiss::MetricType metric_type, const size_t dim,
|
||||
int32_t index_add_loops, QueryMode mode = MODE_CPU) {
|
||||
double t0 = elapsed();
|
||||
|
||||
const std::string ann_file_name = ann_test_name + HDF5_POSTFIX;
|
||||
|
||||
faiss::Index *cpu_index = nullptr, *gpu_index = nullptr;
|
||||
faiss::distance_compute_blas_threshold = 800;
|
||||
|
||||
std::string index_file_name = get_index_file_name(ann_test_name, index_key, index_add_loops);
|
||||
|
||||
try {
|
||||
printf("[%.3f s] Reading index file: %s\n", elapsed() - t0, index_file_name.c_str());
|
||||
cpu_index = faiss::read_index(index_file_name.c_str());
|
||||
} catch (...) {
|
||||
size_t nb, d;
|
||||
printf("[%.3f s] Loading HDF5 file: %s\n", elapsed() - t0, ann_file_name.c_str());
|
||||
float* xb = (float*)hdf5_read(ann_file_name, HDF5_DATASET_TRAIN, H5T_FLOAT, d, nb);
|
||||
assert(d == dim || !"dataset does not have correct dimension");
|
||||
|
||||
if (metric_type == faiss::METRIC_INNER_PRODUCT) {
|
||||
printf("[%.3f s] Normalizing base data set \n", elapsed() - t0);
|
||||
normalize(xb, nb, d);
|
||||
}
|
||||
|
||||
printf("[%.3f s] Creating CPU index \"%s\" d=%ld\n", elapsed() - t0, index_key.c_str(), d);
|
||||
cpu_index = faiss::index_factory(d, index_key.c_str(), metric_type);
|
||||
|
||||
printf("[%.3f s] Cloning CPU index to GPU\n", elapsed() - t0);
|
||||
gpu_index = faiss::gpu::index_cpu_to_gpu(&res, GPU_DEVICE_IDX, cpu_index);
|
||||
delete cpu_index;
|
||||
|
||||
printf("[%.3f s] Training on %ld vectors\n", elapsed() - t0, nb);
|
||||
gpu_index->train(nb, xb);
|
||||
|
||||
// add index multiple times to get ~1G data set
|
||||
for (int i = 0; i < index_add_loops; i++) {
|
||||
printf("[%.3f s] No.%d Indexing database, size %ld*%ld\n", elapsed() - t0, i, nb, d);
|
||||
gpu_index->add(nb, xb);
|
||||
}
|
||||
|
||||
printf("[%.3f s] Coping GPU index to CPU\n", elapsed() - t0);
|
||||
|
||||
cpu_index = faiss::gpu::index_gpu_to_cpu(gpu_index);
|
||||
delete gpu_index;
|
||||
|
||||
faiss::IndexIVF* cpu_ivf_index = dynamic_cast<faiss::IndexIVF*>(cpu_index);
|
||||
if (cpu_ivf_index != nullptr) {
|
||||
cpu_ivf_index->to_readonly();
|
||||
}
|
||||
|
||||
printf("[%.3f s] Writing index file: %s\n", elapsed() - t0, index_file_name.c_str());
|
||||
faiss::write_index(cpu_index, index_file_name.c_str());
|
||||
|
||||
delete[] xb;
|
||||
}
|
||||
|
||||
index = cpu_index;
|
||||
}
|
||||
|
||||
void
|
||||
load_query_data(faiss::Index::distance_t*& xq, size_t& nq, const std::string& ann_test_name,
|
||||
const faiss::MetricType metric_type, const size_t dim) {
|
||||
double t0 = elapsed();
|
||||
size_t d;
|
||||
|
||||
const std::string ann_file_name = ann_test_name + HDF5_POSTFIX;
|
||||
|
||||
xq = (float*)hdf5_read(ann_file_name, HDF5_DATASET_TEST, H5T_FLOAT, d, nq);
|
||||
assert(d == dim || !"query does not have same dimension as train set");
|
||||
|
||||
if (metric_type == faiss::METRIC_INNER_PRODUCT) {
|
||||
printf("[%.3f s] Normalizing query data \n", elapsed() - t0);
|
||||
normalize(xq, nq, d);
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
load_ground_truth(faiss::Index::idx_t*& gt, size_t& k, const std::string& ann_test_name, const size_t nq) {
|
||||
const std::string ann_file_name = ann_test_name + HDF5_POSTFIX;
|
||||
|
||||
// load ground-truth and convert int to long
|
||||
size_t nq2;
|
||||
int* gt_int = (int*)hdf5_read(ann_file_name, HDF5_DATASET_NEIGHBORS, H5T_INTEGER, k, nq2);
|
||||
assert(nq2 == nq || !"incorrect nb of ground truth index");
|
||||
|
||||
gt = new faiss::Index::idx_t[k * nq];
|
||||
for (int i = 0; i < k * nq; i++) {
|
||||
gt[i] = gt_int[i];
|
||||
}
|
||||
delete[] gt_int;
|
||||
|
||||
#if DEBUG_VERBOSE
|
||||
faiss::Index::distance_t* gt_dist; // nq * k matrix of ground-truth nearest-neighbors distances
|
||||
gt_dist = (float*)hdf5_read(ann_file_name, HDF5_DATASET_DISTANCES, H5T_FLOAT, k, nq2);
|
||||
assert(nq2 == nq || !"incorrect nb of ground truth distance");
|
||||
|
||||
std::string str;
|
||||
str = ann_test_name + " ground truth index";
|
||||
print_array(str.c_str(), true, gt, nq, k);
|
||||
str = ann_test_name + " ground truth distance";
|
||||
print_array(str.c_str(), false, gt_dist, nq, k);
|
||||
|
||||
delete gt_dist;
|
||||
#endif
|
||||
}
|
||||
|
||||
void
|
||||
test_with_nprobes(const std::string& ann_test_name, const std::string& index_key, faiss::Index* cpu_index,
|
||||
faiss::gpu::StandardGpuResources& res, const QueryMode query_mode, const faiss::Index::distance_t* xq,
|
||||
const faiss::Index::idx_t* gt, const std::vector<size_t> nprobes, const int32_t index_add_loops,
|
||||
const int32_t search_loops) {
|
||||
double t0 = elapsed();
|
||||
|
||||
const size_t NQ = 1000, NQ_START = 10, NQ_STEP = 10;
|
||||
const size_t K = 1000, K_START = 100, K_STEP = 10;
|
||||
const size_t GK = 100; // topk of ground truth
|
||||
|
||||
std::unordered_map<size_t, std::string> mode_str_map = {
|
||||
{MODE_CPU, "MODE_CPU"}, {MODE_MIX, "MODE_MIX"}, {MODE_GPU, "MODE_GPU"}};
|
||||
|
||||
faiss::Index *gpu_index, *index;
|
||||
if (query_mode != MODE_CPU) {
|
||||
faiss::gpu::GpuClonerOptions option;
|
||||
option.allInGpu = true;
|
||||
|
||||
faiss::IndexComposition index_composition;
|
||||
index_composition.index = cpu_index;
|
||||
index_composition.quantizer = nullptr;
|
||||
|
||||
double copy_time;
|
||||
switch (query_mode) {
|
||||
case MODE_MIX: {
|
||||
index_composition.mode = 1; // 0: all data, 1: copy quantizer, 2: copy data
|
||||
|
||||
// warm up the transmission
|
||||
gpu_index = faiss::gpu::index_cpu_to_gpu(&res, GPU_DEVICE_IDX, &index_composition, &option);
|
||||
delete gpu_index;
|
||||
|
||||
copy_time = elapsed();
|
||||
gpu_index = faiss::gpu::index_cpu_to_gpu(&res, GPU_DEVICE_IDX, &index_composition, &option);
|
||||
delete gpu_index;
|
||||
copy_time = elapsed() - copy_time;
|
||||
printf("[%.3f s] Copy quantizer completed, cost %f s\n", elapsed() - t0, copy_time);
|
||||
|
||||
auto ivf_index = dynamic_cast<faiss::IndexIVF*>(cpu_index);
|
||||
auto is_gpu_flat_index = dynamic_cast<faiss::gpu::GpuIndexFlat*>(ivf_index->quantizer);
|
||||
if (is_gpu_flat_index == nullptr) {
|
||||
delete ivf_index->quantizer;
|
||||
ivf_index->quantizer = index_composition.quantizer;
|
||||
}
|
||||
index = cpu_index;
|
||||
break;
|
||||
}
|
||||
case MODE_GPU:
|
||||
index_composition.mode = 0; // 0: all data, 1: copy quantizer, 2: copy data
|
||||
|
||||
// warm up the transmission
|
||||
gpu_index = faiss::gpu::index_cpu_to_gpu(&res, GPU_DEVICE_IDX, &index_composition, &option);
|
||||
delete gpu_index;
|
||||
|
||||
copy_time = elapsed();
|
||||
gpu_index = faiss::gpu::index_cpu_to_gpu(&res, GPU_DEVICE_IDX, &index_composition, &option);
|
||||
copy_time = elapsed() - copy_time;
|
||||
printf("[%.3f s] Copy data completed, cost %f s\n", elapsed() - t0, copy_time);
|
||||
|
||||
delete cpu_index;
|
||||
index = gpu_index;
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
index = cpu_index;
|
||||
}
|
||||
|
||||
for (auto nprobe : nprobes) {
|
||||
switch (query_mode) {
|
||||
case MODE_CPU:
|
||||
case MODE_MIX: {
|
||||
faiss::ParameterSpace params;
|
||||
std::string nprobe_str = "nprobe=" + std::to_string(nprobe);
|
||||
params.set_index_parameters(index, nprobe_str.c_str());
|
||||
break;
|
||||
}
|
||||
case MODE_GPU: {
|
||||
faiss::gpu::GpuIndexIVF* gpu_index_ivf = dynamic_cast<faiss::gpu::GpuIndexIVF*>(index);
|
||||
gpu_index_ivf->setNumProbes(nprobe);
|
||||
}
|
||||
}
|
||||
|
||||
// output buffers
|
||||
faiss::Index::idx_t* I = new faiss::Index::idx_t[NQ * K];
|
||||
faiss::Index::distance_t* D = new faiss::Index::distance_t[NQ * K];
|
||||
|
||||
printf("\n%s | %s - %s | nprobe=%lu\n", ann_test_name.c_str(), index_key.c_str(),
|
||||
mode_str_map[query_mode].c_str(), nprobe);
|
||||
printf("======================================================================================\n");
|
||||
for (size_t t_nq = NQ_START; t_nq <= NQ; t_nq *= NQ_STEP) { // nq = {10, 100, 1000}
|
||||
for (size_t t_k = K_START; t_k <= K; t_k *= K_STEP) { // k = {100, 1000}
|
||||
faiss::indexIVF_stats.quantization_time = 0.0;
|
||||
faiss::indexIVF_stats.search_time = 0.0;
|
||||
|
||||
double t_start = elapsed(), t_end;
|
||||
for (int i = 0; i < search_loops; i++) {
|
||||
index->search(t_nq, xq, t_k, D, I);
|
||||
}
|
||||
t_end = elapsed();
|
||||
|
||||
#if DEBUG_VERBOSE
|
||||
std::string str;
|
||||
str = "I (" + index_key + ", nq=" + std::to_string(t_nq) + ", k=" + std::to_string(t_k) + ")";
|
||||
print_array(str.c_str(), true, I, t_nq, t_k);
|
||||
str = "D (" + index_key + ", nq=" + std::to_string(t_nq) + ", k=" + std::to_string(t_k) + ")";
|
||||
print_array(str.c_str(), false, D, t_nq, t_k);
|
||||
#endif
|
||||
|
||||
// k = 100 for ground truth
|
||||
int32_t hit = GetResultHitCount(gt, I, GK, t_k, t_nq, index_add_loops);
|
||||
|
||||
printf("nq = %4ld, k = %4ld, elapse = %.4fs (quant = %.4fs, search = %.4fs), R@ = %.4f\n", t_nq, t_k,
|
||||
(t_end - t_start) / search_loops, faiss::indexIVF_stats.quantization_time / 1000 / search_loops,
|
||||
faiss::indexIVF_stats.search_time / 1000 / search_loops,
|
||||
(hit / float(t_nq * GK / index_add_loops)));
|
||||
}
|
||||
}
|
||||
printf("======================================================================================\n");
|
||||
|
||||
delete[] I;
|
||||
delete[] D;
|
||||
}
|
||||
|
||||
delete index;
|
||||
}
|
||||
|
||||
void
|
||||
test_ann_hdf5(const std::string& ann_test_name, const std::string& index_type, const QueryMode query_mode,
|
||||
int32_t index_add_loops, const std::vector<size_t>& nprobes, int32_t search_loops) {
|
||||
double t0 = elapsed();
|
||||
|
||||
faiss::gpu::StandardGpuResources res;
|
||||
|
||||
faiss::MetricType metric_type;
|
||||
size_t dim;
|
||||
|
||||
if (query_mode == MODE_MIX && index_type != "SQ8Hybrid") {
|
||||
assert(index_type == "SQ8Hybrid" || !"Only SQ8Hybrid support MODE_MIX");
|
||||
return;
|
||||
}
|
||||
|
||||
std::string index_key = "IVF16384," + index_type;
|
||||
|
||||
if (!parse_ann_test_name(ann_test_name, dim, metric_type)) {
|
||||
printf("Invalid ann test name: %s\n", ann_test_name.c_str());
|
||||
return;
|
||||
}
|
||||
|
||||
size_t nq, k;
|
||||
faiss::Index* index;
|
||||
faiss::Index::distance_t* xq;
|
||||
faiss::Index::idx_t* gt; // ground-truth index
|
||||
|
||||
printf("[%.3f s] Loading base data\n", elapsed() - t0);
|
||||
load_base_data(index, ann_test_name, index_key, res, metric_type, dim, index_add_loops, query_mode);
|
||||
|
||||
printf("[%.3f s] Loading queries\n", elapsed() - t0);
|
||||
load_query_data(xq, nq, ann_test_name, metric_type, dim);
|
||||
|
||||
printf("[%.3f s] Loading ground truth for %ld queries\n", elapsed() - t0, nq);
|
||||
load_ground_truth(gt, k, ann_test_name, nq);
|
||||
|
||||
test_with_nprobes(ann_test_name, index_key, index, res, query_mode, xq, gt, nprobes, index_add_loops, search_loops);
|
||||
printf("[%.3f s] Search test done\n\n", elapsed() - t0);
|
||||
|
||||
delete[] xq;
|
||||
delete[] gt;
|
||||
}
|
||||
|
||||
/************************************************************************************
|
||||
* https://github.com/erikbern/ann-benchmarks
|
||||
*
|
||||
* Dataset Dimensions Train_size Test_size Neighbors Distance Download
|
||||
* Fashion-
|
||||
* MNIST 784 60,000 10,000 100 Euclidean HDF5 (217MB)
|
||||
* GIST 960 1,000,000 1,000 100 Euclidean HDF5 (3.6GB)
|
||||
* GloVe 100 1,183,514 10,000 100 Angular HDF5 (463MB)
|
||||
* GloVe 200 1,183,514 10,000 100 Angular HDF5 (918MB)
|
||||
* MNIST 784 60,000 10,000 100 Euclidean HDF5 (217MB)
|
||||
* NYTimes 256 290,000 10,000 100 Angular HDF5 (301MB)
|
||||
* SIFT 128 1,000,000 10,000 100 Euclidean HDF5 (501MB)
|
||||
*************************************************************************************/
|
||||
|
||||
TEST(FAISSTEST, BENCHMARK) {
|
||||
std::vector<size_t> param_nprobes = {8, 128};
|
||||
const int32_t SEARCH_LOOPS = 5;
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
|
||||
const int32_t SIFT_INSERT_LOOPS = 2; // insert twice to get ~1G data set
|
||||
|
||||
test_ann_hdf5("sift-128-euclidean", "Flat", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("sift-128-euclidean", "Flat", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
|
||||
test_ann_hdf5("sift-128-euclidean", "SQ8", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("sift-128-euclidean", "SQ8", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
|
||||
#ifdef CUSTOMIZATION
|
||||
test_ann_hdf5("sift-128-euclidean", "SQ8Hybrid", MODE_CPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("sift-128-euclidean", "SQ8Hybrid", MODE_MIX, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("sift-128-euclidean", "SQ8Hybrid", MODE_GPU, SIFT_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
#endif
|
||||
|
||||
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////
|
||||
const int32_t GLOVE_INSERT_LOOPS = 1;
|
||||
|
||||
test_ann_hdf5("glove-200-angular", "Flat", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("glove-200-angular", "Flat", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
|
||||
test_ann_hdf5("glove-200-angular", "SQ8", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("glove-200-angular", "SQ8", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
|
||||
#ifdef CUSTOMIZATION
|
||||
test_ann_hdf5("glove-200-angular", "SQ8Hybrid", MODE_CPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("glove-200-angular", "SQ8Hybrid", MODE_MIX, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
test_ann_hdf5("glove-200-angular", "SQ8Hybrid", MODE_GPU, GLOVE_INSERT_LOOPS, param_nprobes, SEARCH_LOOPS);
|
||||
#endif
|
||||
}
|
||||
@ -16,17 +16,23 @@
|
||||
// under the License.
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
#include <thread>
|
||||
|
||||
#include "unittest/Helper.h"
|
||||
#include "unittest/utils.h"
|
||||
|
||||
#include "knowhere/common/Timer.h"
|
||||
|
||||
class SingleIndexTest : public DataGen, public TestGpuIndexBase {
|
||||
protected:
|
||||
void
|
||||
SetUp() override {
|
||||
TestGpuIndexBase::SetUp();
|
||||
Generate(DIM, NB, NQ);
|
||||
k = K;
|
||||
nb = 1000000;
|
||||
nq = 1000;
|
||||
dim = DIM;
|
||||
Generate(dim, nb, nq);
|
||||
k = 1000;
|
||||
}
|
||||
|
||||
void
|
||||
@ -119,4 +125,113 @@ TEST_F(SingleIndexTest, IVFSQHybrid) {
|
||||
}
|
||||
}
|
||||
|
||||
// TEST_F(SingleIndexTest, thread_safe) {
|
||||
// assert(!xb.empty());
|
||||
//
|
||||
// index_type = "IVFSQHybrid";
|
||||
// index_ = IndexFactory(index_type);
|
||||
// auto base = ParamGenerator::GetInstance().Gen(ParameterType::ivfsq);
|
||||
// auto conf = std::dynamic_pointer_cast<knowhere::IVFSQCfg>(base);
|
||||
// conf->nlist = 16384;
|
||||
// conf->k = k;
|
||||
// conf->nprobe = 10;
|
||||
// conf->d = dim;
|
||||
// auto preprocessor = index_->BuildPreprocessor(base_dataset, conf);
|
||||
// index_->set_preprocessor(preprocessor);
|
||||
//
|
||||
// auto model = index_->Train(base_dataset, conf);
|
||||
// index_->set_index_model(model);
|
||||
// index_->Add(base_dataset, conf);
|
||||
// EXPECT_EQ(index_->Count(), nb);
|
||||
// EXPECT_EQ(index_->Dimension(), dim);
|
||||
//
|
||||
// auto binaryset = index_->Serialize();
|
||||
//
|
||||
//
|
||||
//
|
||||
// auto cpu_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
|
||||
// cpu_idx->Load(binaryset);
|
||||
// auto pair = cpu_idx->CopyCpuToGpuWithQuantizer(DEVICEID, conf);
|
||||
// auto quantizer = pair.second;
|
||||
//
|
||||
// auto quantizer_conf = std::make_shared<knowhere::QuantizerCfg>();
|
||||
// quantizer_conf->mode = 2; // only copy data
|
||||
// quantizer_conf->gpu_id = DEVICEID;
|
||||
//
|
||||
// auto CopyAllToGpu = [&](int64_t search_count, bool do_search = false) {
|
||||
// for (int i = 0; i < search_count; ++i) {
|
||||
// auto gpu_idx = cpu_idx->CopyCpuToGpu(DEVICEID, conf);
|
||||
// if (do_search) {
|
||||
// auto result = gpu_idx->Search(query_dataset, conf);
|
||||
// AssertAnns(result, nq, conf->k);
|
||||
// }
|
||||
// }
|
||||
// };
|
||||
//
|
||||
// auto hybrid_qt_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
|
||||
// hybrid_qt_idx->Load(binaryset);
|
||||
// auto SetQuantizerDoSearch = [&](int64_t search_count) {
|
||||
// for (int i = 0; i < search_count; ++i) {
|
||||
// hybrid_qt_idx->SetQuantizer(quantizer);
|
||||
// auto result = hybrid_qt_idx->Search(query_dataset, conf);
|
||||
// AssertAnns(result, nq, conf->k);
|
||||
// // PrintResult(result, nq, k);
|
||||
// hybrid_qt_idx->UnsetQuantizer();
|
||||
// }
|
||||
// };
|
||||
//
|
||||
// auto hybrid_data_idx = std::make_shared<knowhere::IVFSQHybrid>(DEVICEID);
|
||||
// hybrid_data_idx->Load(binaryset);
|
||||
// auto LoadDataDoSearch = [&](int64_t search_count, bool do_search = false) {
|
||||
// for (int i = 0; i < search_count; ++i) {
|
||||
// auto hybrid_idx = hybrid_data_idx->LoadData(quantizer, quantizer_conf);
|
||||
// if (do_search) {
|
||||
// auto result = hybrid_idx->Search(query_dataset, conf);
|
||||
//// AssertAnns(result, nq, conf->k);
|
||||
// }
|
||||
// }
|
||||
// };
|
||||
//
|
||||
// knowhere::TimeRecorder tc("");
|
||||
// CopyAllToGpu(2000/2, false);
|
||||
// tc.RecordSection("CopyAllToGpu witout search");
|
||||
// CopyAllToGpu(400/2, true);
|
||||
// tc.RecordSection("CopyAllToGpu with search");
|
||||
// SetQuantizerDoSearch(6);
|
||||
// tc.RecordSection("SetQuantizer with search");
|
||||
// LoadDataDoSearch(2000/2, false);
|
||||
// tc.RecordSection("LoadData without search");
|
||||
// LoadDataDoSearch(400/2, true);
|
||||
// tc.RecordSection("LoadData with search");
|
||||
//
|
||||
// {
|
||||
// std::thread t1(CopyAllToGpu, 2000, false);
|
||||
// std::thread t2(CopyAllToGpu, 400, true);
|
||||
// t1.join();
|
||||
// t2.join();
|
||||
// }
|
||||
//
|
||||
// {
|
||||
// std::thread t1(SetQuantizerDoSearch, 12);
|
||||
// std::thread t2(CopyAllToGpu, 400, true);
|
||||
// t1.join();
|
||||
// t2.join();
|
||||
// }
|
||||
//
|
||||
// {
|
||||
// std::thread t1(SetQuantizerDoSearch, 12);
|
||||
// std::thread t2(LoadDataDoSearch, 400, true);
|
||||
// t1.join();
|
||||
// t2.join();
|
||||
// }
|
||||
//
|
||||
// {
|
||||
// std::thread t1(LoadDataDoSearch, 2000, false);
|
||||
// std::thread t2(LoadDataDoSearch, 400, true);
|
||||
// t1.join();
|
||||
// t2.join();
|
||||
// }
|
||||
//
|
||||
//}
|
||||
|
||||
#endif
|
||||
|
||||
@ -20,19 +20,12 @@
|
||||
#include <iostream>
|
||||
#include <thread>
|
||||
|
||||
#include <faiss/AutoTune.h>
|
||||
#include <faiss/gpu/GpuAutoTune.h>
|
||||
#include <faiss/gpu/GpuIndexIVFFlat.h>
|
||||
|
||||
#include "knowhere/common/Exception.h"
|
||||
#include "knowhere/common/Timer.h"
|
||||
#include "knowhere/index/vector_index/IndexGPUIVF.h"
|
||||
#include "knowhere/index/vector_index/IndexGPUIVFPQ.h"
|
||||
#include "knowhere/index/vector_index/IndexGPUIVFSQ.h"
|
||||
#include "knowhere/index/vector_index/IndexIVF.h"
|
||||
#include "knowhere/index/vector_index/IndexIVFPQ.h"
|
||||
#include "knowhere/index/vector_index/IndexIVFSQ.h"
|
||||
#include "knowhere/index/vector_index/IndexIVFSQHybrid.h"
|
||||
#include "knowhere/index/vector_index/helpers/Cloner.h"
|
||||
|
||||
#include "unittest/Helper.h"
|
||||
@ -51,6 +44,9 @@ class IVFTest : public DataGen, public TestWithParam<::std::tuple<std::string, P
|
||||
ParameterType parameter_type;
|
||||
std::tie(index_type, parameter_type) = GetParam();
|
||||
// Init_with_default();
|
||||
// nb = 1000000;
|
||||
// nq = 1000;
|
||||
// k = 1000;
|
||||
Generate(DIM, NB, NQ);
|
||||
index_ = IndexFactory(index_type);
|
||||
conf = ParamGenerator::GetInstance().Gen(parameter_type);
|
||||
@ -61,16 +57,6 @@ class IVFTest : public DataGen, public TestWithParam<::std::tuple<std::string, P
|
||||
knowhere::FaissGpuResourceMgr::GetInstance().Free();
|
||||
}
|
||||
|
||||
knowhere::VectorIndexPtr
|
||||
ChooseTodo() {
|
||||
std::vector<std::string> gpu_idx{"GPUIVFSQ"};
|
||||
auto finder = std::find(gpu_idx.cbegin(), gpu_idx.cend(), index_type);
|
||||
if (finder != gpu_idx.cend()) {
|
||||
return knowhere::cloner::CopyCpuToGpu(index_, DEVICEID, knowhere::Config());
|
||||
}
|
||||
return index_;
|
||||
}
|
||||
|
||||
protected:
|
||||
std::string index_type;
|
||||
knowhere::Config conf;
|
||||
@ -100,8 +86,7 @@ TEST_P(IVFTest, ivf_basic) {
|
||||
EXPECT_EQ(index_->Count(), nb);
|
||||
EXPECT_EQ(index_->Dimension(), dim);
|
||||
|
||||
auto new_idx = ChooseTodo();
|
||||
auto result = new_idx->Search(query_dataset, conf);
|
||||
auto result = index_->Search(query_dataset, conf);
|
||||
AssertAnns(result, nq, conf->k);
|
||||
// PrintResult(result, nq, k);
|
||||
}
|
||||
@ -134,8 +119,7 @@ TEST_P(IVFTest, ivf_serialize) {
|
||||
|
||||
index_->set_index_model(model);
|
||||
index_->Add(base_dataset, conf);
|
||||
auto new_idx = ChooseTodo();
|
||||
auto result = new_idx->Search(query_dataset, conf);
|
||||
auto result = index_->Search(query_dataset, conf);
|
||||
AssertAnns(result, nq, conf->k);
|
||||
}
|
||||
|
||||
@ -159,8 +143,7 @@ TEST_P(IVFTest, ivf_serialize) {
|
||||
index_->Load(binaryset);
|
||||
EXPECT_EQ(index_->Count(), nb);
|
||||
EXPECT_EQ(index_->Dimension(), dim);
|
||||
auto new_idx = ChooseTodo();
|
||||
auto result = new_idx->Search(query_dataset, conf);
|
||||
auto result = index_->Search(query_dataset, conf);
|
||||
AssertAnns(result, nq, conf->k);
|
||||
}
|
||||
}
|
||||
@ -176,8 +159,7 @@ TEST_P(IVFTest, clone_test) {
|
||||
index_->Add(base_dataset, conf);
|
||||
EXPECT_EQ(index_->Count(), nb);
|
||||
EXPECT_EQ(index_->Dimension(), dim);
|
||||
auto new_idx = ChooseTodo();
|
||||
auto result = new_idx->Search(query_dataset, conf);
|
||||
auto result = index_->Search(query_dataset, conf);
|
||||
AssertAnns(result, nq, conf->k);
|
||||
// PrintResult(result, nq, k);
|
||||
|
||||
@ -210,12 +192,6 @@ TEST_P(IVFTest, clone_test) {
|
||||
// }
|
||||
// }
|
||||
|
||||
{
|
||||
if (index_type == "IVFSQHybrid") {
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
// copy from gpu to cpu
|
||||
std::vector<std::string> support_idx_vec{"GPUIVF", "GPUIVFSQ", "IVFSQHybrid"};
|
||||
@ -237,6 +213,10 @@ TEST_P(IVFTest, clone_test) {
|
||||
}
|
||||
}
|
||||
|
||||
if (index_type == "IVFSQHybrid") {
|
||||
return;
|
||||
}
|
||||
|
||||
{
|
||||
// copy to gpu
|
||||
std::vector<std::string> support_idx_vec{"IVF", "GPUIVF", "IVFSQ", "GPUIVFSQ"};
|
||||
@ -277,8 +257,7 @@ TEST_P(IVFTest, gpu_seal_test) {
|
||||
index_->Add(base_dataset, conf);
|
||||
EXPECT_EQ(index_->Count(), nb);
|
||||
EXPECT_EQ(index_->Dimension(), dim);
|
||||
auto new_idx = ChooseTodo();
|
||||
auto result = new_idx->Search(query_dataset, conf);
|
||||
auto result = index_->Search(query_dataset, conf);
|
||||
AssertAnns(result, nq, conf->k);
|
||||
|
||||
auto cpu_idx = knowhere::cloner::CopyGpuToCpu(index_, knowhere::Config());
|
||||
|
||||
@ -22,12 +22,12 @@
|
||||
#include <cstring>
|
||||
#include <string>
|
||||
|
||||
#include "../version.h"
|
||||
#include "external/easyloggingpp/easylogging++.h"
|
||||
#include "metrics/Metrics.h"
|
||||
#include "server/Server.h"
|
||||
#include "src/version.h"
|
||||
#include "utils/CommonUtil.h"
|
||||
#include "utils/SignalUtil.h"
|
||||
#include "utils/easylogging++.h"
|
||||
|
||||
INITIALIZE_EASYLOGGINGPP
|
||||
|
||||
|
||||
@ -54,7 +54,7 @@ ShortestPath(const ResourcePtr& src, const ResourcePtr& dest, const ResourceMgrP
|
||||
auto cur_neighbours = cur_node->GetNeighbours();
|
||||
|
||||
for (auto& neighbour : cur_neighbours) {
|
||||
auto neighbour_res = std::static_pointer_cast<Resource>(neighbour.neighbour_node.lock());
|
||||
auto neighbour_res = std::static_pointer_cast<Resource>(neighbour.neighbour_node);
|
||||
dis_matrix[name_id_map.at(res->name())][name_id_map.at(neighbour_res->name())] =
|
||||
neighbour.connection.transport_cost();
|
||||
}
|
||||
|
||||
@ -34,27 +34,30 @@ namespace scheduler {
|
||||
|
||||
class BuildMgr {
|
||||
public:
|
||||
explicit BuildMgr(int64_t numoftasks) : numoftasks_(numoftasks) {
|
||||
explicit BuildMgr(int64_t concurrent_limit) : available_(concurrent_limit) {
|
||||
}
|
||||
|
||||
public:
|
||||
void
|
||||
Put() {
|
||||
++numoftasks_;
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
++available_;
|
||||
}
|
||||
|
||||
void
|
||||
take() {
|
||||
--numoftasks_;
|
||||
}
|
||||
|
||||
int64_t
|
||||
numoftasks() {
|
||||
return (int64_t)numoftasks_;
|
||||
bool
|
||||
Take() {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (available_ < 1) {
|
||||
return false;
|
||||
} else {
|
||||
--available_;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
private:
|
||||
std::atomic_long numoftasks_;
|
||||
std::int64_t available_;
|
||||
std::mutex mutex_;
|
||||
};
|
||||
|
||||
using BuildMgrPtr = std::shared_ptr<BuildMgr>;
|
||||
|
||||
119
core/src/scheduler/CircleQueue.h
Normal file
119
core/src/scheduler/CircleQueue.h
Normal file
@ -0,0 +1,119 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
#pragma once
|
||||
|
||||
#include <atomic>
|
||||
#include <condition_variable>
|
||||
#include <deque>
|
||||
#include <list>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <queue>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <unordered_map>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
template <typename T>
|
||||
class CircleQueue {
|
||||
using value_type = T;
|
||||
using atomic_size_type = std::atomic_ullong;
|
||||
using size_type = uint64_t;
|
||||
using const_reference = const value_type&;
|
||||
#define MEMORY_ORDER (std::memory_order::memory_order_seq_cst)
|
||||
|
||||
public:
|
||||
explicit CircleQueue(size_type cap) : data_(cap, nullptr), capacity_(cap), front_() {
|
||||
front_.store(cap - 1, MEMORY_ORDER);
|
||||
}
|
||||
|
||||
CircleQueue() = delete;
|
||||
CircleQueue(const CircleQueue& q) = delete;
|
||||
CircleQueue(CircleQueue&& q) = delete;
|
||||
|
||||
public:
|
||||
const_reference operator[](size_type n) {
|
||||
return data_[n % capacity_];
|
||||
}
|
||||
|
||||
size_type
|
||||
front() {
|
||||
return front_.load(MEMORY_ORDER);
|
||||
}
|
||||
|
||||
size_type
|
||||
rear() {
|
||||
return rear_;
|
||||
}
|
||||
|
||||
size_type
|
||||
size() {
|
||||
return size_;
|
||||
}
|
||||
|
||||
size_type
|
||||
capacity() {
|
||||
return capacity_;
|
||||
}
|
||||
|
||||
void
|
||||
set_front(uint64_t last_finish) {
|
||||
if (last_finish == rear_) {
|
||||
throw;
|
||||
}
|
||||
front_.store(last_finish % capacity_, MEMORY_ORDER);
|
||||
}
|
||||
|
||||
void
|
||||
put(const value_type& x) {
|
||||
if ((rear_) % capacity_ == front_.load(MEMORY_ORDER)) {
|
||||
throw;
|
||||
}
|
||||
data_[rear_] = x;
|
||||
rear_ = ++rear_ % capacity_;
|
||||
if (size_ < capacity_) {
|
||||
++size_;
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
put(value_type&& x) {
|
||||
if ((rear_) % capacity_ == front_.load(MEMORY_ORDER)) {
|
||||
throw;
|
||||
}
|
||||
data_[rear_] = std::move(x);
|
||||
rear_ = ++rear_ % capacity_;
|
||||
if (size_ < capacity_) {
|
||||
++size_;
|
||||
}
|
||||
}
|
||||
|
||||
private:
|
||||
std::vector<value_type> data_;
|
||||
size_type capacity_;
|
||||
atomic_size_type front_;
|
||||
size_type rear_ = 0;
|
||||
size_type size_ = 0;
|
||||
#undef MEMORY_ORDER
|
||||
};
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
@ -49,6 +49,15 @@ JobMgr::Stop() {
|
||||
}
|
||||
}
|
||||
|
||||
json
|
||||
JobMgr::Dump() const {
|
||||
json ret{
|
||||
{"running", running_},
|
||||
{"event_queue_length", queue_.size()},
|
||||
};
|
||||
return ret;
|
||||
}
|
||||
|
||||
void
|
||||
JobMgr::Put(const JobPtr& job) {
|
||||
{
|
||||
@ -82,7 +91,7 @@ JobMgr::worker_function() {
|
||||
// disk resources NEVER be empty.
|
||||
if (auto disk = res_mgr_->GetDiskResources()[0].lock()) {
|
||||
for (auto& task : tasks) {
|
||||
disk->task_table().Put(task);
|
||||
disk->task_table().Put(task, nullptr);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -95,20 +104,25 @@ JobMgr::build_task(const JobPtr& job) {
|
||||
|
||||
void
|
||||
JobMgr::calculate_path(const TaskPtr& task) {
|
||||
if (task->type_ != TaskType::SearchTask) {
|
||||
return;
|
||||
}
|
||||
if (task->type_ == TaskType::SearchTask) {
|
||||
if (task->label()->Type() != TaskLabelType::SPECIFIED_RESOURCE) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (task->label()->Type() != TaskLabelType::SPECIFIED_RESOURCE) {
|
||||
return;
|
||||
std::vector<std::string> path;
|
||||
auto spec_label = std::static_pointer_cast<SpecResLabel>(task->label());
|
||||
auto src = res_mgr_->GetDiskResources()[0];
|
||||
auto dest = spec_label->resource();
|
||||
ShortestPath(src.lock(), dest.lock(), res_mgr_, path);
|
||||
task->path() = Path(path, path.size() - 1);
|
||||
} else if (task->type_ == TaskType::BuildIndexTask) {
|
||||
auto spec_label = std::static_pointer_cast<SpecResLabel>(task->label());
|
||||
auto src = res_mgr_->GetDiskResources()[0];
|
||||
auto dest = spec_label->resource();
|
||||
std::vector<std::string> path;
|
||||
ShortestPath(src.lock(), dest.lock(), res_mgr_, path);
|
||||
task->path() = Path(path, path.size() - 1);
|
||||
}
|
||||
|
||||
std::vector<std::string> path;
|
||||
auto spec_label = std::static_pointer_cast<SpecResLabel>(task->label());
|
||||
auto src = res_mgr_->GetDiskResources()[0];
|
||||
auto dest = spec_label->resource();
|
||||
ShortestPath(src.lock(), dest.lock(), res_mgr_, path);
|
||||
task->path() = Path(path, path.size() - 1);
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
|
||||
@ -28,13 +28,14 @@
|
||||
#include <vector>
|
||||
|
||||
#include "ResourceMgr.h"
|
||||
#include "interface/interfaces.h"
|
||||
#include "job/Job.h"
|
||||
#include "task/Task.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
class JobMgr {
|
||||
class JobMgr : public interface::dumpable {
|
||||
public:
|
||||
explicit JobMgr(ResourceMgrPtr res_mgr);
|
||||
|
||||
@ -44,6 +45,9 @@ class JobMgr {
|
||||
void
|
||||
Stop();
|
||||
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
public:
|
||||
void
|
||||
Put(const JobPtr& job);
|
||||
|
||||
@ -170,16 +170,20 @@ ResourceMgr::GetNumGpuResource() const {
|
||||
return num;
|
||||
}
|
||||
|
||||
std::string
|
||||
ResourceMgr::Dump() {
|
||||
std::stringstream ss;
|
||||
ss << "ResourceMgr contains " << resources_.size() << " resources." << std::endl;
|
||||
|
||||
json
|
||||
ResourceMgr::Dump() const {
|
||||
json resources{};
|
||||
for (auto& res : resources_) {
|
||||
ss << res->Dump();
|
||||
resources.push_back(res->Dump());
|
||||
}
|
||||
|
||||
return ss.str();
|
||||
json ret{
|
||||
{"number_of_resource", resources_.size()},
|
||||
{"number_of_disk_resource", disk_resources_.size()},
|
||||
{"number_of_cpu_resource", cpu_resources_.size()},
|
||||
{"number_of_gpu_resource", gpu_resources_.size()},
|
||||
{"resources", resources},
|
||||
};
|
||||
return ret;
|
||||
}
|
||||
|
||||
std::string
|
||||
@ -187,9 +191,9 @@ ResourceMgr::DumpTaskTables() {
|
||||
std::stringstream ss;
|
||||
ss << ">>>>>>>>>>>>>>>ResourceMgr::DumpTaskTable<<<<<<<<<<<<<<<" << std::endl;
|
||||
for (auto& resource : resources_) {
|
||||
ss << resource->Dump() << std::endl;
|
||||
ss << resource->task_table().Dump();
|
||||
ss << resource->Dump() << std::endl << std::endl;
|
||||
ss << resource->name() << std::endl;
|
||||
ss << resource->task_table().Dump().dump();
|
||||
ss << resource->name() << std::endl << std::endl;
|
||||
}
|
||||
return ss.str();
|
||||
}
|
||||
|
||||
@ -25,13 +25,14 @@
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "interface/interfaces.h"
|
||||
#include "resource/Resource.h"
|
||||
#include "utils/Log.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
class ResourceMgr {
|
||||
class ResourceMgr : public interface::dumpable {
|
||||
public:
|
||||
ResourceMgr() = default;
|
||||
|
||||
@ -74,7 +75,6 @@ class ResourceMgr {
|
||||
return gpu_resources_;
|
||||
}
|
||||
|
||||
// TODO(wxyu): why return shared pointer
|
||||
inline std::vector<ResourcePtr>
|
||||
GetAllResources() {
|
||||
return resources_;
|
||||
@ -103,8 +103,8 @@ class ResourceMgr {
|
||||
|
||||
public:
|
||||
/******** Utility Functions ********/
|
||||
std::string
|
||||
Dump();
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
std::string
|
||||
DumpTaskTables();
|
||||
|
||||
@ -55,8 +55,8 @@ load_simple_config() {
|
||||
// get resources
|
||||
auto gpu_ids = get_gpu_pool();
|
||||
|
||||
int32_t build_gpu_id;
|
||||
config.GetResourceConfigIndexBuildDevice(build_gpu_id);
|
||||
int32_t index_build_device_id;
|
||||
config.GetResourceConfigIndexBuildDevice(index_build_device_id);
|
||||
|
||||
// create and connect
|
||||
ResMgrInst::GetInstance()->Add(ResourceFactory::Create("disk", "DISK", 0, true, false));
|
||||
@ -70,91 +70,21 @@ load_simple_config() {
|
||||
for (auto& gpu_id : gpu_ids) {
|
||||
ResMgrInst::GetInstance()->Add(ResourceFactory::Create(std::to_string(gpu_id), "GPU", gpu_id, true, true));
|
||||
ResMgrInst::GetInstance()->Connect("cpu", std::to_string(gpu_id), pcie);
|
||||
if (build_gpu_id == gpu_id) {
|
||||
if (index_build_device_id == gpu_id) {
|
||||
find_build_gpu_id = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (not find_build_gpu_id) {
|
||||
if (not find_build_gpu_id && index_build_device_id != server::CPU_DEVICE_ID) {
|
||||
ResMgrInst::GetInstance()->Add(
|
||||
ResourceFactory::Create(std::to_string(build_gpu_id), "GPU", build_gpu_id, true, true));
|
||||
ResMgrInst::GetInstance()->Connect("cpu", std::to_string(build_gpu_id), pcie);
|
||||
ResourceFactory::Create(std::to_string(index_build_device_id), "GPU", index_build_device_id, true, true));
|
||||
ResMgrInst::GetInstance()->Connect("cpu", std::to_string(index_build_device_id), pcie);
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
load_advance_config() {
|
||||
// try {
|
||||
// server::ConfigNode &config = server::Config::GetInstance().GetConfig(server::CONFIG_RESOURCE);
|
||||
//
|
||||
// if (config.GetChildren().empty()) throw "resource_config null exception";
|
||||
//
|
||||
// auto resources = config.GetChild(server::CONFIG_RESOURCES).GetChildren();
|
||||
//
|
||||
// if (resources.empty()) throw "Children of resource_config null exception";
|
||||
//
|
||||
// for (auto &resource : resources) {
|
||||
// auto &resname = resource.first;
|
||||
// auto &resconf = resource.second;
|
||||
// auto type = resconf.GetValue(server::CONFIG_RESOURCE_TYPE);
|
||||
//// auto memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_MEMORY);
|
||||
// auto device_id = resconf.GetInt64Value(server::CONFIG_RESOURCE_DEVICE_ID);
|
||||
//// auto enable_loader = resconf.GetBoolValue(server::CONFIG_RESOURCE_ENABLE_LOADER);
|
||||
// auto enable_loader = true;
|
||||
// auto enable_executor = resconf.GetBoolValue(server::CONFIG_RESOURCE_ENABLE_EXECUTOR);
|
||||
// auto pinned_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_PIN_MEMORY);
|
||||
// auto temp_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_TEMP_MEMORY);
|
||||
// auto resource_num = resconf.GetInt64Value(server::CONFIG_RESOURCE_NUM);
|
||||
//
|
||||
// auto res = ResMgrInst::GetInstance()->Add(ResourceFactory::Create(resname,
|
||||
// type,
|
||||
// device_id,
|
||||
// enable_loader,
|
||||
// enable_executor));
|
||||
//
|
||||
// if (res.lock()->type() == ResourceType::GPU) {
|
||||
// auto pinned_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_PIN_MEMORY, 300);
|
||||
// auto temp_memory = resconf.GetInt64Value(server::CONFIG_RESOURCE_TEMP_MEMORY, 300);
|
||||
// auto resource_num = resconf.GetInt64Value(server::CONFIG_RESOURCE_NUM, 2);
|
||||
// pinned_memory = 1024 * 1024 * pinned_memory;
|
||||
// temp_memory = 1024 * 1024 * temp_memory;
|
||||
// knowhere::FaissGpuResourceMgr::GetInstance().InitDevice(device_id,
|
||||
// pinned_memory,
|
||||
// temp_memory,
|
||||
// resource_num);
|
||||
// }
|
||||
// }
|
||||
//
|
||||
// knowhere::FaissGpuResourceMgr::GetInstance().InitResource();
|
||||
//
|
||||
// auto connections = config.GetChild(server::CONFIG_RESOURCE_CONNECTIONS).GetChildren();
|
||||
// if (connections.empty()) throw "connections config null exception";
|
||||
// for (auto &conn : connections) {
|
||||
// auto &connect_name = conn.first;
|
||||
// auto &connect_conf = conn.second;
|
||||
// auto connect_speed = connect_conf.GetInt64Value(server::CONFIG_SPEED_CONNECTIONS);
|
||||
// auto connect_endpoint = connect_conf.GetValue(server::CONFIG_ENDPOINT_CONNECTIONS);
|
||||
//
|
||||
// std::string delimiter = "===";
|
||||
// std::string left = connect_endpoint.substr(0, connect_endpoint.find(delimiter));
|
||||
// std::string right = connect_endpoint.substr(connect_endpoint.find(delimiter) + 3,
|
||||
// connect_endpoint.length());
|
||||
//
|
||||
// auto connection = Connection(connect_name, connect_speed);
|
||||
// ResMgrInst::GetInstance()->Connect(left, right, connection);
|
||||
// }
|
||||
// } catch (const char *msg) {
|
||||
// SERVER_LOG_ERROR << msg;
|
||||
// // TODO(wxyu): throw exception instead
|
||||
// exit(-1);
|
||||
//// throw std::exception();
|
||||
// }
|
||||
}
|
||||
|
||||
void
|
||||
StartSchedulerService() {
|
||||
load_simple_config();
|
||||
// load_advance_config();
|
||||
ResMgrInst::GetInstance()->Start();
|
||||
SchedInst::GetInstance()->Start();
|
||||
JobMgrInst::GetInstance()->Start();
|
||||
|
||||
@ -23,10 +23,14 @@
|
||||
#include "Scheduler.h"
|
||||
#include "optimizer/HybridPass.h"
|
||||
#include "optimizer/LargeSQ8HPass.h"
|
||||
#include "optimizer/OnlyCPUPass.h"
|
||||
#include "optimizer/OnlyGPUPass.h"
|
||||
#include "optimizer/Optimizer.h"
|
||||
#include "server/Config.h"
|
||||
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace milvus {
|
||||
@ -93,8 +97,20 @@ class OptimizerInst {
|
||||
if (instance == nullptr) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
if (instance == nullptr) {
|
||||
server::Config& config = server::Config::GetInstance();
|
||||
std::vector<std::string> search_resources;
|
||||
bool has_cpu = false;
|
||||
config.GetResourceConfigSearchResources(search_resources);
|
||||
for (auto& resource : search_resources) {
|
||||
if (resource == "cpu") {
|
||||
has_cpu = true;
|
||||
}
|
||||
}
|
||||
std::vector<PassPtr> pass_list;
|
||||
pass_list.push_back(std::make_shared<LargeSQ8HPass>());
|
||||
pass_list.push_back(std::make_shared<HybridPass>());
|
||||
pass_list.push_back(std::make_shared<OnlyCPUPass>());
|
||||
pass_list.push_back(std::make_shared<OnlyGPUPass>(has_cpu));
|
||||
instance = std::make_shared<Optimizer>(pass_list);
|
||||
}
|
||||
}
|
||||
|
||||
@ -26,10 +26,8 @@
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
Scheduler::Scheduler(ResourceMgrWPtr res_mgr) : running_(false), res_mgr_(std::move(res_mgr)) {
|
||||
if (auto mgr = res_mgr_.lock()) {
|
||||
mgr->RegisterSubscriber(std::bind(&Scheduler::PostEvent, this, std::placeholders::_1));
|
||||
}
|
||||
Scheduler::Scheduler(ResourceMgrPtr res_mgr) : running_(false), res_mgr_(std::move(res_mgr)) {
|
||||
res_mgr_->RegisterSubscriber(std::bind(&Scheduler::PostEvent, this, std::placeholders::_1));
|
||||
event_register_.insert(std::make_pair(static_cast<uint64_t>(EventType::START_UP),
|
||||
std::bind(&Scheduler::OnStartUp, this, std::placeholders::_1)));
|
||||
event_register_.insert(std::make_pair(static_cast<uint64_t>(EventType::LOAD_COMPLETED),
|
||||
@ -40,6 +38,10 @@ Scheduler::Scheduler(ResourceMgrWPtr res_mgr) : running_(false), res_mgr_(std::m
|
||||
std::bind(&Scheduler::OnFinishTask, this, std::placeholders::_1)));
|
||||
}
|
||||
|
||||
Scheduler::~Scheduler() {
|
||||
res_mgr_ = nullptr;
|
||||
}
|
||||
|
||||
void
|
||||
Scheduler::Start() {
|
||||
running_ = true;
|
||||
@ -66,9 +68,13 @@ Scheduler::PostEvent(const EventPtr& event) {
|
||||
event_cv_.notify_one();
|
||||
}
|
||||
|
||||
std::string
|
||||
Scheduler::Dump() {
|
||||
return std::string();
|
||||
json
|
||||
Scheduler::Dump() const {
|
||||
json ret{
|
||||
{"running", running_},
|
||||
{"event_queue_length", event_queue_.size()},
|
||||
};
|
||||
return ret;
|
||||
}
|
||||
|
||||
void
|
||||
@ -96,51 +102,45 @@ Scheduler::Process(const EventPtr& event) {
|
||||
void
|
||||
Scheduler::OnLoadCompleted(const EventPtr& event) {
|
||||
auto load_completed_event = std::static_pointer_cast<LoadCompletedEvent>(event);
|
||||
if (auto resource = event->resource_.lock()) {
|
||||
resource->WakeupExecutor();
|
||||
|
||||
auto task_table_type = load_completed_event->task_table_item_->task->label()->Type();
|
||||
switch (task_table_type) {
|
||||
case TaskLabelType::DEFAULT: {
|
||||
Action::DefaultLabelTaskScheduler(res_mgr_, resource, load_completed_event);
|
||||
break;
|
||||
}
|
||||
case TaskLabelType::SPECIFIED_RESOURCE: {
|
||||
Action::SpecifiedResourceLabelTaskScheduler(res_mgr_, resource, load_completed_event);
|
||||
break;
|
||||
}
|
||||
case TaskLabelType::BROADCAST: {
|
||||
if (resource->HasExecutor() == false) {
|
||||
load_completed_event->task_table_item_->Move();
|
||||
}
|
||||
Action::PushTaskToAllNeighbour(load_completed_event->task_table_item_->task, resource);
|
||||
break;
|
||||
}
|
||||
default: { break; }
|
||||
auto resource = event->resource_;
|
||||
resource->WakeupExecutor();
|
||||
|
||||
auto task_table_type = load_completed_event->task_table_item_->task->label()->Type();
|
||||
switch (task_table_type) {
|
||||
case TaskLabelType::DEFAULT: {
|
||||
Action::DefaultLabelTaskScheduler(res_mgr_, resource, load_completed_event);
|
||||
break;
|
||||
}
|
||||
resource->WakeupLoader();
|
||||
case TaskLabelType::SPECIFIED_RESOURCE: {
|
||||
Action::SpecifiedResourceLabelTaskScheduler(res_mgr_, resource, load_completed_event);
|
||||
break;
|
||||
}
|
||||
case TaskLabelType::BROADCAST: {
|
||||
if (resource->HasExecutor() == false) {
|
||||
load_completed_event->task_table_item_->Move();
|
||||
}
|
||||
Action::PushTaskToAllNeighbour(load_completed_event->task_table_item_, resource);
|
||||
break;
|
||||
}
|
||||
default: { break; }
|
||||
}
|
||||
resource->WakeupLoader();
|
||||
}
|
||||
|
||||
void
|
||||
Scheduler::OnStartUp(const EventPtr& event) {
|
||||
if (auto resource = event->resource_.lock()) {
|
||||
resource->WakeupLoader();
|
||||
}
|
||||
event->resource_->WakeupLoader();
|
||||
}
|
||||
|
||||
void
|
||||
Scheduler::OnFinishTask(const EventPtr& event) {
|
||||
if (auto resource = event->resource_.lock()) {
|
||||
resource->WakeupLoader();
|
||||
}
|
||||
event->resource_->WakeupLoader();
|
||||
}
|
||||
|
||||
void
|
||||
Scheduler::OnTaskTableUpdated(const EventPtr& event) {
|
||||
if (auto resource = event->resource_.lock()) {
|
||||
resource->WakeupLoader();
|
||||
}
|
||||
event->resource_->WakeupLoader();
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
|
||||
@ -25,16 +25,18 @@
|
||||
#include <unordered_map>
|
||||
|
||||
#include "ResourceMgr.h"
|
||||
#include "interface/interfaces.h"
|
||||
#include "resource/Resource.h"
|
||||
#include "utils/Log.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
// TODO(wxyu): refactor, not friendly to unittest, logical in framework code
|
||||
class Scheduler {
|
||||
class Scheduler : public interface::dumpable {
|
||||
public:
|
||||
explicit Scheduler(ResourceMgrWPtr res_mgr);
|
||||
explicit Scheduler(ResourceMgrPtr res_mgr);
|
||||
|
||||
~Scheduler();
|
||||
|
||||
Scheduler(const Scheduler&) = delete;
|
||||
Scheduler(Scheduler&&) = delete;
|
||||
@ -57,11 +59,8 @@ class Scheduler {
|
||||
void
|
||||
PostEvent(const EventPtr& event);
|
||||
|
||||
/*
|
||||
* Dump as string;
|
||||
*/
|
||||
std::string
|
||||
Dump();
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
private:
|
||||
/******** Events ********/
|
||||
@ -121,7 +120,7 @@ class Scheduler {
|
||||
|
||||
std::unordered_map<uint64_t, std::function<void(EventPtr)>> event_register_;
|
||||
|
||||
ResourceMgrWPtr res_mgr_;
|
||||
ResourceMgrPtr res_mgr_;
|
||||
std::queue<EventPtr> event_queue_;
|
||||
std::thread worker_thread_;
|
||||
std::mutex event_mutex_;
|
||||
|
||||
@ -70,8 +70,15 @@ TaskCreator::Create(const DeleteJobPtr& job) {
|
||||
std::vector<TaskPtr>
|
||||
TaskCreator::Create(const BuildIndexJobPtr& job) {
|
||||
std::vector<TaskPtr> tasks;
|
||||
// TODO(yukun): remove "disk" hardcode here
|
||||
ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource("disk");
|
||||
server::Config& config = server::Config::GetInstance();
|
||||
int32_t build_index_id;
|
||||
Status stat = config.GetResourceConfigIndexBuildDevice(build_index_id);
|
||||
ResourcePtr res_ptr;
|
||||
if (build_index_id == server::CPU_DEVICE_ID) {
|
||||
res_ptr = ResMgrInst::GetInstance()->GetResource("cpu");
|
||||
} else {
|
||||
res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, build_index_id);
|
||||
}
|
||||
|
||||
for (auto& to_index_file : job->to_index_files()) {
|
||||
auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
|
||||
|
||||
@ -20,6 +20,7 @@
|
||||
#include "event/TaskTableUpdatedEvent.h"
|
||||
#include "scheduler/SchedInst.h"
|
||||
#include "utils/Log.h"
|
||||
#include "utils/TimeRecorder.h"
|
||||
|
||||
#include <ctime>
|
||||
#include <sstream>
|
||||
@ -53,7 +54,7 @@ ToString(TaskTableItemState state) {
|
||||
}
|
||||
|
||||
json
|
||||
TaskTimestamp::Dump() {
|
||||
TaskTimestamp::Dump() const {
|
||||
json ret{
|
||||
{"start", start}, {"load", load}, {"loaded", loaded}, {"execute", execute},
|
||||
{"executed", executed}, {"move", move}, {"moved", moved}, {"finish", finish},
|
||||
@ -141,7 +142,7 @@ TaskTableItem::Moved() {
|
||||
}
|
||||
|
||||
json
|
||||
TaskTableItem::Dump() {
|
||||
TaskTableItem::Dump() const {
|
||||
json ret{
|
||||
{"id", id},
|
||||
{"task", (int64_t)task.get()},
|
||||
@ -153,7 +154,42 @@ TaskTableItem::Dump() {
|
||||
|
||||
std::vector<uint64_t>
|
||||
TaskTable::PickToLoad(uint64_t limit) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
#if 1
|
||||
TimeRecorder rc("");
|
||||
std::vector<uint64_t> indexes;
|
||||
bool cross = false;
|
||||
|
||||
uint64_t available_begin = table_.front() + 1;
|
||||
for (uint64_t i = 0, loaded_count = 0, pick_count = 0; i < table_.size() && pick_count < limit; ++i) {
|
||||
auto index = available_begin + i;
|
||||
if (not table_[index])
|
||||
break;
|
||||
if (index % table_.capacity() == table_.rear())
|
||||
break;
|
||||
if (not cross && table_[index]->IsFinish()) {
|
||||
table_.set_front(index);
|
||||
} else if (table_[index]->state == TaskTableItemState::LOADED) {
|
||||
cross = true;
|
||||
++loaded_count;
|
||||
if (loaded_count > 2)
|
||||
return std::vector<uint64_t>();
|
||||
} else if (table_[index]->state == TaskTableItemState::START) {
|
||||
auto task = table_[index]->task;
|
||||
|
||||
// if task is a build index task, limit it
|
||||
if (task->Type() == TaskType::BuildIndexTask && task->path().Current() == "cpu") {
|
||||
if (not BuildMgrInst::GetInstance()->Take()) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
cross = true;
|
||||
indexes.push_back(index);
|
||||
++pick_count;
|
||||
}
|
||||
}
|
||||
rc.ElapseFromBegin("PickToLoad ");
|
||||
return indexes;
|
||||
#else
|
||||
size_t count = 0;
|
||||
for (uint64_t j = last_finish_ + 1; j < table_.size(); ++j) {
|
||||
if (not table_[j]) {
|
||||
@ -197,77 +233,65 @@ TaskTable::PickToLoad(uint64_t limit) {
|
||||
}
|
||||
}
|
||||
return indexes;
|
||||
#endif
|
||||
}
|
||||
|
||||
std::vector<uint64_t>
|
||||
TaskTable::PickToExecute(uint64_t limit) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
TimeRecorder rc("");
|
||||
std::vector<uint64_t> indexes;
|
||||
bool cross = false;
|
||||
for (uint64_t i = last_finish_ + 1, count = 0; i < table_.size() && count < limit; ++i) {
|
||||
if (not cross && table_[i]->IsFinish()) {
|
||||
last_finish_ = i;
|
||||
} else if (table_[i]->state == TaskTableItemState::LOADED) {
|
||||
uint64_t available_begin = table_.front() + 1;
|
||||
for (uint64_t i = 0, pick_count = 0; i < table_.size() && pick_count < limit; ++i) {
|
||||
uint64_t index = available_begin + i;
|
||||
if (not table_[index]) {
|
||||
break;
|
||||
}
|
||||
if (index % table_.capacity() == table_.rear()) {
|
||||
break;
|
||||
}
|
||||
|
||||
if (not cross && table_[index]->IsFinish()) {
|
||||
table_.set_front(index);
|
||||
} else if (table_[index]->state == TaskTableItemState::LOADED) {
|
||||
cross = true;
|
||||
indexes.push_back(i);
|
||||
++count;
|
||||
indexes.push_back(index);
|
||||
++pick_count;
|
||||
}
|
||||
}
|
||||
rc.ElapseFromBegin("PickToExecute ");
|
||||
return indexes;
|
||||
}
|
||||
|
||||
void
|
||||
TaskTable::Put(TaskPtr task) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
auto item = std::make_shared<TaskTableItem>();
|
||||
TaskTable::Put(TaskPtr task, TaskTableItemPtr from) {
|
||||
auto item = std::make_shared<TaskTableItem>(std::move(from));
|
||||
item->id = id_++;
|
||||
item->task = std::move(task);
|
||||
item->state = TaskTableItemState::START;
|
||||
item->timestamp.start = get_current_timestamp();
|
||||
table_.push_back(item);
|
||||
table_.put(std::move(item));
|
||||
if (subscriber_) {
|
||||
subscriber_();
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
TaskTable::Put(std::vector<TaskPtr>& tasks) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
for (auto& task : tasks) {
|
||||
auto item = std::make_shared<TaskTableItem>();
|
||||
item->id = id_++;
|
||||
item->task = std::move(task);
|
||||
item->state = TaskTableItemState::START;
|
||||
item->timestamp.start = get_current_timestamp();
|
||||
table_.push_back(item);
|
||||
}
|
||||
if (subscriber_) {
|
||||
subscriber_();
|
||||
size_t
|
||||
TaskTable::TaskToExecute() {
|
||||
size_t count = 0;
|
||||
auto begin = table_.front() + 1;
|
||||
for (size_t i = 0; i < table_.size(); ++i) {
|
||||
auto index = begin + i;
|
||||
if (table_[index]->state == TaskTableItemState::LOADED) {
|
||||
++count;
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
TaskTableItemPtr
|
||||
TaskTable::Get(uint64_t index) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return table_[index];
|
||||
}
|
||||
|
||||
// void
|
||||
// TaskTable::Clear() {
|
||||
//// find first task is NOT (done or moved), erase from begin to it;
|
||||
//// auto iterator = table_.begin();
|
||||
//// while (iterator->state == TaskTableItemState::EXECUTED or
|
||||
//// iterator->state == TaskTableItemState::MOVED)
|
||||
//// iterator++;
|
||||
//// table_.erase(table_.begin(), iterator);
|
||||
//}
|
||||
|
||||
json
|
||||
TaskTable::Dump() {
|
||||
json ret;
|
||||
for (auto& item : table_) {
|
||||
ret.push_back(item->Dump());
|
||||
}
|
||||
TaskTable::Dump() const {
|
||||
json ret{{"error.message", "not support yet."}};
|
||||
return ret;
|
||||
}
|
||||
|
||||
|
||||
@ -25,6 +25,7 @@
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#include "CircleQueue.h"
|
||||
#include "event/Event.h"
|
||||
#include "interface/interfaces.h"
|
||||
#include "task/SearchTask.h"
|
||||
@ -54,11 +55,15 @@ struct TaskTimestamp : public interface::dumpable {
|
||||
uint64_t finish = 0;
|
||||
|
||||
json
|
||||
Dump() override;
|
||||
Dump() const override;
|
||||
};
|
||||
|
||||
struct TaskTableItem;
|
||||
using TaskTableItemPtr = std::shared_ptr<TaskTableItem>;
|
||||
|
||||
struct TaskTableItem : public interface::dumpable {
|
||||
TaskTableItem() : id(0), task(nullptr), state(TaskTableItemState::INVALID), mutex() {
|
||||
explicit TaskTableItem(TaskTableItemPtr f = nullptr)
|
||||
: id(0), task(nullptr), state(TaskTableItemState::INVALID), mutex(), from(std::move(f)) {
|
||||
}
|
||||
|
||||
TaskTableItem(const TaskTableItem& src) = delete;
|
||||
@ -69,6 +74,7 @@ struct TaskTableItem : public interface::dumpable {
|
||||
TaskTableItemState state; // the state;
|
||||
std::mutex mutex;
|
||||
TaskTimestamp timestamp;
|
||||
TaskTableItemPtr from;
|
||||
|
||||
bool
|
||||
IsFinish();
|
||||
@ -92,18 +98,22 @@ struct TaskTableItem : public interface::dumpable {
|
||||
Moved();
|
||||
|
||||
json
|
||||
Dump() override;
|
||||
Dump() const override;
|
||||
};
|
||||
|
||||
using TaskTableItemPtr = std::shared_ptr<TaskTableItem>;
|
||||
|
||||
class TaskTable : public interface::dumpable {
|
||||
public:
|
||||
TaskTable() = default;
|
||||
TaskTable() : table_(1ULL << 16ULL) {
|
||||
}
|
||||
|
||||
TaskTable(const TaskTable&) = delete;
|
||||
TaskTable(TaskTable&&) = delete;
|
||||
|
||||
public:
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
public:
|
||||
inline void
|
||||
RegisterSubscriber(std::function<void(void)> subscriber) {
|
||||
subscriber_ = std::move(subscriber);
|
||||
@ -113,68 +123,37 @@ class TaskTable : public interface::dumpable {
|
||||
* Put one task;
|
||||
*/
|
||||
void
|
||||
Put(TaskPtr task);
|
||||
Put(TaskPtr task, TaskTableItemPtr from = nullptr);
|
||||
|
||||
/*
|
||||
* Put tasks back of task table;
|
||||
* Called by DBImpl;
|
||||
*/
|
||||
void
|
||||
Put(std::vector<TaskPtr>& tasks);
|
||||
size_t
|
||||
TaskToExecute();
|
||||
|
||||
/*
|
||||
* Return task table item reference;
|
||||
*/
|
||||
TaskTableItemPtr
|
||||
Get(uint64_t index);
|
||||
|
||||
/*
|
||||
* TODO(wxyu): BIG GC
|
||||
* Remove sequence task which is DONE or MOVED from front;
|
||||
* Called by ?
|
||||
*/
|
||||
// void
|
||||
// Clear();
|
||||
|
||||
/*
|
||||
* Return true if task table empty, otherwise false;
|
||||
*/
|
||||
inline bool
|
||||
Empty() {
|
||||
return table_.empty();
|
||||
}
|
||||
|
||||
/*
|
||||
* Return size of task table;
|
||||
*/
|
||||
inline size_t
|
||||
Size() {
|
||||
return table_.size();
|
||||
}
|
||||
|
||||
public:
|
||||
TaskTableItemPtr& operator[](uint64_t index) {
|
||||
std::lock_guard<std::mutex> lock(mutex_);
|
||||
return table_[index];
|
||||
}
|
||||
|
||||
std::deque<TaskTableItemPtr>::iterator
|
||||
begin() {
|
||||
return table_.begin();
|
||||
}
|
||||
|
||||
std::deque<TaskTableItemPtr>::iterator
|
||||
end() {
|
||||
return table_.end();
|
||||
}
|
||||
|
||||
public:
|
||||
std::vector<uint64_t>
|
||||
PickToLoad(uint64_t limit);
|
||||
|
||||
std::vector<uint64_t>
|
||||
PickToExecute(uint64_t limit);
|
||||
|
||||
public:
|
||||
inline const TaskTableItemPtr& operator[](uint64_t index) {
|
||||
return table_[index];
|
||||
}
|
||||
|
||||
inline const TaskTableItemPtr&
|
||||
at(uint64_t index) {
|
||||
return table_[index];
|
||||
}
|
||||
|
||||
inline size_t
|
||||
capacity() {
|
||||
return table_.capacity();
|
||||
}
|
||||
|
||||
inline size_t
|
||||
size() {
|
||||
return table_.size();
|
||||
}
|
||||
|
||||
public:
|
||||
/******** Action ********/
|
||||
|
||||
@ -240,17 +219,9 @@ class TaskTable : public interface::dumpable {
|
||||
return table_[index]->Moved();
|
||||
}
|
||||
|
||||
public:
|
||||
/*
|
||||
* Dump;
|
||||
*/
|
||||
json
|
||||
Dump() override;
|
||||
|
||||
private:
|
||||
std::uint64_t id_ = 0;
|
||||
mutable std::mutex mutex_;
|
||||
std::deque<TaskTableItemPtr> table_;
|
||||
CircleQueue<TaskTableItemPtr> table_;
|
||||
std::function<void(void)> subscriber_ = nullptr;
|
||||
|
||||
// cache last finish avoid Pick task from begin always
|
||||
|
||||
@ -28,19 +28,20 @@ namespace scheduler {
|
||||
class Action {
|
||||
public:
|
||||
static void
|
||||
PushTaskToNeighbourRandomly(const TaskPtr& task, const ResourcePtr& self);
|
||||
PushTaskToNeighbourRandomly(TaskTableItemPtr task_item, const ResourcePtr& self);
|
||||
|
||||
static void
|
||||
PushTaskToAllNeighbour(const TaskPtr& task, const ResourcePtr& self);
|
||||
PushTaskToAllNeighbour(TaskTableItemPtr task_item, const ResourcePtr& self);
|
||||
|
||||
static void
|
||||
PushTaskToResource(const TaskPtr& task, const ResourcePtr& dest);
|
||||
PushTaskToResource(TaskTableItemPtr task_item, const ResourcePtr& dest);
|
||||
|
||||
static void
|
||||
DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource, std::shared_ptr<LoadCompletedEvent> event);
|
||||
DefaultLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
|
||||
std::shared_ptr<LoadCompletedEvent> event);
|
||||
|
||||
static void
|
||||
SpecifiedResourceLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
|
||||
SpecifiedResourceLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
|
||||
std::shared_ptr<LoadCompletedEvent> event);
|
||||
};
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ std::vector<ResourcePtr>
|
||||
get_neighbours(const ResourcePtr& self) {
|
||||
std::vector<ResourcePtr> neighbours;
|
||||
for (auto& neighbour_node : self->GetNeighbours()) {
|
||||
auto node = neighbour_node.neighbour_node.lock();
|
||||
auto node = neighbour_node.neighbour_node;
|
||||
if (not node)
|
||||
continue;
|
||||
|
||||
@ -46,7 +46,7 @@ std::vector<std::pair<ResourcePtr, Connection>>
|
||||
get_neighbours_with_connetion(const ResourcePtr& self) {
|
||||
std::vector<std::pair<ResourcePtr, Connection>> neighbours;
|
||||
for (auto& neighbour_node : self->GetNeighbours()) {
|
||||
auto node = neighbour_node.neighbour_node.lock();
|
||||
auto node = neighbour_node.neighbour_node;
|
||||
if (not node)
|
||||
continue;
|
||||
|
||||
@ -59,7 +59,7 @@ get_neighbours_with_connetion(const ResourcePtr& self) {
|
||||
}
|
||||
|
||||
void
|
||||
Action::PushTaskToNeighbourRandomly(const TaskPtr& task, const ResourcePtr& self) {
|
||||
Action::PushTaskToNeighbourRandomly(TaskTableItemPtr task_item, const ResourcePtr& self) {
|
||||
auto neighbours = get_neighbours_with_connetion(self);
|
||||
if (not neighbours.empty()) {
|
||||
std::vector<uint64_t> speeds;
|
||||
@ -78,7 +78,7 @@ Action::PushTaskToNeighbourRandomly(const TaskPtr& task, const ResourcePtr& self
|
||||
for (uint64_t i = 0; i < speeds.size(); ++i) {
|
||||
rd_speed -= speeds[i];
|
||||
if (rd_speed <= 0) {
|
||||
neighbours[i].first->task_table().Put(task);
|
||||
neighbours[i].first->task_table().Put(task_item->task, task_item);
|
||||
return;
|
||||
}
|
||||
}
|
||||
@ -89,22 +89,23 @@ Action::PushTaskToNeighbourRandomly(const TaskPtr& task, const ResourcePtr& self
|
||||
}
|
||||
|
||||
void
|
||||
Action::PushTaskToAllNeighbour(const TaskPtr& task, const ResourcePtr& self) {
|
||||
Action::PushTaskToAllNeighbour(TaskTableItemPtr task_item, const ResourcePtr& self) {
|
||||
auto neighbours = get_neighbours(self);
|
||||
for (auto& neighbour : neighbours) {
|
||||
neighbour->task_table().Put(task);
|
||||
neighbour->task_table().Put(task_item->task, task_item);
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
Action::PushTaskToResource(const TaskPtr& task, const ResourcePtr& dest) {
|
||||
dest->task_table().Put(task);
|
||||
Action::PushTaskToResource(TaskTableItemPtr task_item, const ResourcePtr& dest) {
|
||||
dest->task_table().Put(task_item->task, task_item);
|
||||
}
|
||||
|
||||
void
|
||||
Action::DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
|
||||
Action::DefaultLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
|
||||
std::shared_ptr<LoadCompletedEvent> event) {
|
||||
if (not resource->HasExecutor() && event->task_table_item_->Move()) {
|
||||
auto task_item = event->task_table_item_;
|
||||
auto task = event->task_table_item_->task;
|
||||
auto search_task = std::static_pointer_cast<XSearchTask>(task);
|
||||
bool moved = false;
|
||||
@ -114,12 +115,12 @@ Action::DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
|
||||
if (auto index_engine = search_task->index_engine_) {
|
||||
auto location = index_engine->GetLocation();
|
||||
|
||||
for (auto i = 0; i < res_mgr.lock()->GetNumGpuResource(); ++i) {
|
||||
for (auto i = 0; i < res_mgr->GetNumGpuResource(); ++i) {
|
||||
auto index = milvus::cache::GpuCacheMgr::GetInstance(i)->GetIndex(location);
|
||||
if (index != nullptr) {
|
||||
moved = true;
|
||||
auto dest_resource = res_mgr.lock()->GetResource(ResourceType::GPU, i);
|
||||
PushTaskToResource(event->task_table_item_->task, dest_resource);
|
||||
auto dest_resource = res_mgr->GetResource(ResourceType::GPU, i);
|
||||
PushTaskToResource(event->task_table_item_, dest_resource);
|
||||
break;
|
||||
}
|
||||
}
|
||||
@ -127,93 +128,62 @@ Action::DefaultLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
|
||||
}
|
||||
|
||||
if (not moved) {
|
||||
PushTaskToNeighbourRandomly(task, resource);
|
||||
PushTaskToNeighbourRandomly(task_item, resource);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
Action::SpecifiedResourceLabelTaskScheduler(ResourceMgrWPtr res_mgr, ResourcePtr resource,
|
||||
Action::SpecifiedResourceLabelTaskScheduler(const ResourceMgrPtr& res_mgr, ResourcePtr resource,
|
||||
std::shared_ptr<LoadCompletedEvent> event) {
|
||||
auto task_item = event->task_table_item_;
|
||||
auto task = event->task_table_item_->task;
|
||||
if (resource->type() == ResourceType::DISK) {
|
||||
// step 1: calculate shortest path per resource, from disk to compute resource
|
||||
auto compute_resources = res_mgr.lock()->GetComputeResources();
|
||||
std::vector<std::vector<std::string>> paths;
|
||||
std::vector<uint64_t> transport_costs;
|
||||
for (auto& res : compute_resources) {
|
||||
std::vector<std::string> path;
|
||||
uint64_t transport_cost = ShortestPath(resource, res, res_mgr.lock(), path);
|
||||
transport_costs.push_back(transport_cost);
|
||||
paths.emplace_back(path);
|
||||
}
|
||||
// if (task->job_.lock()->type() == JobType::SEARCH) {
|
||||
// auto label = task->label();
|
||||
// auto spec_label = std::static_pointer_cast<SpecResLabel>(label);
|
||||
// if (spec_label->resource().lock()->type() == ResourceType::CPU) {
|
||||
// std::vector<std::string> spec_path;
|
||||
// spec_path.push_back(spec_label->resource().lock()->name());
|
||||
// spec_path.push_back(resource->name());
|
||||
// task->path() = Path(spec_path, spec_path.size() - 1);
|
||||
// } else {
|
||||
// // step 2: select min cost, cost(resource) = avg_cost * task_to_do + transport_cost
|
||||
// uint64_t min_cost = std::numeric_limits<uint64_t>::max();
|
||||
// uint64_t min_cost_idx = 0;
|
||||
// for (uint64_t i = 0; i < compute_resources.size(); ++i) {
|
||||
// if (compute_resources[i]->TotalTasks() == 0) {
|
||||
// min_cost_idx = i;
|
||||
// break;
|
||||
// }
|
||||
// uint64_t cost = compute_resources[i]->TaskAvgCost() *
|
||||
// compute_resources[i]->NumOfTaskToExec() +
|
||||
// transport_costs[i];
|
||||
// if (min_cost > cost) {
|
||||
// min_cost = cost;
|
||||
// min_cost_idx = i;
|
||||
// }
|
||||
// }
|
||||
//
|
||||
// // step 3: set path in task
|
||||
// Path task_path(paths[min_cost_idx], paths[min_cost_idx].size() - 1);
|
||||
// task->path() = task_path;
|
||||
// }
|
||||
//
|
||||
// } else
|
||||
if (task->job_.lock()->type() == JobType::BUILD) {
|
||||
// step2: Read device id in config
|
||||
// get build index gpu resource
|
||||
server::Config& config = server::Config::GetInstance();
|
||||
int32_t build_index_gpu;
|
||||
Status stat = config.GetResourceConfigIndexBuildDevice(build_index_gpu);
|
||||
|
||||
bool find_gpu_res = false;
|
||||
if (res_mgr.lock()->GetResource(ResourceType::GPU, build_index_gpu) != nullptr) {
|
||||
for (uint64_t i = 0; i < compute_resources.size(); ++i) {
|
||||
if (compute_resources[i]->name() ==
|
||||
res_mgr.lock()->GetResource(ResourceType::GPU, build_index_gpu)->name()) {
|
||||
find_gpu_res = true;
|
||||
Path task_path(paths[i], paths[i].size() - 1);
|
||||
task->path() = task_path;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (not find_gpu_res) {
|
||||
task->path() = Path(paths[0], paths[0].size() - 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
// if (resource->type() == ResourceType::DISK) {
|
||||
// // step 1: calculate shortest path per resource, from disk to compute resource
|
||||
// auto compute_resources = res_mgr->GetComputeResources();
|
||||
// std::vector<std::vector<std::string>> paths;
|
||||
// std::vector<uint64_t> transport_costs;
|
||||
// for (auto& res : compute_resources) {
|
||||
// std::vector<std::string> path;
|
||||
// uint64_t transport_cost = ShortestPath(resource, res, res_mgr, path);
|
||||
// transport_costs.push_back(transport_cost);
|
||||
// paths.emplace_back(path);
|
||||
// }
|
||||
// if (task->job_.lock()->type() == JobType::BUILD) {
|
||||
// // step2: Read device id in config
|
||||
// // get build index gpu resource
|
||||
// server::Config& config = server::Config::GetInstance();
|
||||
// int32_t build_index_gpu;
|
||||
// Status stat = config.GetResourceConfigIndexBuildDevice(build_index_gpu);
|
||||
//
|
||||
// bool find_gpu_res = false;
|
||||
// if (res_mgr->GetResource(ResourceType::GPU, build_index_gpu) != nullptr) {
|
||||
// for (uint64_t i = 0; i < compute_resources.size(); ++i) {
|
||||
// if (compute_resources[i]->name() ==
|
||||
// res_mgr->GetResource(ResourceType::GPU, build_index_gpu)->name()) {
|
||||
// find_gpu_res = true;
|
||||
// Path task_path(paths[i], paths[i].size() - 1);
|
||||
// task->path() = task_path;
|
||||
// break;
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
// if (not find_gpu_res) {
|
||||
// task->path() = Path(paths[0], paths[0].size() - 1);
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
|
||||
if (resource->name() == task->path().Last()) {
|
||||
resource->WakeupExecutor();
|
||||
} else {
|
||||
auto next_res_name = task->path().Next();
|
||||
auto next_res = res_mgr.lock()->GetResource(next_res_name);
|
||||
auto next_res = res_mgr->GetResource(next_res_name);
|
||||
// if (event->task_table_item_->Move()) {
|
||||
// next_res->task_table().Put(task);
|
||||
// }
|
||||
event->task_table_item_->Move();
|
||||
next_res->task_table().Put(task);
|
||||
next_res->task_table().Put(task, task_item);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ class Resource;
|
||||
|
||||
class Event {
|
||||
public:
|
||||
explicit Event(EventType type, std::weak_ptr<Resource> resource) : type_(type), resource_(std::move(resource)) {
|
||||
explicit Event(EventType type, std::shared_ptr<Resource> resource) : type_(type), resource_(std::move(resource)) {
|
||||
}
|
||||
|
||||
inline EventType
|
||||
@ -46,7 +46,7 @@ class Event {
|
||||
|
||||
public:
|
||||
EventType type_;
|
||||
std::weak_ptr<Resource> resource_;
|
||||
std::shared_ptr<Resource> resource_;
|
||||
};
|
||||
|
||||
using EventPtr = std::shared_ptr<Event>;
|
||||
|
||||
@ -29,7 +29,7 @@ namespace scheduler {
|
||||
|
||||
class FinishTaskEvent : public Event {
|
||||
public:
|
||||
FinishTaskEvent(std::weak_ptr<Resource> resource, TaskTableItemPtr task_table_item)
|
||||
FinishTaskEvent(std::shared_ptr<Resource> resource, TaskTableItemPtr task_table_item)
|
||||
: Event(EventType::FINISH_TASK, std::move(resource)), task_table_item_(std::move(task_table_item)) {
|
||||
}
|
||||
|
||||
|
||||
@ -29,7 +29,7 @@ namespace scheduler {
|
||||
|
||||
class LoadCompletedEvent : public Event {
|
||||
public:
|
||||
LoadCompletedEvent(std::weak_ptr<Resource> resource, TaskTableItemPtr task_table_item)
|
||||
LoadCompletedEvent(std::shared_ptr<Resource> resource, TaskTableItemPtr task_table_item)
|
||||
: Event(EventType::LOAD_COMPLETED, std::move(resource)), task_table_item_(std::move(task_table_item)) {
|
||||
}
|
||||
|
||||
|
||||
@ -28,7 +28,7 @@ namespace scheduler {
|
||||
|
||||
class StartUpEvent : public Event {
|
||||
public:
|
||||
explicit StartUpEvent(std::weak_ptr<Resource> resource) : Event(EventType::START_UP, std::move(resource)) {
|
||||
explicit StartUpEvent(std::shared_ptr<Resource> resource) : Event(EventType::START_UP, std::move(resource)) {
|
||||
}
|
||||
|
||||
inline std::string
|
||||
|
||||
@ -28,7 +28,7 @@ namespace scheduler {
|
||||
|
||||
class TaskTableUpdatedEvent : public Event {
|
||||
public:
|
||||
explicit TaskTableUpdatedEvent(std::weak_ptr<Resource> resource)
|
||||
explicit TaskTableUpdatedEvent(std::shared_ptr<Resource> resource)
|
||||
: Event(EventType::TASK_TABLE_UPDATED, std::move(resource)) {
|
||||
}
|
||||
|
||||
|
||||
@ -37,7 +37,7 @@ struct dumpable {
|
||||
}
|
||||
|
||||
virtual json
|
||||
Dump() = 0;
|
||||
Dump() const = 0;
|
||||
};
|
||||
|
||||
} // namespace interface
|
||||
|
||||
@ -23,8 +23,8 @@
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
BuildIndexJob::BuildIndexJob(JobId id, engine::meta::MetaPtr meta_ptr, engine::DBOptions options)
|
||||
: Job(id, JobType::BUILD), meta_ptr_(std::move(meta_ptr)), options_(std::move(options)) {
|
||||
BuildIndexJob::BuildIndexJob(engine::meta::MetaPtr meta_ptr, engine::DBOptions options)
|
||||
: Job(JobType::BUILD), meta_ptr_(std::move(meta_ptr)), options_(std::move(options)) {
|
||||
}
|
||||
|
||||
bool
|
||||
@ -50,9 +50,22 @@ void
|
||||
BuildIndexJob::BuildIndexDone(size_t to_index_id) {
|
||||
std::unique_lock<std::mutex> lock(mutex_);
|
||||
to_index_files_.erase(to_index_id);
|
||||
cv_.notify_all();
|
||||
if (to_index_files_.empty()) {
|
||||
cv_.notify_all();
|
||||
}
|
||||
|
||||
SERVER_LOG_DEBUG << "BuildIndexJob " << id() << " finish index file: " << to_index_id;
|
||||
}
|
||||
|
||||
json
|
||||
BuildIndexJob::Dump() const {
|
||||
json ret{
|
||||
{"number_of_to_index_file", to_index_files_.size()},
|
||||
};
|
||||
auto base = Job::Dump();
|
||||
ret.insert(base.begin(), base.end());
|
||||
return ret;
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
|
||||
@ -41,7 +41,7 @@ using Id2ToTableFileMap = std::unordered_map<size_t, TableFileSchema>;
|
||||
|
||||
class BuildIndexJob : public Job {
|
||||
public:
|
||||
explicit BuildIndexJob(JobId id, engine::meta::MetaPtr meta_ptr, engine::DBOptions options);
|
||||
explicit BuildIndexJob(engine::meta::MetaPtr meta_ptr, engine::DBOptions options);
|
||||
|
||||
public:
|
||||
bool
|
||||
@ -53,6 +53,9 @@ class BuildIndexJob : public Job {
|
||||
void
|
||||
BuildIndexDone(size_t to_index_id);
|
||||
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
public:
|
||||
Status&
|
||||
GetStatus() {
|
||||
|
||||
@ -22,8 +22,8 @@
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
DeleteJob::DeleteJob(JobId id, std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource)
|
||||
: Job(id, JobType::DELETE),
|
||||
DeleteJob::DeleteJob(std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource)
|
||||
: Job(JobType::DELETE),
|
||||
table_id_(std::move(table_id)),
|
||||
meta_ptr_(std::move(meta_ptr)),
|
||||
num_resource_(num_resource) {
|
||||
@ -45,5 +45,17 @@ DeleteJob::ResourceDone() {
|
||||
cv_.notify_one();
|
||||
}
|
||||
|
||||
json
|
||||
DeleteJob::Dump() const {
|
||||
json ret{
|
||||
{"table_id", table_id_},
|
||||
{"number_of_resource", num_resource_},
|
||||
{"number_of_done", done_resource},
|
||||
};
|
||||
auto base = Job::Dump();
|
||||
ret.insert(base.begin(), base.end());
|
||||
return ret;
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
|
||||
@ -35,7 +35,7 @@ namespace scheduler {
|
||||
|
||||
class DeleteJob : public Job {
|
||||
public:
|
||||
DeleteJob(JobId id, std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource);
|
||||
DeleteJob(std::string table_id, engine::meta::MetaPtr meta_ptr, uint64_t num_resource);
|
||||
|
||||
public:
|
||||
void
|
||||
@ -44,6 +44,9 @@ class DeleteJob : public Job {
|
||||
void
|
||||
ResourceDone();
|
||||
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
public:
|
||||
std::string
|
||||
table_id() const {
|
||||
|
||||
43
core/src/scheduler/job/Job.cpp
Normal file
43
core/src/scheduler/job/Job.cpp
Normal file
@ -0,0 +1,43 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include "scheduler/job/Job.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
namespace {
|
||||
std::mutex unique_job_mutex;
|
||||
uint64_t unique_job_id = 0;
|
||||
} // namespace
|
||||
|
||||
Job::Job(JobType type) : type_(type) {
|
||||
std::lock_guard<std::mutex> lock(unique_job_mutex);
|
||||
id_ = unique_job_id++;
|
||||
}
|
||||
|
||||
json
|
||||
Job::Dump() const {
|
||||
json ret{
|
||||
{"id", id_},
|
||||
{"type", type_},
|
||||
};
|
||||
return ret;
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
@ -27,6 +27,8 @@
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#include "scheduler/interface/interfaces.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
@ -39,7 +41,7 @@ enum class JobType {
|
||||
|
||||
using JobId = std::uint64_t;
|
||||
|
||||
class Job {
|
||||
class Job : public interface::dumpable {
|
||||
public:
|
||||
inline JobId
|
||||
id() const {
|
||||
@ -51,12 +53,14 @@ class Job {
|
||||
return type_;
|
||||
}
|
||||
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
protected:
|
||||
Job(JobId id, JobType type) : id_(id), type_(type) {
|
||||
}
|
||||
explicit Job(JobType type);
|
||||
|
||||
private:
|
||||
JobId id_;
|
||||
JobId id_ = 0;
|
||||
JobType type_;
|
||||
};
|
||||
|
||||
|
||||
@ -21,8 +21,8 @@
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
SearchJob::SearchJob(milvus::scheduler::JobId id, uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors)
|
||||
: Job(id, JobType::SEARCH), topk_(topk), nq_(nq), nprobe_(nprobe), vectors_(vectors) {
|
||||
SearchJob::SearchJob(uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors)
|
||||
: Job(JobType::SEARCH), topk_(topk), nq_(nq), nprobe_(nprobe), vectors_(vectors) {
|
||||
}
|
||||
|
||||
bool
|
||||
@ -49,13 +49,21 @@ void
|
||||
SearchJob::SearchDone(size_t index_id) {
|
||||
std::unique_lock<std::mutex> lock(mutex_);
|
||||
index_files_.erase(index_id);
|
||||
cv_.notify_all();
|
||||
if (index_files_.empty()) {
|
||||
cv_.notify_all();
|
||||
}
|
||||
|
||||
SERVER_LOG_DEBUG << "SearchJob " << id() << " finish index file: " << index_id;
|
||||
}
|
||||
|
||||
ResultSet&
|
||||
SearchJob::GetResult() {
|
||||
return result_;
|
||||
ResultIds&
|
||||
SearchJob::GetResultIds() {
|
||||
return result_ids_;
|
||||
}
|
||||
|
||||
ResultDistances&
|
||||
SearchJob::GetResultDistances() {
|
||||
return result_distances_;
|
||||
}
|
||||
|
||||
Status&
|
||||
@ -63,5 +71,17 @@ SearchJob::GetStatus() {
|
||||
return status_;
|
||||
}
|
||||
|
||||
json
|
||||
SearchJob::Dump() const {
|
||||
json ret{
|
||||
{"topk", topk_},
|
||||
{"nq", nq_},
|
||||
{"nprobe", nprobe_},
|
||||
};
|
||||
auto base = Job::Dump();
|
||||
ret.insert(base.begin(), base.end());
|
||||
return ret;
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
|
||||
@ -29,6 +29,7 @@
|
||||
#include <vector>
|
||||
|
||||
#include "Job.h"
|
||||
#include "db/Types.h"
|
||||
#include "db/meta/MetaTypes.h"
|
||||
|
||||
namespace milvus {
|
||||
@ -37,13 +38,13 @@ namespace scheduler {
|
||||
using engine::meta::TableFileSchemaPtr;
|
||||
|
||||
using Id2IndexMap = std::unordered_map<size_t, TableFileSchemaPtr>;
|
||||
using IdDistPair = std::pair<int64_t, double>;
|
||||
using Id2DistVec = std::vector<IdDistPair>;
|
||||
using ResultSet = std::vector<Id2DistVec>;
|
||||
|
||||
using ResultIds = engine::ResultIds;
|
||||
using ResultDistances = engine::ResultDistances;
|
||||
|
||||
class SearchJob : public Job {
|
||||
public:
|
||||
SearchJob(JobId id, uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors);
|
||||
SearchJob(uint64_t topk, uint64_t nq, uint64_t nprobe, const float* vectors);
|
||||
|
||||
public:
|
||||
bool
|
||||
@ -55,12 +56,18 @@ class SearchJob : public Job {
|
||||
void
|
||||
SearchDone(size_t index_id);
|
||||
|
||||
ResultSet&
|
||||
GetResult();
|
||||
ResultIds&
|
||||
GetResultIds();
|
||||
|
||||
ResultDistances&
|
||||
GetResultDistances();
|
||||
|
||||
Status&
|
||||
GetStatus();
|
||||
|
||||
json
|
||||
Dump() const override;
|
||||
|
||||
public:
|
||||
uint64_t
|
||||
topk() const {
|
||||
@ -87,6 +94,11 @@ class SearchJob : public Job {
|
||||
return index_files_;
|
||||
}
|
||||
|
||||
std::mutex&
|
||||
mutex() {
|
||||
return mutex_;
|
||||
}
|
||||
|
||||
private:
|
||||
uint64_t topk_ = 0;
|
||||
uint64_t nq_ = 0;
|
||||
@ -96,7 +108,8 @@ class SearchJob : public Job {
|
||||
|
||||
Id2IndexMap index_files_;
|
||||
// TODO: column-base better ?
|
||||
ResultSet result_;
|
||||
ResultIds result_ids_;
|
||||
ResultDistances result_distances_;
|
||||
Status status_;
|
||||
|
||||
std::mutex mutex_;
|
||||
|
||||
@ -21,53 +21,65 @@
|
||||
#include "scheduler/Utils.h"
|
||||
#include "scheduler/task/SearchTask.h"
|
||||
#include "scheduler/tasklabel/SpecResLabel.h"
|
||||
#include "server/Config.h"
|
||||
#include "utils/Log.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
// bool
|
||||
// LargeSQ8HPass::Run(const TaskPtr& task) {
|
||||
// if (task->Type() != TaskType::SearchTask) {
|
||||
// return false;
|
||||
// }
|
||||
//
|
||||
// auto search_task = std::static_pointer_cast<XSearchTask>(task);
|
||||
// if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8H) {
|
||||
// return false;
|
||||
// }
|
||||
//
|
||||
// auto search_job = std::static_pointer_cast<SearchJob>(search_task->job_.lock());
|
||||
//
|
||||
// // TODO: future, Index::IVFSQ8H, if nq < threshold set cpu, else set gpu
|
||||
// if (search_job->nq() < 100) {
|
||||
// return false;
|
||||
// }
|
||||
//
|
||||
// std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
|
||||
// std::vector<int64_t> all_free_mem;
|
||||
// for (auto& gpu : gpus) {
|
||||
// auto cache = cache::GpuCacheMgr::GetInstance(gpu);
|
||||
// auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
|
||||
// all_free_mem.push_back(free_mem);
|
||||
// }
|
||||
//
|
||||
// auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
|
||||
// auto best_index = std::distance(all_free_mem.begin(), max_e);
|
||||
// auto best_device_id = gpus[best_index];
|
||||
//
|
||||
// ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, best_device_id);
|
||||
// if (not res_ptr) {
|
||||
// SERVER_LOG_ERROR << "GpuResource " << best_device_id << " invalid.";
|
||||
// // TODO: throw critical error and exit
|
||||
// return false;
|
||||
// }
|
||||
//
|
||||
// auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
|
||||
// task->label() = label;
|
||||
//
|
||||
// return true;
|
||||
// }
|
||||
LargeSQ8HPass::LargeSQ8HPass() {
|
||||
server::Config& config = server::Config::GetInstance();
|
||||
Status s = config.GetEngineConfigGpuSearchThreshold(threshold_);
|
||||
if (!s.ok()) {
|
||||
threshold_ = std::numeric_limits<int32_t>::max();
|
||||
}
|
||||
}
|
||||
|
||||
bool
|
||||
LargeSQ8HPass::Run(const TaskPtr& task) {
|
||||
if (task->Type() != TaskType::SearchTask) {
|
||||
return false;
|
||||
}
|
||||
|
||||
auto search_task = std::static_pointer_cast<XSearchTask>(task);
|
||||
if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8H) {
|
||||
return false;
|
||||
}
|
||||
|
||||
auto search_job = std::static_pointer_cast<SearchJob>(search_task->job_.lock());
|
||||
|
||||
// TODO: future, Index::IVFSQ8H, if nq < threshold set cpu, else set gpu
|
||||
|
||||
if (search_job->nq() < threshold_) {
|
||||
return false;
|
||||
}
|
||||
|
||||
std::vector<uint64_t> gpus = scheduler::get_gpu_pool();
|
||||
// std::vector<int64_t> all_free_mem;
|
||||
// for (auto& gpu : gpus) {
|
||||
// auto cache = cache::GpuCacheMgr::GetInstance(gpu);
|
||||
// auto free_mem = cache->CacheCapacity() - cache->CacheUsage();
|
||||
// all_free_mem.push_back(free_mem);
|
||||
// }
|
||||
//
|
||||
// auto max_e = std::max_element(all_free_mem.begin(), all_free_mem.end());
|
||||
// auto best_index = std::distance(all_free_mem.begin(), max_e);
|
||||
// auto best_device_id = gpus[best_index];
|
||||
auto best_device_id = count_ % gpus.size();
|
||||
count_++;
|
||||
|
||||
ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, best_device_id);
|
||||
if (not res_ptr) {
|
||||
SERVER_LOG_ERROR << "GpuResource " << best_device_id << " invalid.";
|
||||
// TODO: throw critical error and exit
|
||||
return false;
|
||||
}
|
||||
|
||||
auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
|
||||
task->label() = label;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
|
||||
@ -18,6 +18,7 @@
|
||||
|
||||
#include <condition_variable>
|
||||
#include <deque>
|
||||
#include <limits>
|
||||
#include <list>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
@ -34,11 +35,15 @@ namespace scheduler {
|
||||
|
||||
class LargeSQ8HPass : public Pass {
|
||||
public:
|
||||
LargeSQ8HPass() = default;
|
||||
LargeSQ8HPass();
|
||||
|
||||
public:
|
||||
// bool
|
||||
// Run(const TaskPtr& task) override;
|
||||
bool
|
||||
Run(const TaskPtr& task) override;
|
||||
|
||||
private:
|
||||
int32_t threshold_ = std::numeric_limits<int32_t>::max();
|
||||
int64_t count_ = 0;
|
||||
};
|
||||
|
||||
using LargeSQ8HPassPtr = std::shared_ptr<LargeSQ8HPass>;
|
||||
|
||||
48
core/src/scheduler/optimizer/OnlyCPUPass.cpp
Normal file
48
core/src/scheduler/optimizer/OnlyCPUPass.cpp
Normal file
@ -0,0 +1,48 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include "scheduler/optimizer/OnlyCPUPass.h"
|
||||
#include "scheduler/SchedInst.h"
|
||||
#include "scheduler/Utils.h"
|
||||
#include "scheduler/task/SearchTask.h"
|
||||
#include "scheduler/tasklabel/SpecResLabel.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
bool
|
||||
OnlyCPUPass::Run(const TaskPtr& task) {
|
||||
if (task->Type() != TaskType::SearchTask)
|
||||
return false;
|
||||
auto search_task = std::static_pointer_cast<XSearchTask>(task);
|
||||
if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8 &&
|
||||
search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFFLAT) {
|
||||
return false;
|
||||
}
|
||||
|
||||
auto gpu_id = get_gpu_pool();
|
||||
if (not gpu_id.empty())
|
||||
return false;
|
||||
|
||||
ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource("cpu");
|
||||
auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
|
||||
task->label() = label;
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
47
core/src/scheduler/optimizer/OnlyCPUPass.h
Normal file
47
core/src/scheduler/optimizer/OnlyCPUPass.h
Normal file
@ -0,0 +1,47 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
#pragma once
|
||||
|
||||
#include <condition_variable>
|
||||
#include <deque>
|
||||
#include <list>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <queue>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#include "Pass.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
class OnlyCPUPass : public Pass {
|
||||
public:
|
||||
OnlyCPUPass() = default;
|
||||
|
||||
public:
|
||||
bool
|
||||
Run(const TaskPtr& task) override;
|
||||
};
|
||||
|
||||
using OnlyCPUPassPtr = std::shared_ptr<OnlyCPUPass>;
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
54
core/src/scheduler/optimizer/OnlyGPUPass.cpp
Normal file
54
core/src/scheduler/optimizer/OnlyGPUPass.cpp
Normal file
@ -0,0 +1,54 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
#include "scheduler/optimizer/OnlyGPUPass.h"
|
||||
#include "scheduler/SchedInst.h"
|
||||
#include "scheduler/Utils.h"
|
||||
#include "scheduler/task/SearchTask.h"
|
||||
#include "scheduler/tasklabel/SpecResLabel.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
OnlyGPUPass::OnlyGPUPass(bool has_cpu) : has_cpu_(has_cpu) {
|
||||
}
|
||||
|
||||
bool
|
||||
OnlyGPUPass::Run(const TaskPtr& task) {
|
||||
if (task->Type() != TaskType::SearchTask || has_cpu_)
|
||||
return false;
|
||||
|
||||
auto search_task = std::static_pointer_cast<XSearchTask>(task);
|
||||
if (search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFSQ8 &&
|
||||
search_task->file_->engine_type_ != (int)engine::EngineType::FAISS_IVFFLAT) {
|
||||
return false;
|
||||
}
|
||||
|
||||
auto gpu_id = get_gpu_pool();
|
||||
if (gpu_id.empty())
|
||||
return false;
|
||||
|
||||
ResourcePtr res_ptr = ResMgrInst::GetInstance()->GetResource(ResourceType::GPU, gpu_id[specified_gpu_id_]);
|
||||
auto label = std::make_shared<SpecResLabel>(std::weak_ptr<Resource>(res_ptr));
|
||||
task->label() = label;
|
||||
|
||||
specified_gpu_id_ = (specified_gpu_id_ + 1) % gpu_id.size();
|
||||
return true;
|
||||
}
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
51
core/src/scheduler/optimizer/OnlyGPUPass.h
Normal file
51
core/src/scheduler/optimizer/OnlyGPUPass.h
Normal file
@ -0,0 +1,51 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
#pragma once
|
||||
|
||||
#include <condition_variable>
|
||||
#include <deque>
|
||||
#include <list>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <queue>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#include "Pass.h"
|
||||
|
||||
namespace milvus {
|
||||
namespace scheduler {
|
||||
|
||||
class OnlyGPUPass : public Pass {
|
||||
public:
|
||||
explicit OnlyGPUPass(bool has_cpu);
|
||||
|
||||
public:
|
||||
bool
|
||||
Run(const TaskPtr& task) override;
|
||||
|
||||
private:
|
||||
uint64_t specified_gpu_id_ = 0;
|
||||
bool has_cpu_ = false;
|
||||
};
|
||||
|
||||
using OnlyGPUPassPtr = std::shared_ptr<OnlyGPUPass>;
|
||||
|
||||
} // namespace scheduler
|
||||
} // namespace milvus
|
||||
@ -24,7 +24,7 @@ namespace scheduler {
|
||||
|
||||
std::ostream&
|
||||
operator<<(std::ostream& out, const CpuResource& resource) {
|
||||
out << resource.Dump();
|
||||
out << resource.Dump().dump();
|
||||
return out;
|
||||
}
|
||||
|
||||
|
||||
@ -28,11 +28,6 @@ class CpuResource : public Resource {
|
||||
public:
|
||||
explicit CpuResource(std::string name, uint64_t device_id, bool enable_loader, bool enable_executor);
|
||||
|
||||
inline std::string
|
||||
Dump() const override {
|
||||
return "<CpuResource, name=" + name_ + ">";
|
||||
}
|
||||
|
||||
friend std::ostream&
|
||||
operator<<(std::ostream& out, const CpuResource& resource);
|
||||
|
||||
|
||||
@ -28,11 +28,6 @@ class DiskResource : public Resource {
|
||||
public:
|
||||
explicit DiskResource(std::string name, uint64_t device_id, bool enable_loader, bool enable_executor);
|
||||
|
||||
inline std::string
|
||||
Dump() const override {
|
||||
return "<DiskResource, name=" + name_ + ">";
|
||||
}
|
||||
|
||||
friend std::ostream&
|
||||
operator<<(std::ostream& out, const DiskResource& resource);
|
||||
|
||||
|
||||
@ -22,7 +22,7 @@ namespace scheduler {
|
||||
|
||||
std::ostream&
|
||||
operator<<(std::ostream& out, const GpuResource& resource) {
|
||||
out << resource.Dump();
|
||||
out << resource.Dump().dump();
|
||||
return out;
|
||||
}
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user