2021-06-21 12:08:06 +08:00
# Development
2021-09-28 09:52:04 +08:00
This document will help to setup your development environment and run tests. If you encounter a problem, please file an issue.
2021-06-21 12:08:06 +08:00
2021-07-07 15:12:00 +08:00
Table of contents
=================
2021-09-22 17:47:53 +08:00
- [Development ](#development )
- [Table of contents ](#table-of-contents )
- [Building Milvus with Docker ](#building-milvus-with-docker )
- [Building Milvus on a local OS/shell environment ](#building-milvus-on-a-local-osshell-environment )
- [Hardware Requirements ](#hardware-requirements )
2021-09-28 09:52:04 +08:00
- [Software Requirements ](#software-requirements )
2021-09-22 17:47:53 +08:00
- [Dependencies ](#dependencies )
- [CMake ](#cmake )
- [Go ](#go )
- [Docker & Docker Compose ](#docker--docker-compose )
- [Building Milvus ](#building-milvus )
- [A Quick Start for Testing Milvus ](#a-quick-start-for-testing-milvus )
- [Presubmission Verification ](#presubmission-verification )
- [Unit Tests ](#unit-tests )
- [Code coverage ](#code-coverage )
- [E2E Tests ](#e2e-tests )
2021-10-01 12:30:45 +08:00
- [Test on local branch ](#test-on-local-branch )
- [On Linux ](#on-linux )
- [With docker ](#with-docker )
2021-09-22 17:47:53 +08:00
- [GitHub Flow ](#github-flow )
2021-07-07 15:12:00 +08:00
## Building Milvus with Docker
Official releases are built using Docker containers. To build Milvus using Docker please follow [these instructions ](https://github.com/milvus-io/milvus/blob/master/build/README.md ).
2021-06-21 12:08:06 +08:00
## Building Milvus on a local OS/shell environment
The details below outline the hardware and software requirements for building on Linux.
### Hardware Requirements
Milvus is written in Go and C++, compiling it can use a lot of resources. We recommend the following for any physical or virtual machine being used for building Milvus.
```
- 8GB of RAM
- 50GB of free disk space
```
2021-09-28 09:52:04 +08:00
### Software Requirements
2021-06-21 12:08:06 +08:00
2021-09-28 09:52:04 +08:00
In fact, all Linux distributions are available to develop Milvus. The following only contains commands on Ubuntu and CentOS, because we mainly use them. If you develop Milvus on other distributions, you are welcome to improve this document.
2021-06-21 12:08:06 +08:00
2021-06-28 16:30:17 +08:00
#### Dependencies
- Debian/Ubuntu
2021-06-21 12:08:06 +08:00
```shell
2021-09-17 10:25:50 +08:00
$ sudo apt update
$ sudo apt install -y build-essential ccache gfortran \
libssl-dev zlib1g-dev python3-dev libcurl4-openssl-dev libtbb-dev\
libboost-regex-dev libboost-program-options-dev libboost-system-dev \
libboost-filesystem-dev libboost-serialization-dev libboost-python-dev
2021-06-21 12:08:06 +08:00
```
2021-06-28 16:30:17 +08:00
- CentOS
```shell
2021-09-17 10:25:50 +08:00
$ sudo yum install -y epel-release centos-release-scl-rh & & \
$ sudo yum install -y git make automake openssl-devel zlib-devel \
libcurl-devel python3-devel \
devtoolset-7-gcc devtoolset-7-gcc-c++ devtoolset-7-gcc-gfortran \
llvm-toolset-7.0-clang llvm-toolset-7.0-clang-tools-extra \
ccache lcov
$ echo "source scl_source enable devtoolset-7" | sudo tee -a /etc/profile.d/devtoolset-7.sh
$ echo "source scl_source enable llvm-toolset-7.0" | sudo tee -a /etc/profile.d/llvm-toolset-7.sh
$ echo "export CLANG_TOOLS_PATH=/opt/rh/llvm-toolset-7.0/root/usr/bin" | sudo tee -a /etc/profile.d/llvm-toolset-7.sh
$ source "/etc/profile.d/llvm-toolset-7.sh"
2021-07-09 16:24:39 +08:00
# Install tbb
2021-09-17 10:25:50 +08:00
$ git clone https://github.com/wjakob/tbb.git & & \
cd tbb/build & & \
cmake .. & & make -j & & \
sudo make install & & \
cd ../../ & & rm -rf tbb/
2021-07-09 16:24:39 +08:00
# Install boost
2021-09-17 10:25:50 +08:00
$ wget -q https://boostorg.jfrog.io/artifactory/main/release/1.65.1/source/boost_1_65_1.tar.gz & & \
tar zxf boost_1_65_1.tar.gz & & cd boost_1_65_1 & & \
./bootstrap.sh --prefix=/usr/local --with-toolset=gcc --without-libraries=python & & \
sudo ./b2 -j2 --prefix=/usr/local --without-python toolset=gcc install & & \
cd ../ & & rm -rf ./boost_1_65_1*
2021-07-09 16:24:39 +08:00
2021-06-28 16:30:17 +08:00
```
2021-06-21 12:08:06 +08:00
Once you have finished, confirm that `gcc` and `make` are installed:
```shell
2021-09-17 10:25:50 +08:00
$ gcc --version
$ make --version
2021-06-21 12:08:06 +08:00
```
#### CMake
The algorithm library of Milvus, Knowhere is written in c++. CMake is required in the Milvus compilation. If you don't have it, please follow the instructions in the [Installing CMake ](https://cmake.org/install/ ).
Confirm that cmake is available:
```shell
2021-09-17 10:25:50 +08:00
$ cmake --version
2021-06-21 12:08:06 +08:00
```
#### Go
Milvus is written in [Go ](http://golang.org/ ). If you don't have a Go development environment, please follow the instructions in the [Go Getting Started guide ](https://golang.org/doc/install ).
Confirm that your `GOPATH` and `GOBIN` environment variables are correctly set as detailed in [How to Write Go Code ](https://golang.org/doc/code.html ) before proceeding.
```shell
2021-09-17 10:25:50 +08:00
$ go version
2021-06-21 12:08:06 +08:00
```
#### Docker & Docker Compose
Milvus depends on Etcd, Pulsar and minIO. Using Docker Compose to manage these is an easy way in a local development. To install Docker and Docker Compose in your development environment, follow the instructions from the Docker website below:
- Docker: https://docs.docker.com/get-docker/
- Docker Compose: https://docs.docker.com/compose/install/
### Building Milvus
To build the Milvus project, run the following command:
```shell
2021-09-17 10:25:50 +08:00
$ make all
2021-06-21 12:08:06 +08:00
```
If this command succeed, you will now have an executable at `bin/milvus` off of your Milvus project directory.
## A Quick Start for Testing Milvus
### Presubmission Verification
Presubmission verification provides a battery of checks and tests to give your pull request the best chance of being accepted. Developers need to run as many verification tests as possible locally.
To run all presubmission verification tests, use this command:
```shell
2021-09-17 10:25:50 +08:00
$ make verifiers
2021-06-21 12:08:06 +08:00
```
### Unit Tests
Pull requests need to pass all unit tests. To run every unit test, use this command:
```shell
2021-09-17 10:25:50 +08:00
$ make unittest
2021-06-21 12:08:06 +08:00
```
2021-09-13 09:44:02 +08:00
To run single test case, for instance, run TestSearchTask in /internal/proxy directory, use
```shell
2021-09-17 10:25:50 +08:00
$ go test -v ./internal/proxy/ -test.run TestSearchTask
2021-09-13 09:44:02 +08:00
```
2021-09-22 17:47:53 +08:00
### Code coverage
Before submitting your Pull Request, make sure your code change is covered by unit test. Use the following commands to check code coverage rate:
Install lcov(cpp code coverage tool)
```shell
$ sudo apt-get install lcov
```
Run unit test and generate report for code coverage
```shell
$ make codecov
```
This command will generate html report for Golang and C++ respectively.
For Golang report, open the `go_coverage.html` under milvus project path.
For C++ report, open the `cpp_coverage/index.html` under milvus project path.
You also can generate Golang coverage report by:
```shell
$ make codecov-go
```
Or C++ coverage report by:
```shell
$ make codecov-cpp
```
2021-09-13 09:44:02 +08:00
2021-06-21 12:08:06 +08:00
### E2E Tests
Milvus uses Python SDK to write test cases to verify the correctness of Milvus functions. Before run E2E tests, you need a running Milvus:
```shell
2021-09-17 10:25:50 +08:00
$ cd deployments/docker/dev
$ docker-compose up -d
$ cd ../../../
# Running Milvus standalone
$ ./scripts/start_standalone.sh
2021-09-28 09:52:04 +08:00
# or running a Milvus cluster
2021-09-17 10:25:50 +08:00
$ ./scripts/start_cluster.sh
2021-06-21 12:08:06 +08:00
```
To run E2E tests, use these command:
```shell
2021-09-17 10:25:50 +08:00
$ cd tests/python_client
$ pip install -r requirements.txt
$ pytest --tags=L0 -n auto
2021-06-21 12:08:06 +08:00
```
2021-10-01 08:55:21 +08:00
### Test on local branch
2021-10-01 12:30:45 +08:00
#### On Linux
2021-10-01 08:55:21 +08:00
start the cluster on your host machine
```shell
$ ./scripts/start_cluster.sh
```
#### With docker
start the cluster on your host machine
```shell
$ ./build/builder.sh make install // build milvus
$ ./build/build_image.sh // build milvus lastest docker
$ docker images // check if milvus latest image is ready
REPOSITORY TAG IMAGE ID CREATED SIZE
milvusdb/milvus latest 63c62ff7c1b7 52 minutes ago 570MB
$ install with docker compose
```
2021-06-21 12:08:06 +08:00
## GitHub Flow
2021-09-28 09:52:04 +08:00
To check out code to work on, please refer to the [GitHub Flow ](https://guides.github.com/introduction/flow/ ).