Commit Graph

45 Commits

Author SHA1 Message Date
BossZou
4ab92f3032
Add config to switch off web server (#2137)
* Add a config parameter to switch off http server (fix #2057)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* modify web enable config name

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix cmake bug & output exanption msg

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* rename web enbale config as web_enable

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* optimize code

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* log out errno

Signed-off-by: yhz <413554850@qq.com>
2020-04-28 17:04:04 +08:00
Jin Hai
59035d9892
#1454 Improve code quality (#1948)
* Improve code quality

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Improve code quality

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Improve code quality

Signed-off-by: jinhai <hai.jin@zilliz.com>

* Fix lint

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix compiling error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix compiling error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix compiling error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix issue

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>
2020-04-17 21:05:17 +08:00
Wang XiangYu
3bc17d8cc2
Fix multi client search crash in tracing module fix #1789 fix #1832 (#1899)
* Fix multi client search crash in tracing module fix #1789 fix #1832

Signed-off-by: wxyu <xy.wang@zilliz.com>

* add lock for every context_map_ access

Signed-off-by: wxyu <xy.wang@zilliz.com>

* remove never used variable

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-04-11 12:12:38 +08:00
Cai Yudong
dc707bb8a3
Caiyd 1858 disable s3 build (#1860)
* #1858 disable s3 build

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* disable s3 in unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-04-03 16:35:48 +08:00
Cai Yudong
5950deddfc
#1678 remove CUSTOMIZATION macro (#1707)
* #1678 remove CUSTOMIZATION macro

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* disable faiss_benchmark

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-03-20 10:24:27 +08:00
BossZou
504a9e30ab
Optimize config cpu_cache_capacity / gpu_cache_capacity setter (#1572) (#1629)
* add gpu cache config handler

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* remove cpu/gpu cache mgr from Config class by using cache config handler (fix #1572)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* remove 0.8.0 from config version map

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* clean config header reference

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix bug in web readme

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* reduce gpu config handler to gpu resources handler

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add engine config

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* modify handler hook(fix #1572)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update changlog

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* initalize value in handler by config default

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* code style format

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix compile error in release mode

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* resolve faiss blas threshold init in DBWrapper

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* modify cache header

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* remove comments

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* order headers

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* convert gpu res config to lower case

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* CI retry

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* adjust header order in cpu cache mar file

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* improve config test case

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* code format

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-03-19 10:17:53 +08:00
Jin Hai
e1027e9e65
#1240 Update license declaration of each file (#1241)
* #1240 Update license declaration of each files

Signed-off-by: jinhai <hai.jin@zilliz.com>

* #1240 Update CHANGELOG

Signed-off-by: jinhai <hai.jin@zilliz.com>
2020-02-17 23:40:58 +08:00
BossZou
a0faf1a786 Add http server (#956)
* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t

* add web server

* [skip ci] add license in web module

* modify header include & comment oatpp environment config

* add port configure & create table in handler

* modify web url

* simple url complation done & add swagger

* make sure web url

* web functionality done. debuging

* add web unittest

* web test pass

* add web server port

* add web server port in template

* update unittest cmake file

* change web server default port to 19121

* rename method in web module & unittest pass

* add search case in unittest for web module

* rename some variables

* fix bug

* unittest pass

* web prepare

* fix cmd bug(check server status)

* update changlog

* add web port validate & default set

* clang-format pass

* add web port test in unittest

* add CORS & redirect root to swagger ui

* add web status

* web table method func cascade test pass

* add config url in web module

* modify thirdparty cmake to avoid building oatpp test

* clang format

* update changlog

* add constants in web module

* reserve Config.cpp

* fix constants reference bug

* replace web server with async module

* modify component to support async

* format

* developing controller & add test clent into unittest

* add web port into demo/server_config

* modify thirdparty cmake to allow build test

* remove  unnecessary comment

* add endpoint info in controller

* finish web test(bug here)

* clang format

* add web test cpp to lint exclusions

* check null field in GetConfig

* add macro RETURN STATUS DTo

* fix cmake conflict

* fix crash when exit server

* remove surplus comments & add http param check

* add uri /docs to direct swagger

* format

* change cmd to system

* add default value & unittest in web module

* add macros to judge if GPU supported

* add macros in unit & add default in index dto & print error message when bind http port fail

* format (fix #788)

* fix cors bug (not completed)

* comment cors

* change web framework to simple api

* comments optimize

* change to simple API

* remove comments in controller.hpp

* remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger

* add ep cmake args to sqlite

* clang-format

* change a format

* test pass

* change name to

* fix compiler issue(oatpp-swagger depend on oatpp)

* add & in start_server.h

* specify lib location with oatpp and oatpp-swagger

* add comments

* add swagger definition

* [skip ci] change http method options status code

* remove oatpp swagger(fix #970)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* [skip ci] Fix some broken links (#960)

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken links

* fix issue 373 (#964)

* fix issue 373

* Adjustment format

* Adjustment format

* Adjustment format

* change readme

* #966 update NOTICE.md (#967)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* adjust web port cofig place

* rename web_port variable

* set advanced config name add DEFAULT

Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2020-01-13 13:55:48 +08:00
Cai Yudong
b9d9b8ecc2 support minio storage (#895)
* #815 add S3 client

* #815 update S3 interface

* #815 add bucket_name_ as class field, add StartService() and StopService()

* #815 add FileIOReader and FileIOWriter

* #815 fix unittest test_storage build error

* #815 add GetObjectStr()

* #815 add S3IOReader/S3IOWriter

* #815 add ListObjects(), support put string with \0, launch S3 service when server start

* #815 read/write s3 index file pass

* #815 add MinIO related config

* #815 fix storage config issuegs

* #815 code clean

* #815 use local file or S3 via minio_enable

* #815 add DeleteObjects() for S3

* #815 print out IO rate when read_index & write_index

* #815 clean code

* #815 fix centos build error

* #815 support MinIO storage

* #815 fix typo
2020-01-04 10:39:52 +08:00
Heisenberg-Y
1e72267a65 add cpu mode for built-in Faiss (#841)
* add cpu mode for built-in Faiss

* delete customization options on build stage

* delete faiss version

* delete customization options on build stage

* fix unittest problem in cpu mode

* fix unittest problem in cpu mode

Co-authored-by: quicksilver <zhifeng.zhang@zilliz.com>
Co-authored-by: Jin Hai <hai.jin@zilliz.com>
2019-12-28 17:28:11 +08:00
quicksilver
03ec41070a Remove Jfrog Cache on Jenkins CI (#827)
* specify multiple urls on sqlite_orm download stage

* fix bug

* fix bug

* specify multiple urls on opentracing download stage

* fix bug

* specify multiple urls on download stage

* delete jfrog cache

* print jenkins enviroment variables

* print jenkins enviroment variables

* fix check_ccache.sh bug

* debug

* Update Jenkinfile

* Add build enviroment resource limit on Jenkins CI

* remove Jfrog cache build option
2019-12-26 10:33:05 +08:00
quicksilver
685d199ae9 Specify multiple urls on download stage (#820)
* specify multiple urls on sqlite_orm download stage

* fix bug

* fix bug

* specify multiple urls on opentracing download stage

* fix bug

* specify multiple urls on download stage
2019-12-25 09:56:47 +08:00
quicksilver
e1d20e4446
Merge pull request #790 from ZhifengZhang-CN/master
Add Codecov on Github Actions
2019-12-19 12:07:46 +08:00
XuPeng-SH
d316f18cbc Integrate libfiu for fault injection (#784)
* (feat): add libfiu for fault injection

* (feat): change libfiu from static to shared

* (feat): install libfiu
2019-12-18 18:11:18 +08:00
quicksilver
bdc0239eba update GRPC url 2019-12-17 19:15:17 +08:00
quicksilver
c32f58d474 update GRPC url 2019-12-17 16:45:48 +08:00
Zhiru Zhu
e99896ac62 Add Opentracing (#729)
* add TracerUtil

* Interceptor ok

* add handler

* add context

* minor update

* keep span in trace context

* add span in search okay

* Update Context.cpp

* refactor

* refactor

* refactor

* format

* add context in SearchJob

* trace search okay

* add back finish span in interceptor

* add namespace

* add tracing config in server config

* add random id

* debug mode okay

* update CMakeLists

* add opentracing to cmake

* update unittest

* add tracing namespace

* remove std::run_time error

* add lock when erasing context_map

* update tracing config

* lint

* update CHANGELOG

* small fix

* fix server unit test

* fix scheduler unit test

* fix db unit test

* lint

* fix db unit test gpu version

* rename to tracing_config

* fix

* update

* trigger ci
2019-12-10 15:24:54 +08:00
JinHai-CN
88f7dc56a2 Integrate internal faiss
Remove faiss docs
2019-11-28 11:42:54 +08:00
quicksilver
8164bb031c disable untar verbose 2019-11-22 18:46:06 +08:00
ZhifengZhang-CN
9c954b562a delete FindGTest.cmake 2019-11-12 17:30:19 +08:00
ZhifengZhang-CN
d04f7b56a2 update FindGTest.cmake 2019-11-12 17:11:32 +08:00
quicksilver
f8e66f12a9 add FindGTest.cmake 2019-11-12 11:12:03 +08:00
quicksilver
d4372d28b7 use ccache when compiling (if available) 2019-11-11 11:12:59 +08:00
Zhiru Zhu
7fa712e45c
Update DefineOptions.cmake 2019-11-07 16:11:55 +08:00
Zhiru Zhu
3c0c4646b0 issue 229, 230 and 237 2019-11-07 15:35:53 +08:00
youny626
29e8ef110f change default cpu version option to off 2019-11-05 13:21:32 +08:00
youny626
4ca6b59b91 add CPU version 2019-11-05 10:23:59 +08:00
Zhiru Zhu
530b2a0f28 fix cmake error: STRING sub-command REGEX, mode REPLACE needs at least 6 arguments total to
command


Former-commit-id: 773f938125ed03582fc8242af0a23076677bf848
2019-10-31 14:57:09 +08:00
Zhiru Zhu
c99f407099 delete message
Former-commit-id: 577b8693eca42bf3537c8691f754dd1876d579a3
2019-10-30 15:30:28 +08:00
Zhiru Zhu
c41ebb7e7d fix JFrog cache error
Former-commit-id: 8b75351080e9201d6f9f47c643dfb7eec32d5a86
2019-10-30 15:27:05 +08:00
Zhiru Zhu
a6279d2acb Speed up CMake build process
Former-commit-id: 2edb2ae26d93ed9b7a8d25b89d30152e60257250
2019-10-23 21:45:48 +08:00
JinHai-CN
839009ff26 Use system boost anymore
Former-commit-id: 159bcdb3599fb72ddbd9b35b388ced78150deb03
2019-10-15 20:51:30 +08:00
JinHai-CN
33f01a0653 Update CI files
Former-commit-id: 973fecf6705d97ca8ce5e6bc1ede5267fdbc59cc
2019-10-15 18:12:22 +08:00
quicksilver
0beb4329fb update ExternalProject_Create_Cache cmake function
Former-commit-id: 9a073bf36e524cb1a8c64097bff35c09b021596c
2019-10-15 10:54:10 +08:00
Jeff
83fb150370 Merge branch 'branch-0.5.0' into 'branch-0.5.0'
switch back sqlite orm url

See merge request megasearch/milvus!720

Former-commit-id: 148ac2d9aa77bd67b1d7aec1b1ca43e2afa6eb3f
2019-10-14 20:21:43 +08:00
zhiru
63fbc43fbc switch back sqlite orm url
Former-commit-id: 1de6661e2ede4ef8336d812df61decca50f40d93
2019-10-14 20:20:52 +08:00
Jeff
8b952fcd7f Merge branch 'branch-0.5.0' into 'branch-0.5.0'
update grpc download md5

See merge request megasearch/milvus!719

Former-commit-id: ac6924caee8df7c83b4466b2c6f5bd12c4f9f981
2019-10-14 20:18:02 +08:00
zhiru
b904029529 update grpc download md5
Former-commit-id: c6840e3bc4e8baaa09c457e3dd80c931f88c112c
2019-10-14 20:17:20 +08:00
quicksilver
b5d1f57ff2 check thirdparty cache directory status
Former-commit-id: 573e28369c6466f9de23c6e92dac903bde23429e
2019-10-14 20:00:08 +08:00
zhiru
44eda65b41 update grpc download url to https://github.com/youny626/grpc-milvus/archive/master.zip
Former-commit-id: ae1134f8ab9b2e7c7f1c4c1464993c35d1491fd5
2019-10-14 17:35:21 +08:00
zhiru
0d6966b41b update grpc download url to https://github.com/youny626/grpc-milvus/archive/master.zip
Former-commit-id: 06dfa0e0876376498d48d09d5b7c4d2ee634fbbc
2019-10-14 17:19:10 +08:00
peng.xu
6b12678f06 Merge branch 'branch-0.5.0' into 'branch-0.5.0'
re-organize project directory

See merge request megasearch/milvus!706

Former-commit-id: 18490c4712a13e55cffd178d00821a70741fb9b7
2019-10-14 12:06:38 +08:00
starlord
2fc179104d Merge remote-tracking branch 'source/branch-0.5.0' into branch-0.5.0
Former-commit-id: 7e210e1c775b9acef194ac6f53cf0dc1a671b04d
2019-10-14 11:45:40 +08:00
starlord
004bd81ac8 Merge remote-tracking branch 'source/branch-0.5.0' into branch-0.5.0
Former-commit-id: 4496b80869e6be02b8fbbe92a3ab08f8bf763acf
2019-10-14 09:51:56 +08:00
starlord
41d37f9648 re-organize project
Former-commit-id: 898c2c8b0b2cfdc1b8664062aa0ea910a93566dd
2019-10-14 09:51:48 +08:00