test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-11-30 10:59:32 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity

Fix http server bug (#1096)

Browse Source 
* refactoring(create_table done)

* refactoring

* refactor server delivery (insert done)

* refactoring server module (count_table done)

* server refactor done

* cmake pass

* refactor server module done.

* set grpc response status correctly

* format done.

* fix redefine ErrorMap()

* optimize insert reducing ids data copy

* optimize grpc request with reducing data copy

* clang format

* [skip ci] Refactor server module done. update changlog. prepare for PR

* remove explicit and change int32_t to int64_t

* add web server

* [skip ci] add license in web module

* modify header include & comment oatpp environment config

* add port configure & create table in handler

* modify web url

* simple url complation done & add swagger

* make sure web url

* web functionality done. debuging

* add web unittest

* web test pass

* add web server port

* add web server port in template

* update unittest cmake file

* change web server default port to 19121

* rename method in web module & unittest pass

* add search case in unittest for web module

* rename some variables

* fix bug

* unittest pass

* web prepare

* fix cmd bug(check server status)

* update changlog

* add web port validate & default set

* clang-format pass

* add web port test in unittest

* add CORS & redirect root to swagger ui

* add web status

* web table method func cascade test pass

* add config url in web module

* modify thirdparty cmake to avoid building oatpp test

* clang format

* update changlog

* add constants in web module

* reserve Config.cpp

* fix constants reference bug

* replace web server with async module

* modify component to support async

* format

* developing controller & add test clent into unittest

* add web port into demo/server_config

* modify thirdparty cmake to allow build test

* remove  unnecessary comment

* add endpoint info in controller

* finish web test(bug here)

* clang format

* add web test cpp to lint exclusions

* check null field in GetConfig

* add macro RETURN STATUS DTo

* fix cmake conflict

* fix crash when exit server

* remove surplus comments & add http param check

* add uri /docs to direct swagger

* format

* change cmd to system

* add default value & unittest in web module

* add macros to judge if GPU supported

* add macros in unit & add default in index dto & print error message when bind http port fail

* format (fix #788)

* fix cors bug (not completed)

* comment cors

* change web framework to simple api

* comments optimize

* change to simple API

* remove comments in controller.hpp

* remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger

* add ep cmake args to sqlite

* clang-format

* change a format

* test pass

* change name to

* fix compiler issue(oatpp-swagger depend on oatpp)

* add & in start_server.h

* specify lib location with oatpp and oatpp-swagger

* add comments

* add swagger definition

* [skip ci] change http method options status code

* remove oatpp swagger(fix #970)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* [skip ci] Fix some broken links (#960)

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken link

* [skip ci] Fix broken links

* fix issue 373 (#964)

* fix issue 373

* Adjustment format

* Adjustment format

* Adjustment format

* change readme

* #966 update NOTICE.md (#967)

* remove comments

* check Start web behavior

* add default to cpu_cache_capacity

* remove swagger component.hpp & /docs url

* remove /docs info

* remove /docs in unittest

* remove space in test rpc

* remove repeate info in CHANGLOG

* change cache_insert_data default value as a constant

* adjust web port cofig place

* rename web_port variable

* change gpu resources invoke way to cmd()

* set advanced config name add DEFAULT

* change config setting to cmd

* modify ..

* optimize code

* assign TableDto' count default value 0 (fix #995)

* check if table exists when show partitions (fix #1028)

* check table exists when drop partition (fix #1029)

* check if partition name is legal (fix #1022)

* modify status code when partition tag is illegal

* update changlog

* add info to /system url

* add binary index and add bin uri & handler method(not completed)

* optimize http insert and search time(fix #1066) | add binary vectors support(fix #1067)

* fix test partition bug

* fix test bug when check insert records

* add binary vectors test

* add default for offset and page_size

* fix uinttest bug

* [skip ci] remove comments

* optimize web code for PR comments

* add new folder named utils

* check offset and pagesize (fix #1082)

* improve error message if offset or page_size is not legal (fix #1075)

* add log into web module

* update changlog

* check gpu sources setting when assign repeated value (fix #990)

* update changlog

* clang-format pass

* add default handler in http handler

* [skip ci] improve error msg when check gpu resources

* change check offset way

* remove func IsIntStr

* add case

* change int32 to int64 when check number str

* add log in we module(doing)

* update test case

* add log in web controller

Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com>
Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
This commit is contained in:
BossZou 2020-01-19 19:19:09 +08:00 committed by Jin Hai
parent 658f100a10
commit 61517339ea
10 changed files with 291 additions and 103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -11,6 +11,7 @@ Please mark all change in change log and use the issue from GitHub
- \#805 - IVFTest.gpu_seal_test unittest failed
- \#831 - Judge branch error in CommonUtil.cpp
- \#977 - Server crash when create tables concurrently
- \#990 - check gpu resources setting when assign repeated value
- \#995 - table count set to 0 if no tables found
- \#1010 - improve error message when offset or page_size is equal 0
- \#1022 - check if partition name is legal
@ -18,6 +19,8 @@ Please mark all change in change log and use the issue from GitHub
- \#1029 - check if table exists when try to delete partition
- \#1066 - optimize http insert and search speed
- \#1067 - Add binary vectors support in http server
- \#1075 - improve error message when page size or offset is illegal
- \#1082 - check page_size or offset value to avoid float
## Feature
- \#216 - Add CLI to get server info

20
core/src/server/Config.cpp
View File

@ -22,6 +22,7 @@
#include <regex>
#include <string>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "config/YamlConfigMgr.h"
@ -791,9 +792,19 @@ Config::CheckGpuResourceConfigSearchResources(const std::vector<std::string>& va
return Status(SERVER_INVALID_ARGUMENT, msg);
}
std::unordered_set<std::string> value_set;
for (auto& resource : value) {
CONFIG_CHECK(CheckGpuResource(resource));
value_set.insert(resource);
}
if (value_set.size() != value.size()) {
std::string msg =
"Invalid gpu build search resource. "
"Possible reason: gpu_resource_config.gpu_search_resources contains duplicate resources.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();
}
@ -806,8 +817,17 @@ Config::CheckGpuResourceConfigBuildIndexResources(const std::vector<std::string>
return Status(SERVER_INVALID_ARGUMENT, msg);
}
std::unordered_set<std::string> value_set;
for (auto& resource : value) {
CONFIG_CHECK(CheckGpuResource(resource));
value_set.insert(resource);
}
if (value_set.size() != value.size()) {
std::string msg =
"Invalid gpu build index resource. "
"Possible reason: gpu_resource_config.build_index_resources contains duplicate resources.";
return Status(SERVER_INVALID_ARGUMENT, msg);
}
return Status::OK();

3
core/src/server/web_impl/Constants.h
View File

@ -21,6 +21,9 @@ namespace milvus {
namespace server {
namespace web {
////////////////////////////////////////////////////
static const char* WEB_LOG_PREFIX = "[Web] ";
////////////////////////////////////////////////////
static const char* CORS_KEY_METHODS = "Access-Control-Allow-Methods";

179
core/src/server/web_impl/controller/WebController.hpp
View File

@ -25,16 +25,16 @@
#include <oatpp/core/macro/codegen.hpp>
#include <oatpp/core/macro/component.hpp>
#include "server/web_impl/dto/ConfigDto.hpp"
#include "server/web_impl/dto/TableDto.hpp"
#include "utils/Log.h"
#include "utils/TimeRecorder.h"
#include "server/web_impl/Constants.h"
#include "server/web_impl/dto/CmdDto.hpp"
#include "server/web_impl/dto/ConfigDto.hpp"
#include "server/web_impl/dto/IndexDto.hpp"
#include "server/web_impl/dto/PartitionDto.hpp"
#include "server/web_impl/dto/TableDto.hpp"
#include "server/web_impl/dto/VectorDto.hpp"
#include "server/web_impl/dto/ConfigDto.hpp"
#include "utils/Log.h"
#include "server/delivery/RequestHandler.h"
#include "server/web_impl/handler/WebRequestHandler.h"
namespace milvus {
@ -47,9 +47,8 @@ class WebController : public oatpp::web::server::api::ApiController {
: oatpp::web::server::api::ApiController(objectMapper) {}
public:
static std::shared_ptr<WebController> createShared(OATPP_COMPONENT(std::shared_ptr<ObjectMapper>,
objectMapper)) {
static std::shared_ptr<WebController> createShared(
OATPP_COMPONENT(std::shared_ptr<ObjectMapper>, objectMapper)) {
return std::make_shared<WebController>(objectMapper);
}
@ -80,6 +79,8 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(State)
ENDPOINT("GET", "/state", State) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/state\'");
tr.ElapseFromBegin("Total cost ");
return createDtoResponse(Status::CODE_200, StatusDto::createShared());
}
@ -93,9 +94,11 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(GetDevices)
ENDPOINT("GET", "/devices", GetDevices) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/devices\'");
tr.RecordSection("Receive request");
auto devices_dto = DevicesDto::createShared();
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
auto status_dto = handler.GetDevices(devices_dto);
std::shared_ptr<OutgoingResponse> response;
switch (status_dto->code->getValue()) {
@ -106,6 +109,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -125,9 +131,11 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(GetAdvancedConfig)
ENDPOINT("GET", "/config/advanced", GetAdvancedConfig) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/config/advanced\'");
tr.RecordSection("Received request.");
auto config_dto = AdvancedConfigDto::createShared();
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
auto status_dto = handler.GetAdvancedConfig(config_dto);
std::shared_ptr<OutgoingResponse> response;
@ -139,6 +147,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -154,8 +165,10 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(SetAdvancedConfig)
ENDPOINT("PUT", "/config/advanced", SetAdvancedConfig, BODY_DTO(AdvancedConfigDto::ObjectWrapper, body)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "PUT \'/config/advanced\'");
tr.RecordSection("Received request.");
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.SetAdvancedConfig(body);
@ -167,6 +180,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -188,9 +204,11 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(GetGPUConfig)
ENDPOINT("GET", "/config/gpu_resources", GetGPUConfig) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/config/gpu_resources\'");
tr.RecordSection("Received request");
auto gpu_config_dto = GPUConfigDto::createShared();
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.GetGpuConfig(gpu_config_dto);
@ -202,6 +220,10 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -216,8 +238,10 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(SetGPUConfig)
ENDPOINT("PUT", "/config/gpu_resources", SetGPUConfig, BODY_DTO(GPUConfigDto::ObjectWrapper, body)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "PUT \'/config/gpu_resources\'");
tr.RecordSection("Received request.");
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
auto status_dto = handler.SetGpuConfig(body);
std::shared_ptr<OutgoingResponse> response;
@ -229,6 +253,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -252,8 +279,10 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(CreateTable)
ENDPOINT("POST", "/tables", CreateTable, BODY_DTO(TableRequestDto::ObjectWrapper, body)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "POST \'/tables\'");
tr.RecordSection("Received request.");
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.CreateTable(body);
@ -265,6 +294,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -280,13 +312,15 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(ShowTables)
ENDPOINT("GET", "/tables", ShowTables, REQUEST(
const std::shared_ptr<IncomingRequest>&, request)) {
ENDPOINT("GET", "/tables", ShowTables, QUERIES(const QueryParams&, query_params)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/tables\'");
tr.RecordSection("Received request.");
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
auto response_dto = TableListFieldsDto::createShared();
auto offset = request->getQueryParameter("offset", "0");
auto page_size = request->getQueryParameter("page_size", "10");
auto offset = query_params.get("offset");
auto page_size = query_params.get("page_size");
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.ShowTables(offset, page_size, response_dto);
@ -298,6 +332,10 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -319,12 +357,13 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(GetTable)
ENDPOINT("GET", "/tables/{table_name}", GetTable, PATH(String, table_name), QUERIES(
const QueryParams&, query_params)) {
auto error_status_dto = StatusDto::createShared();
ENDPOINT("GET", "/tables/{table_name}", GetTable,
PATH(String, table_name), QUERIES(const QueryParams&, query_params)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/tables/" + table_name->std_str() + "\'");
tr.RecordSection("Received request.");
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
auto fields_dto = TableFieldsDto::createShared();
auto status_dto = handler.GetTable(table_name, query_params, fields_dto);
@ -340,6 +379,10 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -354,8 +397,10 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(DropTable)
ENDPOINT("DELETE", "/tables/{table_name}", DropTable, PATH(String, table_name)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "DELETE \'/tables/" + table_name->std_str() + "\'");
tr.RecordSection("Received request.");
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.DropTable(table_name);
@ -370,6 +415,10 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -393,8 +442,10 @@ class WebController : public oatpp::web::server::api::ApiController {
ENDPOINT("POST", "/tables/{table_name}/indexes", CreateIndex,
PATH(String, table_name), BODY_DTO(IndexRequestDto::ObjectWrapper, body)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "POST \'/tables/" + table_name->std_str() + "/indexes\'");
tr.RecordSection("Received request.");
auto handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.CreateIndex(table_name, body);
@ -409,6 +460,10 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -425,12 +480,15 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(GetIndex)
ENDPOINT("GET", "/tables/{table_name}/indexes", GetIndex, PATH(String, table_name)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/tables/" + table_name->std_str() + "/indexes\'");
tr.RecordSection("Received request.");
auto index_dto = IndexDto::createShared();
auto handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
auto status_dto = handler.GetIndex(table_name, index_dto);
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.GetIndex(table_name, index_dto);
switch (status_dto->code->getValue()) {
case StatusCode::SUCCESS:
response = createDtoResponse(Status::CODE_200, index_dto);
@ -442,6 +500,10 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -458,8 +520,10 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(DropIndex)
ENDPOINT("DELETE", "/tables/{table_name}/indexes", DropIndex, PATH(String, table_name)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "DELETE \'/tables/" + table_name->std_str() + "/indexes\'");
tr.RecordSection("Received request.");
auto handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.DropIndex(table_name);
@ -474,6 +538,10 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
std::string ttr = "Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost";
tr.ElapseFromBegin(ttr);
return response;
}
@ -499,8 +567,10 @@ class WebController : public oatpp::web::server::api::ApiController {
ENDPOINT("POST", "/tables/{table_name}/partitions",
CreatePartition, PATH(String, table_name), BODY_DTO(PartitionRequestDto::ObjectWrapper, body)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "POST \'/tables/" + table_name->std_str() + "/partitions\'");
tr.RecordSection("Received request.");
auto handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.CreatePartition(table_name, body);
@ -515,6 +585,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -537,14 +610,15 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(ShowPartitions)
ENDPOINT("GET", "/tables/{table_name}/partitions", ShowPartitions,
PATH(String, table_name), REQUEST(
const std::shared_ptr<IncomingRequest>&, request)) {
auto offset = request->getQueryParameter("offset", "0");
auto page_size = request->getQueryParameter("page_size", "10");
PATH(String, table_name), QUERIES(const QueryParams&, query_params)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/tables/" + table_name->std_str() + "/partitions\'");
tr.RecordSection("Received request.");
auto offset = query_params.get("offset");
auto page_size = query_params.get("page_size");
auto partition_list_dto = PartitionListDto::createShared();
auto handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.ShowPartitions(offset, page_size, table_name, partition_list_dto);
@ -558,6 +632,9 @@ class WebController : public oatpp::web::server::api::ApiController {
default:response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -582,8 +659,11 @@ class WebController : public oatpp::web::server::api::ApiController {
ENDPOINT("DELETE", "/tables/{table_name}/partitions/{partition_tag}", DropPartition,
PATH(String, table_name), PATH(String, partition_tag)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) +
"DELETE \'/tables/" + table_name->std_str() + "/partitions/" + partition_tag->std_str() + "\'");
tr.RecordSection("Received request.");
auto handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.DropPartition(table_name, partition_tag);
@ -598,6 +678,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -623,9 +706,11 @@ class WebController : public oatpp::web::server::api::ApiController {
ENDPOINT("POST", "/tables/{table_name}/vectors", Insert,
PATH(String, table_name), BODY_DTO(InsertRequestDto::ObjectWrapper, body)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "POST \'/tables/" + table_name->std_str() + "/vectors\'");
tr.RecordSection("Received request.");
auto ids_dto = VectorIdsDto::createShared();
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.Insert(table_name, body, ids_dto);
@ -640,6 +725,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -659,9 +747,11 @@ class WebController : public oatpp::web::server::api::ApiController {
ENDPOINT("PUT", "/tables/{table_name}/vectors", Search,
PATH(String, table_name), BODY_DTO(SearchRequestDto::ObjectWrapper, body)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "PUT \'/tables/" + table_name->std_str() + "/vectors\'");
tr.RecordSection("Received request.");
auto results_dto = TopkResultsDto::createShared();
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.Search(table_name, body, results_dto);
@ -676,6 +766,9 @@ class WebController : public oatpp::web::server::api::ApiController {
response = createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}
@ -691,13 +784,14 @@ class WebController : public oatpp::web::server::api::ApiController {
ADD_CORS(SystemMsg)
ENDPOINT("GET", "/system/{msg}", SystemMsg, PATH(String, msg)) {
TimeRecorder tr(std::string(WEB_LOG_PREFIX) + "GET \'/system/" + msg->std_str() + "\'");
tr.RecordSection("Received request.");
auto cmd_dto = CommandDto::createShared();
WebRequestHandler handler = WebRequestHandler();
handler.RegisterRequestHandler(::milvus::server::RequestHandler());
std::shared_ptr<OutgoingResponse> response;
auto status_dto = handler.Cmd(msg, cmd_dto);
std::shared_ptr<OutgoingResponse> response;
switch (status_dto->code->getValue()) {
case StatusCode::SUCCESS:
response = createDtoResponse(Status::CODE_200, cmd_dto);
@ -706,6 +800,9 @@ class WebController : public oatpp::web::server::api::ApiController {
return createDtoResponse(Status::CODE_400, status_dto);
}
tr.ElapseFromBegin("Done. Status: code = " + std::to_string(status_dto->code->getValue())
+ ", reason = " + status_dto->message->std_str() + ". Total cost");
return response;
}

38
core/src/server/web_impl/handler/WebRequestHandler.cpp
View File

@ -31,6 +31,7 @@
#include "server/web_impl/utils/Util.h"
#include "utils/StringHelpFunctions.h"
#include "utils/TimeRecorder.h"
#include "utils/ValidationUtil.h"
namespace milvus {
namespace server {
@ -439,20 +440,21 @@ WebRequestHandler::ShowTables(const OString& offset, const OString& page_size,
int64_t page_size_value = 10;
if (nullptr != offset.get()) {
try {
offset_value = std::stol(offset->std_str());
} catch (const std::exception& e) {
RETURN_STATUS_DTO(ILLEGAL_QUERY_PARAM, "Query param \'offset\' is illegal, only type of \'int\' allowed");
std::string offset_str = offset->std_str();
if (!ValidationUtil::ValidateStringIsNumber(offset_str).ok()) {
RETURN_STATUS_DTO(ILLEGAL_QUERY_PARAM,
"Query param \'offset\' is illegal, only non-negative integer supported");
}
offset_value = std::stol(offset_str);
}
if (nullptr != page_size.get()) {
try {
page_size_value = std::stol(page_size->std_str());
} catch (const std::exception& e) {
std::string page_size_str = page_size->std_str();
if (!ValidationUtil::ValidateStringIsNumber(page_size_str).ok()) {
RETURN_STATUS_DTO(ILLEGAL_QUERY_PARAM,
"Query param \'page_size\' is illegal, only type of \'int\' allowed");
"Query param \'page_size\' is illegal, only non-negative integer supported");
}
page_size_value = std::stol(page_size_str);
}
if (offset_value < 0 || page_size_value < 0) {
@ -557,21 +559,21 @@ WebRequestHandler::ShowPartitions(const OString& offset, const OString& page_siz
int64_t page_size_value = 10;
if (nullptr != offset.get()) {
try {
offset_value = std::stol(offset->std_str());
} catch (const std::exception& e) {
std::string msg = "Query param \'offset\' is illegal. Reason: " + std::string(e.what());
RETURN_STATUS_DTO(ILLEGAL_QUERY_PARAM, msg.c_str());
std::string offset_str = offset->std_str();
if (!ValidationUtil::ValidateStringIsNumber(offset_str).ok()) {
RETURN_STATUS_DTO(ILLEGAL_QUERY_PARAM,
"Query param \'offset\' is illegal, only non-negative integer supported");
}
offset_value = std::stol(offset_str);
}
if (nullptr != page_size.get()) {
try {
page_size_value = std::stol(page_size->std_str());
} catch (const std::exception& e) {
std::string msg = "Query param \'page_size\' is illegal. Reason: " + std::string(e.what());
RETURN_STATUS_DTO(ILLEGAL_QUERY_PARAM, msg.c_str());
std::string page_size_str = page_size->std_str();
if (!ValidationUtil::ValidateStringIsNumber(page_size_str).ok()) {
RETURN_STATUS_DTO(ILLEGAL_QUERY_PARAM,
"Query param \'page_size\' is illegal, only non-negative integer supported");
}
page_size_value = std::stol(page_size_str);
}
if (offset_value < 0 || page_size_value < 0) {

1
core/src/server/web_impl/handler/WebRequestHandler.h
View File

@ -92,6 +92,7 @@ class WebRequestHandler {
public:
WebRequestHandler() {
context_ptr_ = GenContextPtr("Web Handler");
request_handler_ = RequestHandler();
}
StatusDto::ObjectWrapper

1
core/src/server/web_impl/utils/Util.h
View File

@ -17,6 +17,7 @@
#pragma once
#include <string>
#include <vector>
#include "db/Types.h"

2
core/src/utils/ValidationUtil.cpp
View File

@ -308,7 +308,7 @@ ValidationUtil::ValidateStringIsNumber(const std::string& str) {
return Status(SERVER_INVALID_ARGUMENT, "Invalid number");
}
try {
int32_t value = std::stoi(str);
int64_t value = std::stol(str);
if (value < 0) {
return Status(SERVER_INVALID_ARGUMENT, "Negative number");
}

2
core/unittest/server/test_config.cpp
View File

@ -578,9 +578,11 @@ TEST_F(ConfigTest, SERVER_CONFIG_INVALID_TEST) {
ASSERT_FALSE(config.SetGpuResourceConfigCacheThreshold("-0.1").ok());
ASSERT_FALSE(config.SetGpuResourceConfigSearchResources("gpu10").ok());
ASSERT_FALSE(config.SetGpuResourceConfigSearchResources("gpu0, gpu0").ok());
ASSERT_FALSE(config.SetGpuResourceConfigBuildIndexResources("gup2").ok());
ASSERT_FALSE(config.SetGpuResourceConfigBuildIndexResources("gpu16").ok());
ASSERT_FALSE(config.SetGpuResourceConfigBuildIndexResources("gpu0, gpu0, gpu1").ok());
#endif
}

145
core/unittest/server/test_web.cpp
View File

@ -508,7 +508,7 @@ class TestClient : public oatpp::web::client::ApiClient {
API_CALL("POST", "/tables", createTable, BODY_DTO(milvus::server::web::TableRequestDto::ObjectWrapper, body))
API_CALL("GET", "/tables", showTables, QUERY(Int64, offset), QUERY(Int64, page_size))
API_CALL("GET", "/tables", showTables, QUERY(String, offset), QUERY(String, page_size))
API_CALL("OPTIONS", "/tables/{table_name}", optionsTable, PATH(String, table_name, "table_name"))
@ -518,7 +518,7 @@ class TestClient : public oatpp::web::client::ApiClient {
API_CALL("OPTIONS", "/tables/{table_name}/indexes", optionsIndexes, PATH(String, table_name, "table_name"))
API_CALL("POST", "/tables/{table_name}/indexes",createIndex,
API_CALL("POST", "/tables/{table_name}/indexes", createIndex,
PATH(String, table_name, "table_name"), BODY_DTO(milvus::server::web::IndexRequestDto::ObjectWrapper, body))
API_CALL("GET", "/tables/{table_name}/indexes", getIndex, PATH(String, table_name, "table_name"))
@ -527,44 +527,25 @@ class TestClient : public oatpp::web::client::ApiClient {
API_CALL("OPTIONS", "/tables/{table_name}/partitions", optionsPartitions, PATH(String, table_name, "table_name"))
API_CALL("POST",
"/tables/{table_name}/partitions",
createPartition,
PATH(String, table_name, "table_name"),
BODY_DTO(milvus::server::web::PartitionRequestDto::ObjectWrapper, body))
API_CALL("POST", "/tables/{table_name}/partitions", createPartition,
PATH(String, table_name, "table_name"), BODY_DTO(milvus::server::web::PartitionRequestDto::ObjectWrapper, body))
API_CALL("GET",
"/tables/{table_name}/partitions",
showPartitions,
PATH(String, table_name, "table_name"),
QUERY(Int64, offset),
QUERY(Int64, page_size))
API_CALL("GET", "/tables/{table_name}/partitions", showPartitions,
PATH(String, table_name, "table_name"), QUERY(String, offset), QUERY(String, page_size))
API_CALL("OPTIONS",
"/tables/{table_name}/partitions/{partition_tag}",
optionsParTag,
PATH(String, table_name, "table_name"),
PATH(String, partition_tag, "partition_tag"))
API_CALL("OPTIONS", "/tables/{table_name}/partitions/{partition_tag}", optionsParTag,
PATH(String, table_name, "table_name"), PATH(String, partition_tag, "partition_tag"))
API_CALL("DELETE",
"/tables/{table_name}/partitions/{partition_tag}",
dropPartition,
PATH(String, table_name, "table_name"),
PATH(String, partition_tag))
API_CALL("DELETE", "/tables/{table_name}/partitions/{partition_tag}", dropPartition,
PATH(String, table_name, "table_name"), PATH(String, partition_tag))
API_CALL("OPTIONS", "/tables/{table_name}/vectors", optionsVectors, PATH(String, table_name, "table_name"))
API_CALL("POST",
"/tables/{table_name}/vectors",
insert,
PATH(String, table_name, "table_name"),
BODY_DTO(milvus::server::web::InsertRequestDto::ObjectWrapper, body))
API_CALL("POST", "/tables/{table_name}/vectors", insert,
PATH(String, table_name, "table_name"), BODY_DTO(milvus::server::web::InsertRequestDto::ObjectWrapper, body))
API_CALL("PUT",
"/tables/{table_name}/vectors",
search,
PATH(String, table_name, "table_name"),
BODY_DTO(milvus::server::web::SearchRequestDto::ObjectWrapper, body))
API_CALL("PUT", "/tables/{table_name}/vectors", search,
PATH(String, table_name, "table_name"), BODY_DTO(milvus::server::web::SearchRequestDto::ObjectWrapper, body))
API_CALL("GET", "/system/{msg}", cmd, PATH(String, cmd_str, "msg"))
@ -769,30 +750,45 @@ TEST_F(WebControllerTest, GET_TABLE) {
TEST_F(WebControllerTest, SHOW_TABLES) {
// test query table limit 1
auto response = client_ptr->showTables(1, 1, conncetion_ptr);
auto response = client_ptr->showTables("1", "1", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
auto result_dto = response->readBodyToDto<milvus::server::web::TableListFieldsDto>(object_mapper.get());
ASSERT_TRUE(result_dto->count->getValue() > 0);
ASSERT_TRUE(result_dto->count->getValue() >= 0);
// test query table empty
response = client_ptr->showTables(0, 0, conncetion_ptr);
response = client_ptr->showTables("0", "0", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
response = client_ptr->showTables(-1, 0, conncetion_ptr);
response = client_ptr->showTables("-1", "0", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
response = client_ptr->showTables(0, -10, conncetion_ptr);
response = client_ptr->showTables("0", "-10", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
// test wrong param
response = client_ptr->showTables("0.1", "1", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
response = client_ptr->showTables("1", "1.1", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
response = client_ptr->showTables("0", "90000000000000000000000000000000000000000000000000000000", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
}
TEST_F(WebControllerTest, DROP_TABLE) {
auto table_name = "table_drop_test" + OString(RandomName().c_str());
GenTable(table_name, 128, 100, "L2");
sleep(1);
auto response = client_ptr->dropTable(table_name, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_204.code, response->getStatusCode());
table_name = "table_drop_test_not_exists_" + OString(RandomName().c_str());
response = client_ptr->dropTable(table_name, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
auto error_dto = response->readBodyToDto<milvus::server::web::StatusDto>(object_mapper.get());
ASSERT_EQ(milvus::server::web::StatusCode::TABLE_NOT_EXISTS, error_dto->code->getValue());
}
TEST_F(WebControllerTest, INSERT) {
@ -809,6 +805,9 @@ TEST_F(WebControllerTest, INSERT) {
auto result_dto = response->readBodyToDto<milvus::server::web::VectorIdsDto>(object_mapper.get());
ASSERT_EQ(20, result_dto->ids->count());
response = client_ptr->insert(table_name + "ooowrweindexsgs", insert_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
response = client_ptr->dropTable(table_name, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_204.code, response->getStatusCode());
}
@ -876,6 +875,10 @@ TEST_F(WebControllerTest, INDEX) {
response = client_ptr->dropIndex(table_name, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_204.code, response->getStatusCode());
// create index without existing table
response = client_ptr->createIndex(table_name + "fgafafafafafUUUUUUa124254", index_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
index_dto->index_type = "J46";
response = client_ptr->createIndex(table_name, index_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
@ -918,6 +921,11 @@ TEST_F(WebControllerTest, INDEX) {
auto result_index_dto = response->readBodyToDto<milvus::server::web::IndexDto>(object_mapper.get());
ASSERT_EQ("FLAT", result_index_dto->index_type->std_str());
ASSERT_EQ(10, result_index_dto->nlist->getValue());
// get index of table which not exists
response = client_ptr->getIndex(table_name + "dfaedXXXdfdfet4t343aa4", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
auto error_dto = response->readBodyToDto<milvus::server::web::StatusDto>(object_mapper.get());
ASSERT_EQ(milvus::server::web::StatusCode::TABLE_NOT_EXISTS, error_dto->code->getValue());
}
TEST_F(WebControllerTest, PARTITION) {
@ -942,6 +950,11 @@ TEST_F(WebControllerTest, PARTITION) {
auto create_result_dto = response->readBodyToDto<milvus::server::web::StatusDto>(object_mapper.get());
ASSERT_EQ(milvus::server::web::StatusCode::SUCCESS, create_result_dto->code);
response = client_ptr->createPartition(table_name + "afafanotgitdiexists", par_param);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
error_dto = response->readBodyToDto<milvus::server::web::StatusDto>(object_mapper.get());
ASSERT_EQ(milvus::server::web::StatusCode::TABLE_NOT_EXISTS, error_dto->code);
// insert 200 vectors into table with tag = 'tag01'
OQueryParams query_params;
// add partition tag
@ -956,15 +969,34 @@ TEST_F(WebControllerTest, PARTITION) {
ASSERT_EQ(OStatus::CODE_201.code, response->getStatusCode());
// Show all partitins
response = client_ptr->showPartitions(table_name, 0, 10, conncetion_ptr);
response = client_ptr->showPartitions(table_name, "0", "10", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
auto result_dto = response->readBodyToDto<milvus::server::web::PartitionListDto>(object_mapper.get());
ASSERT_EQ(1, result_dto->partitions->count());
ASSERT_EQ("tag01", result_dto->partitions->get(0)->partition_tag->std_str());
ASSERT_EQ(par_param->partition_name->std_str(), result_dto->partitions->get(0)->partition_name->std_str());
response = client_ptr->showPartitions(table_name, "0", "-1", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
response = client_ptr->showPartitions(table_name, "0.1", "7", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
response = client_ptr->showPartitions(table_name, "0", "1.6", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
response = client_ptr->showPartitions(table_name, "567a", "1", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
// show without existing tables
response = client_ptr->showPartitions(table_name + "dfafaefaluanqibazao990099", "0", "10", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
error_dto = response->readBodyToDto<milvus::server::web::StatusDto>(object_mapper.get());
ASSERT_EQ(milvus::server::web::StatusCode::TABLE_NOT_EXISTS, error_dto->code->getValue());
response = client_ptr->dropPartition(table_name, "tag01", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_204.code, response->getStatusCode());
// drop without existing tables
response = client_ptr->dropPartition(table_name + "565755682353464aaasafdsfagagqq1223", "tag01", conncetion_ptr);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
}
TEST_F(WebControllerTest, SEARCH) {
@ -1026,6 +1058,12 @@ TEST_F(WebControllerTest, SEARCH) {
search_request_dto->tags->pushBack(par_param->partition_tag);
response = client_ptr->search(table_name, search_request_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
// Test search without existing table
response = client_ptr->search(table_name + "999piyanning", search_request_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_404.code, response->getStatusCode());
error_dto = response->readBodyToDto<milvus::server::web::StatusDto>(object_mapper.get());
ASSERT_EQ(milvus::server::web::StatusCode::TABLE_NOT_EXISTS, error_dto->code->getValue());
}
TEST_F(WebControllerTest, SEARCH_BIN) {
@ -1091,7 +1129,7 @@ TEST_F(WebControllerTest, CMD) {
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
}
TEST_F(WebControllerTest, ADVANCEDCONFIG) {
TEST_F(WebControllerTest, ADVANCED_CONFIG) {
auto response = client_ptr->getAdvanced(conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
@ -1119,10 +1157,16 @@ TEST_F(WebControllerTest, ADVANCEDCONFIG) {
config_dto->use_blas_threshold = 1000;
response = client_ptr->setAdvanced(config_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
//// test fault
// cpu cache capacity exceed total memory
config_dto->cpu_cache_capacity = 10000000;
response = client_ptr->setAdvanced(config_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
}
#ifdef MILVUS_GPU_VERSION
TEST_F(WebControllerTest, GPUCONFIG) {
TEST_F(WebControllerTest, GPU_CONFIG) {
auto response = client_ptr->getGPUConfig(conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
@ -1149,11 +1193,26 @@ TEST_F(WebControllerTest, GPUCONFIG) {
response = client_ptr->setGPUConfig(gpu_config_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
//// test fault config
// cache capacity exceed GPU mem size
gpu_config_dto->cache_capacity = 100000;
response = client_ptr->setGPUConfig(gpu_config_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
gpu_config_dto->cache_capacity = 1;
// duplicate resources
gpu_config_dto->search_resources->clear();
gpu_config_dto->search_resources->pushBack("GPU0");
gpu_config_dto->search_resources->pushBack("GPU1");
gpu_config_dto->search_resources->pushBack("GPU0");
response = client_ptr->setGPUConfig(gpu_config_dto, conncetion_ptr);
ASSERT_EQ(OStatus::CODE_400.code, response->getStatusCode());
}
#endif
TEST_F(WebControllerTest, DEVICESCONFIG) {
TEST_F(WebControllerTest, DEVICES_CONFIG) {
auto response = WebControllerTest::client_ptr->getDevices(conncetion_ptr);
ASSERT_EQ(OStatus::CODE_200.code, response->getStatusCode());
}