Commit Graph

84 Commits

Author SHA1 Message Date
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
bigsheeper
5cdd6ca251
Add quotaCenter, rateLimiter and rateCollector (#18636)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-09-16 09:56:47 +08:00
Bingyi Sun
626854cf0c
Refactor QueryCoord (#18836)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-15 18:48:32 +08:00
XuanYang-cn
086eb92ade
Enable sort for segcore RetrieveResults (#18894)
See also: #18893

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-14 11:04:30 +08:00
Cai Yudong
9dc3bbecbd
Change the type of slice_nqs and slice_topks from int32_t[] to int64_t[] (#18867)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-08-29 11:36:56 +08:00
congqixia
179b496824
Add cgo worker pool for querynode (#18461)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-09 16:34:37 +08:00
Jeng.Gwan
638f6c36e9
Support to get real row count of segment (#18115)
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-18 09:58:28 +08:00
Zach
1c9647ff31
Merging multiple shard query/search requests into one rpc request if QueryNode is the leader of multiple shards of Collection (#17588)
Signed-off-by: Zach41 <zongmei.zhang@zilliz.com>
2022-07-06 15:06:21 +08:00
Jiquan Long
216e2f80aa
Add msg id to log in query path (#17677)
/kind improvement

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-23 10:46:13 +08:00
Jiquan Long
875b6f88b0
Add channel information to flow graph node (#17349)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-06-15 21:04:10 +08:00
yah01
4e83f37899
Improve load performance (#17273)
Load segment's fields concurrently

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-06-02 17:56:03 +08:00
xige-16
8c69790383
Fix lost delete msg caused by loadSegment after watchDeltaChannel (#17308)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-02 16:56:04 +08:00
bigsheeper
f55fe6be3c
Merge historical replica and streaming replica in QueryNode (#17217)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-31 13:42:03 +08:00
congqixia
c88514bc49
Remove not used QueryChannel in Proxy and Query Cluster (#16856)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-30 19:50:04 +08:00
bigsheeper
a4ea2fb18a
Handle errors in DataNode and QueryNode flow graph (#17096)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-24 21:11:59 +08:00
zhenshan.cao
08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-23 16:41:58 +08:00
Letian Jiang
b3eb2b1d0d
Support deltaLog loading on growing segment (#16903)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-05-12 11:59:53 +08:00
Ten Thousand Leaves
216c45fbd6
Add tests where segments load delta logs (#16825)
issue: #16821

/kind improvements

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-10 17:59:52 +08:00
xige-16
515d0369de
Support string type in segcore (#16546)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
Xiaofan
89b4a34892
Fix Data race in NodeID (#16603)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
congqixia
69252f812d
Implement memory replica in Proxy, QueryNode and QueryCoord (#16470)
Related to #16298 #16291 #16154
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-04-20 16:15:41 +08:00
xige-16
27b4cbc098
Cherry pick remove translateHits commit to mater (#16436)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-04-08 20:27:31 +08:00
godchen
bb7a0766fe
Add dependency factory (#16204)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
Letian Jiang
09b90fd8bf
Fix QueryCollection retrieve unit test (#16376)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-04-04 21:07:30 +08:00
xige-16
99984b88e1
Support delete varChar value (#16229)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
Jiquan Long
ba37531456
Add support for loading multiple indexes (#16138)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-30 21:11:28 +08:00
godchen
478f6ca11e
Add cache for vector chunk (#15912)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-26 22:05:26 +08:00
xiyichan
ca129d4308
Support configurable msgstream (#16131)
Signed-off-by: xiyichan <2863768433@qq.com>
2022-03-24 10:15:25 +08:00
Jiquan Long
f8d9bc919d
Unify interface of vector index & scalar index. (#15959)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-21 14:23:24 +08:00
godchen
78557ca6d7
Replace minio kv with minio chunk manager (#15936)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-17 18:03:23 +08:00
jaime
97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API (#15971)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-15 14:45:22 +08:00
jaime
29975a7a26
Remove Consume() API from mqstream interface (#15886)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-11 20:09:59 +08:00
bigsheeper
9ad5c14a4a
Return error when create message stream failed in QueryNode (#15317)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-09 15:17:59 +08:00
zhenshan.cao
dff08dbf47
Refactor querynode unittest (#15929)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-08 17:39:58 +08:00
bigsheeper
6fec7ed6fb
Add search benchmark in QueryNode (#15854)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-08 10:15:59 +08:00
jaime
7a3a721380
Reconstruct mqstream module (#15784)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Cai Yudong
92c8e32ebd
Let MemoryKV.Load return error when key not exist (#15814)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-02 18:51:55 +08:00
bigsheeper
12eb8c3da5
Add prometheus metrics for QueryNode (#15649)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-02 14:49:55 +08:00
godchen
fcf0887d92
Add and implement chunk manager interface. (#15541)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-02-24 23:53:53 +08:00
xige-16
55cdc5aa35
Get indexInfo and segment size by queryCoord (#14207)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-02-08 21:57:46 +08:00
Cai Yudong
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
xige-16
c132302129
Clean flowGraph if watchChannel failed (#15303)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-20 10:01:38 +08:00
xige-16
dfc6670f0f
Unsubscribe channel after query node down (#15230)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-17 17:37:37 +08:00
bigsheeper
0c98f21d4d
Remove global sealed segments in historical (#15177)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-01-12 17:01:38 +08:00
Cai Yudong
b6a48817b0
Add MinioConfig in GlobalParams for all components (#15099)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-10 17:29:34 +08:00
Cai Yudong
7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-09 23:45:33 +08:00
Cai Yudong
f8f61fb116
Use BaseParams etcd configuration for all components (#15012)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-07 16:15:26 +08:00
bigsheeper
8a3a1b556a
Free OS memory in QueryNode and IndexNode (#14305)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-12-30 20:03:29 +08:00
Xiaofan
69087ff8dd
Support embedded etcd (#14333)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-12-29 14:35:21 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components (#13885)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00