congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs ( #13606 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
bigsheeper
5e5787ddd2
Update memory util in metricinfo ( #13320 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-12-16 17:25:13 +08:00
XuanYang-cn
b1ec235644
[skip ci]Update license for qn segment ( #13193 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-10 20:59:42 +08:00
xige-16
8b2cd271c7
Improve querynode log ( #13158 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-12-10 18:51:08 +08:00
congqixia
8f87cc7ef1
Fix JoinIDPath logic problem ( #13039 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-09 12:05:04 +08:00
Cai Yudong
1a806a174a
Support load collection with multiple vector column ( #12630 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-03 11:29:33 +08:00
godchen
c4f0837d84
Fix insert pks overwrite ( #12365 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-29 22:01:41 +08:00
godchen
f31ed089b5
Add timeout for reader next ( #12308 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-26 22:45:24 +08:00
godchen
e3f49858c5
Add subscription prefix ( #12251 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-24 17:47:15 +08:00
godchen
3fa5b4a9ce
Delete for segment indexing ( #12197 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-23 17:17:16 +08:00
godchen
9d5bcd3e3a
Close event and binlog reader ( #12173 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-22 17:27:14 +08:00
godchen
02bf77aabf
Close reader stream ( #12167 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-22 16:17:14 +08:00
godchen
116a503b4a
From dml cp load delete msg ( #12144 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-21 07:33:14 +08:00
Xiaofan
f654621650
Remove the etcd deletion of segment data path on querynode ( #11863 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-17 12:13:10 +08:00
xige-16
d857577a7b
Postpone the execution of handoff until index creation is complete ( #11648 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-11-12 18:49:10 +08:00
bigsheeper
93149c5ad9
Load growing segment in query node ( #11664 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-11-12 18:27:10 +08:00
godchen
863f1bb34e
Fix multi delete data not effect ( #11422 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-09 15:01:17 +08:00
cai.zhang
5b42a3223c
Increase compatibility for EstimateMemorySize interface ( #10603 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-10-26 15:34:21 +08:00
yukun
6949350b8b
Call segmentLoadDeletedRecord in segment_loader ( #10472 )
...
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-10-23 09:31:10 +08:00
bigsheeper
5c58924420
Add cgroups to get container memory and check index memory in segment loader ( #10363 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-10-22 20:49:10 +08:00
xige-16
8a046a6495
BroadCast sealedSegmentChangeInfo to queryChannel ( #10084 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-10-22 19:07:15 +08:00
godchen
ffc0c07610
Change delete data primary key to int64 ( #10438 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-22 15:37:12 +08:00
godchen
d6bf0c928f
Get stats delta path from datacoord ( #10362 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-22 14:31:13 +08:00
congqixia
f39e06b24e
Filters out nil BF stats ( #10279 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-20 16:14:36 +08:00
sunby
db53eccec9
Add binlog iterators ( #10173 )
...
issue: #9530
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-20 11:52:35 +08:00
godchen
8daeb0d519
Change bloom filter use pk ( #10193 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-19 20:18:47 +08:00
godchen
24a6fc096b
Update query node bloom filter ( #10109 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-18 20:08:42 +08:00
bigsheeper
e7d92e4e79
Add streaming segments memory to used memory when checkLoadMemory ( #8940 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-09-30 21:54:08 +08:00
zhenshan.cao
d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal ( #8542 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-09-29 20:26:00 +08:00
bigsheeper
9d6c95bf25
Add cacheSize to prevent OOM in query node ( #8765 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-09-28 22:24:03 +08:00
bigsheeper
338e32362b
Return error if setSegment failed when loadSegment ( #8075 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-09-18 10:03:50 +08:00
bigsheeper
691e4c5bc9
Use definitional type instead of raw type ( #7797 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-09-14 10:25:26 +08:00
bigsheeper
cb5cd1cf12
Add index unittests for query node ( #7674 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-09-10 17:11:09 +08:00
bigsheeper
b7103357fd
Load failed if collection beyonds the memory limit ( #7434 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-09-07 11:35:18 +08:00
bigsheeper
b9d3897deb
Return error when any failed detected in loadSegment ( #7005 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-08-12 14:14:08 +08:00
bigsheeper
4123deef9e
Return error message in grpc.LoadSegment in query node ( #6908 )
...
* return load error
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* init streaming meta in loadSegment
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* fix excluded init
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
* improve impl
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-08-03 22:01:27 +08:00
bigsheeper
e8ea8b51d5
improve query node log ( #6897 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-07-31 10:47:22 +08:00
godchen
db94d7771f
Read vector from disk ( #6707 )
...
* Read vector from disk
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* go fmt
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix git action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix test error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix caculate error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change var name
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix len error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* remove unused code
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change bytes to float method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* change float to bytes method
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
* fix action error
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-07-24 09:25:22 +08:00
Cai Yudong
a7b27db63e
update fillRetrieveResults parameter ( #6598 )
...
* update fillRetrieveResults parameter
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* optimize fillVectorFieldData process
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-07-17 15:17:30 +08:00
Cai Yudong
a992dcf6a8
Support query return vector output field ( #6570 )
...
* improve code readibility
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add offset in RetrieveResults
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add VectorFieldInfo into Segment struct
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add new interface for query vector
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update load vector field logic
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update load vector field logic
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fill in field name in query result
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add FieldId into FieldData
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add fillVectorOutputFieldsIfNeeded
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* update data_codec_test.go
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* add DeserializeFieldData
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* realize query return vector output field
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix static-check
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* disable query vector case
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-07-16 17:19:55 +08:00
xige-16
2b720fd2f7
Fix load hang when restart querynode many time in short time ( #6351 )
...
* fix load hang when restart querynode many time in short time
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add multi queryNode ut
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add ut for restart querynode
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* set queryCoord contex to load collection
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-07-13 14:16:00 +08:00
Cai Yudong
95135c1681
fix blob file sort error ( #6184 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-29 13:24:15 +08:00
xige-16
2af1c27811
Rename queryservice to querycoordinator ( #5931 )
...
* Rename queryservice to querycoordinator
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix param table
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* code format
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix unittest
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* service to coordinator
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* code format
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* set querycoord in ci
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-22 16:44:09 +08:00
xige-16
7039fb7c82
Fix the problem of stuck after loadBalance and loadFieldData error ( #5960 )
...
* delete nodeInfo after nodeDown
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix load balance can's stop
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix load field data error
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* contiue task loop after error in queryService
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-22 14:10:09 +08:00
sunby
7f2d7c1da3
Rename dataservice to datacoordinator ( #5912 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 18:22:13 +08:00
Cai Yudong
92e429d812
Rename IndexService to IndexCoord ( #5932 )
...
* rename package indexservice to indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename indexservice to indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix queryservice static-check
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move distributed/indexservice to distributed/indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* move internal/indexservice to internal/indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename indexservice to indexcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename MasterComponent to RootCoordComponent
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for queryservice
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for dataservice
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for datanode
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for proxynode
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord for querynode
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename master to rootcoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename IndexService to IndexCoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* rename IndexService to IndexCoord
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
* fix rebase issue
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-21 17:28:03 +08:00
FluorineDog
255e3959af
support time travel ( #5894 )
...
* support time travel
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
* lint
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-19 17:38:11 +08:00
xige-16
e4c51aae36
Add recovery logic for querynode and queryservice ( #5843 )
...
* merge milvus/recovery2
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add recovery logic in queryservice
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* debug smoke case
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add etcd to querynode
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix release partition error
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix load balance error
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* debug querynode down and recovery
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* add log
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* fix showCollection
Signed-off-by: xige-16 <xi.ge@zilliz.com>
* skip smoke test search without insert
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-19 11:45:09 +08:00
xige-16
97049ce293
use dataservice's getRecoveryInfo to get load and watch info ( #5790 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-16 11:09:56 +08:00
bigsheeper
cdbc6d2c94
Refactor query node and query service ( #5751 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 12:41:40 +08:00
