Commit Graph

4988 Commits

Author SHA1 Message Date
bigsheeper
12eb8c3da5
Add prometheus metrics for QueryNode (#15649)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-03-02 14:49:55 +08:00
Xiaofan
827389c704
Refine tso log print (#15813)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-02 14:31:55 +08:00
zhenshan.cao
f1aec49a74
Fixbug: building index return nil by mistake when error not occur (#15809)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-02 09:41:56 +08:00
XuanYang-cn
e394ba8c88
Refactor rocksmq comments and logs for readability (#15801)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-03-01 18:55:55 +08:00
Cai Yudong
529098fd87
[skip ci] Update knowhere/CMakeLists.txt (#15804)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-01 17:55:55 +08:00
Jiquan Long
001567640e
Fix shallow copy of typeutil.AppendFieldData (#15790)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-01 15:13:54 +08:00
Cai Yudong
503724be19
Optimize CMakeLists.txt under internal/core (#15770)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-01 10:31:55 +08:00
Nemo
a77e0562b2
Make C++ log level configurable and add log configuration while building embedded milvus. (#15743)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-01 10:15:55 +08:00
godchen
2e4391e6c5
Add lru cache (#15733)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-02-28 21:43:55 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode (#15650)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
Bingyi Sun
d6857f4b24
Remove globalZapWrapper in logutil.go (#15726)
issue: #15720

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-02-28 17:23:54 +08:00
bigsheeper
e1bfd14db1
Add Prometheus metrics for QueryCoord (#15606)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: jingkl <34296482+jingkl@users.noreply.github.com>
2022-02-28 16:51:55 +08:00
Bingyi Sun
b41992eda1
Fix querycoord ut bug (#15752)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-02-28 11:27:54 +08:00
yah01
0cfb6a85be
Make compaction not generate empty segment (#15707)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-28 10:17:54 +08:00
XuanYang-cn
e8edaa02fa
Refine code of DataNode binlogIO (#15759)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-28 10:09:54 +08:00
yah01
867cf620ee
Add Extract() method to take out C pointer from BytesConverter without releasing it (#15614)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-27 09:47:53 +08:00
bigsheeper
6a48071a5c
Set BaseMsg for LoadSegmentRequest, prevent crashing in log printing (#15725)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-02-26 20:09:54 +08:00
Cai Yudong
adca79facc
Fix centos build error (#15760)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-25 18:49:54 +08:00
yah01
d67c801511
Add bytes methods for RocksdbKV (#15749)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-02-25 17:19:53 +08:00
yah01
1a9933a7c3
Add methods to load/save bytes for MinIOKV (#15732)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-02-25 16:21:53 +08:00
Nemo
cdcb3627b4
Make Etcd log level configurable. (#15737)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-25 15:03:53 +08:00
congqixia
a05df77e96
Remove unused ValueKV (#15748)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-25 14:47:52 +08:00
Letian Jiang
b56ec7ea97
Add ByteSlice Method for Etcd-kv (#15738)
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-02-25 11:25:53 +08:00
godchen
fcf0887d92
Add and implement chunk manager interface. (#15541)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-02-24 23:53:53 +08:00
xige-16
1e8e3eb912
Fix wrong flushSegmentInfos in watchdmChannel request (#15740)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-02-24 19:23:52 +08:00
XuanYang-cn
6f85b49a4f
Add more compaction test cases (#15715)
Test merge compaction with 2 segments with the same PK and only 1
valid deletion.

This test can varify the compaction behavior of DataNode for the
dup PK in different segment cases.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-24 11:27:53 +08:00
shaoyue
83fdc1da7c
Add web server in proxy with sample handler (#15647)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-02-23 14:37:52 +08:00
congqixia
b0923f1299
Fix DataCoord panics if message queue service quits before it (#15702)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-23 11:09:52 +08:00
Cai Yudong
54b8b24151
Rename variable names for better readibility (#15700)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-22 22:15:52 +08:00
Bingyi Sun
ce9662c140
Fix data race in test (#15690)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-02-22 17:43:52 +08:00
Cai Yudong
9a6ee61731
Change all fields in Expr to const to avoid modification (#15663)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-22 14:45:51 +08:00
Nemo
3eb074a9b0
Update some comments for data coord with some minor code refactoring. (#15686)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-22 13:15:51 +08:00
xige-16
782f350da5
Modify parameter spelling of task queue (#15581)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-02-22 10:25:51 +08:00
cai.zhang
b435c422c1
Add prometheus metrics for IndexCoord (#15638)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-02-21 17:15:51 +08:00
XuanYang-cn
d011073269
Fix DataNode panic when compact empty segment (#15580)
remove the TODO in compactor

See also: #15573

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-18 18:47:51 +08:00
zhenshan.cao
142848fcc3
Abandon using protobuf to pass binaryset parameter (#15626)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-02-18 18:39:50 +08:00
Jenny Li
83f1358408
Update codecov.yml (#15639)
Signed-off-by: Jenny Li <jing.li@zilliz.com>
2022-02-18 18:29:50 +08:00
XuanYang-cn
23dfbf1f34
Fix eventmanager retry interval (#15631)
See also: #15608

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-18 17:31:49 +08:00
Bingyi Sun
2e676a4b78
Extract segments info into a new struct. (#15537)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-02-18 15:27:50 +08:00
congqixia
5751759c85
Implement mix compaction logic (#15542)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-18 14:51:49 +08:00
Bingyi Sun
daf25bcb81
Add log when remove keys in garbage collector. (#15453)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-02-18 11:59:49 +08:00
Nemo
dc65b1aeea
Switch to knowhere v1.0.1. Also switch to forked gorocksdb v0.0.1 (#15603)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-02-17 19:05:49 +08:00
congqixia
c7f55c2e34
Make SessionWatch keep watch even no Rewatch func when find ErrCompacted (#15497)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-15 15:07:48 +08:00
yah01
684110bc9a
Avoid to copy during converting C bytes to Go bytes (#15551)
* Avoid to copy during converting C bytes to Go bytes

Signed-off-by: yah01 <yah2er0ne@outlook.com>

* Add BytesConverter to achieve no-copy converting C bytes to Go bytes

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-15 14:17:48 +08:00
congqixia
7daed4ee47
Return flush state true when segment in Flushing state (#15530)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-10 16:55:46 +08:00
Xiaofan
d499ffc421
Fix confusing logs in rootcoord (#15431)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-10 13:51:47 +08:00
Xiaofan
a18d6b46a4
Fix watch dml channel fail because of no collection meta (#15436)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-10 13:19:46 +08:00
Xiaofan
be42627ab2
Move Logutil to Util dir (#15447)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-02-09 18:55:46 +08:00
jaime
87dd4e8f92
Fix clean up data dir failed for TestEtcdRestartLoad (#15506)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-02-09 18:37:45 +08:00
jaime
cb9c63d039
Remove index dir (#15467)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-02-09 16:07:45 +08:00
Cai Yudong
8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-09 14:41:45 +08:00
jaime
307a8ce535
Support compile and run on Mac (#15491)
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
2022-02-09 14:27:46 +08:00
guoxiangzhou
f43d96b58d
[skip e2e] Modify code comments according to the naming conventions (#15173)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-02-09 10:19:56 +08:00
guoxiangzhou
a157487f8c
[skip e2e] Modify code comments according to the naming conventions (#15172)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-02-09 10:18:02 +08:00
guoxiangzhou
c7e93f95aa
[skip e2e] Modify code comments according to the naming conventions (#15171)
Signed-off-by: guoxiangzhou <xiangzhou.guo@zilliz.com>
2022-02-09 10:14:05 +08:00
congqixia
2263425cde
Fix EntriesNum is not passed in flush task (#15482)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-09 10:05:46 +08:00
xige-16
55cdc5aa35
Get indexInfo and segment size by queryCoord (#14207)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-02-08 21:57:46 +08:00
Cai Yudong
1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-08 20:57:47 +08:00
XuanYang-cn
43b6c295a4
Put logs of compaction elapse in defer (#15485)
Before this PR, time elapse are logged after `compact()` finished
normally. No elapse will be logged with any errors.

This PR put elapse logs in defer once time-consuming steps are finished.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-08 19:15:45 +08:00
XuanYang-cn
46dea0f006
Add elapse logs in compaction (#15414)
See also: #15413

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-08 14:11:45 +08:00
Ji Bin
d83f69bb21
Support windows compliation (#15448)
- remove dl dependency while compiling some lib under MSYS
- int types aligement in golang
- core/src/utils adaptation for compile under MSYS

issue: #7706

Signed-off-by: Ji Bin <matrixji@live.com>
2022-02-07 23:43:46 +08:00
JunSung Park
f6873d3dc1
Implement automatic item expiration on compaction (#14418)
* Implement automatic item expiration on compaction

Signed-off-by: JunSung Park <writtic@gmail.com>

* Change entityExpiration value comparison logic

Signed-off-by: JunSung Park <writtic@gmail.com>

* Add logging for the number of expired entity

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default value of CompactionEntityExpiration to off

Signed-off-by: JunSung Park <writtic@gmail.com>

* Fix errors for code checker

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default entityExpiration to max value

Signed-off-by: JunSung Park <writtic@gmail.com>
2022-02-07 22:45:46 +08:00
XuanYang-cn
5eef9886b3
Add more specific log when exiting BackGroundGC goroutine (#15273)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-02-07 16:55:44 +08:00
congqixia
d830791e4f
Respect SegmentIDs in querynode.GetSegmentInfo (#15455)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-02-07 15:35:44 +08:00
Cai Yudong
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-07 10:09:45 +08:00
Cai Yudong
4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
congqixia
b4bfe58fd9
[skip e2e] Fix component name for IndexCoordComponent comment (#15138)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-29 22:39:44 +08:00
XuanYang-cn
eef295465a
Fix timetick inconsistency in drop collection (#15408)
When rootcoord drops a collection, it'll produce a `drop-collection DDL` msg
and one last timetick into DML channels of this collection.
So that when DataNode receives this msg, DN can release the resources for the specific collection.

Before this PR, RootCoord produced these two msgs with an older timestamp, generated
before many time-consuming RPCs.

Once these RPCs spend more time than timetick producing interval, the timetick of these 2 msgs are
older to the channel current timestamp, causing in-consistency in time, thus
making msgstream fail to consume the last `drop-collection DDL` msg.

This PR generates a new timestamp for `drop-collection DDL` msg and timetick msg after those time-consuming
RPCs

Fixes: #15406

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-28 20:49:41 +08:00
Filipe Caixeta
dee1961b4d
Make SearchResults and QueryResults return collectionID (#15377)
* Make SearchResults and QueryResults return collectionID

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>

* add return collection name from QueryRequest and SearchRequest

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>

* remove comment

Signed-off-by: Filipe Caixeta <filipecaixeta@gmail.com>
2022-01-27 22:55:41 +08:00
Xiaofan
885ecaf222
Refine time sync logic log (#15251)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-01-27 11:25:41 +08:00
congqixia
273890a91e
Fix datanode flush manager injection unit test (#15358)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-27 10:23:40 +08:00
xige-16
b9d1396258
Delete ut's useless param for genCollectionSchema (#15119)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-26 14:21:40 +08:00
xige-16
6336e2326c
Set triggerCondition when queryCoord reload loadbalanceTask meta (#15380)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-25 17:26:13 +08:00
XuanYang-cn
8ab9d769de
Fix logs of compaction_executor (#15365)
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-25 13:29:40 +08:00
zhenshan.cao
3f48a40ab0
Load index meta in every retry (#15376)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-25 13:28:05 +08:00
Bingyi Sun
515fe962d5
Fix subscription path (#15373)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-25 13:27:44 +08:00
xige-16
3a024307f4
Release collection first when drop collection (#15370)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-24 21:51:22 +08:00
zhenshan.cao
5bc6ec7250
Fixbug: QueryNode search exceptions will cause memory leak (#15305)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-24 21:19:39 +08:00
Bingyi Sun
3987574448
Unsubscribe topic in DataCoord (#15353)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-24 19:47:38 +08:00
zhenshan.cao
6bc2efe429
Fixbug: IndexNode should panic when save meta failed to MetaKV (#15347)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-24 17:18:46 +08:00
bigsheeper
390cdd2515
Remove tmporary debug log in core (#15330)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-01-24 10:03:41 +08:00
xige-16
1cd2363bd8
Delete usless params SkipQueryChannelRecovery (#15289)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-21 21:43:40 +08:00
xige-16
57383c9f76
Delete Useless timetick channel (#15296)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-21 14:31:39 +08:00
Bingyi Sun
c877f33084
Downgrade pulsar client (#15325)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-01-20 22:25:38 +08:00
xige-16
c132302129
Clean flowGraph if watchChannel failed (#15303)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-20 10:01:38 +08:00
congqixia
1e61112b5a
Add retry logic in pulsar consumer unsubscribe (#15284)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-20 01:23:37 +08:00
xige-16
bbb672cb00
Remove state service in querynode (#15281)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-18 18:43:37 +08:00
XuanYang-cn
87394e29b6
Fix compaction ut datarace (#15267)
Fixes: #15241

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-18 17:49:39 +08:00
Cai Yudong
356efee492
Rename timetickSync APIs and variables for better readability (#15264)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-18 14:47:36 +08:00
cai.zhang
f564ea7fc7
RootCoord set healthy state after registered service (#15261)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-01-18 12:09:37 +08:00
Cai Yudong
1b1d4e502b
Remove session from timetickSync (#15255)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-18 00:07:37 +08:00
bigsheeper
7f775096c5
Unsubscribe when close pulsar consumer (#15221)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-01-17 18:53:36 +08:00
xige-16
408a6e9e58
Fix collectResultLoop hang after search timeout (#15250)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-17 18:11:36 +08:00
xige-16
dfc6670f0f
Unsubscribe channel after query node down (#15230)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-01-17 17:37:37 +08:00
zhenshan.cao
871e093dc8
Call release memory in case error occured (#15237)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-17 14:43:38 +08:00
Jiquan Long
675e6d352b
QueryNodes send search/query results by rpc (#15223)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-01-17 14:41:35 +08:00
yah01
fd8d452cfd
remove unused fieldIDs and fieldTypes (#15229)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-01-16 17:37:35 +08:00
Xiaofan
f36e02dfbb
Add more log when tt failed (#15227)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-01-15 18:53:34 +08:00
Xiaofan
f617c8887b
Time Tick Failed need log (#15225)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-01-14 23:55:34 +08:00
zhenshan.cao
1601a61b78
Move logic of checking available port to service (#15222)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-14 21:17:34 +08:00