test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
15,160 Commits 96 Branches 105 Tags 185 MiB
53734bef58
Commit Graph

524 Commits

Author SHA1 Message Date
godchen
bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
Ten Thousand Leaves
768bd4035f
Drop log level in delete node showDelBuf as they confuse our users. (#16410) 
issue: 16349

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-07 17:59:31 +08:00
xige-16
99984b88e1
Support delete varChar value (#16229) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-04-02 17:43:29 +08:00
congqixia
6111dcde52
Fix ticker leakage in datanode flowgraph (#16346) 
Using  in case ticker need to be closed may cause leakage
Use time.NewTicker with deferred Stop instead

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-01 19:35:29 +08:00
Ten Thousand Leaves
45be3deb3f
Implement RC, DC, DN calling path for import. (#16321) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-01 11:33:28 +08:00
Ten Thousand Leaves
0d54697d42
Implement ReportImport related logic. (#16312) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-31 13:51:28 +08:00
XuanYang-cn
23b3a9ed8a
Log state in datanode handleWatchInfo (#16231) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-29 00:15:28 +08:00
XuanYang-cn
a77dd1034c
Adopt the extended state in DataCoord (#16200) 
ChannelWatchInfo with ToWatch, ToRelease will trigger a timer.
ChannelManager now reacts to different ChannelWatch states.

- WatchSuccess > log this info

- WatchFailure/WatchTimeout > ToRelease

- ReleaseSuccess > Delete, reassign if not from DropCollection

- ReleaseFailure/ReleaseTimeout > Cleanup subscription and Delete,
  reassgin if not from DropCollection.

Some Notes:
1. Reassignment will add this channel to buffer if there's only one node.

See also: #15846

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-28 22:33:27 +08:00
xige-16
205c92e54b
Support insert string data (#15993) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-25 14:27:25 +08:00
xiyichan
ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
groot
2078b24dca
Prepare for import (#16083) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-21 15:47:23 +08:00
godchen
78557ca6d7
Replace minio kv with minio chunk manager (#15936) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-03-17 18:03:23 +08:00
Ji Bin
3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
jaime
97b1ed7bca
Remove reader API from Mqstream and replace it with consumer API (#15971) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-15 14:45:22 +08:00
congqixia
fab0263616
Use RmqFactory in DataNode unit test (#16039) 
Use Rocksmq instead of Pulsar in DataNode unittest
- Avoid pulsar seek problem caused timeout issue
- Remove third party component dependency in unit test

See also: #15986

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-14 19:22:01 +08:00
jaime
29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
groot
89356ebcb9
Modify import rpc interfaces (#16006) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-11 17:13:59 +08:00
groot
5fdef607d5
Add rpc interfaces for import (#15930) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-09 18:33:59 +08:00
congqixia
6513be5c44
Extend watch/release channel logic in DataNode (#15925) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-09 15:39:58 +08:00
Letian Jiang
74f66dce3b
Fix passing empty binlog content in compaction (#15909) (#15927) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-08 16:34:01 +08:00
Jiquan Long
f71651e294
Support column-based insert data in message stream (#15802) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-04 15:09:56 +08:00
Cai Yudong
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
jaime
7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
Letian Jiang
2be217bcf4
Specify appropriate timestamp in initPKBloomFilter (#15823) (#15826) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-03-03 17:33:56 +08:00
cai.zhang
88527fc0a9
Fix metrics label conflict (#15839) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-03-03 16:05:57 +08:00
XuanYang-cn
63eec7ffc8
Refactor logs in binlog_io to reduce output (#15827) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-02 16:49:55 +08:00
Xiaofan
bd7d294c6d
Refine log level in data (#15821) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-02 15:35:55 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode (#15650) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2022-02-28 19:11:55 +08:00
yah01
0cfb6a85be
Make compaction not generate empty segment (#15707) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-02-28 10:17:54 +08:00
XuanYang-cn
e8edaa02fa
Refine code of DataNode binlogIO (#15759) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-28 10:09:54 +08:00
XuanYang-cn
6f85b49a4f
Add more compaction test cases (#15715) 
Test merge compaction with 2 segments with the same PK and only 1
valid deletion.

This test can varify the compaction behavior of DataNode for the
dup PK in different segment cases.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-24 11:27:53 +08:00
XuanYang-cn
d011073269
Fix DataNode panic when compact empty segment (#15580) 
remove the TODO in compactor

See also: #15573

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-18 18:47:51 +08:00
XuanYang-cn
23dfbf1f34
Fix eventmanager retry interval (#15631) 
See also: #15608

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-18 17:31:49 +08:00
congqixia
5751759c85
Implement mix compaction logic (#15542) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-18 14:51:49 +08:00
Xiaofan
be42627ab2
Move Logutil to Util dir (#15447) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-09 18:55:46 +08:00
Cai Yudong
8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-09 14:41:45 +08:00
congqixia
2263425cde
Fix EntriesNum is not passed in flush task (#15482) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-09 10:05:46 +08:00
Cai Yudong
1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-08 20:57:47 +08:00
XuanYang-cn
43b6c295a4
Put logs of compaction elapse in defer (#15485) 
Before this PR, time elapse are logged after `compact()` finished
normally. No elapse will be logged with any errors.

This PR put elapse logs in defer once time-consuming steps are finished.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-08 19:15:45 +08:00
XuanYang-cn
46dea0f006
Add elapse logs in compaction (#15414) 
See also: #15413

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-08 14:11:45 +08:00
JunSung Park
f6873d3dc1
Implement automatic item expiration on compaction (#14418) 
* Implement automatic item expiration on compaction

Signed-off-by: JunSung Park <writtic@gmail.com>

* Change entityExpiration value comparison logic

Signed-off-by: JunSung Park <writtic@gmail.com>

* Add logging for the number of expired entity

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default value of CompactionEntityExpiration to off

Signed-off-by: JunSung Park <writtic@gmail.com>

* Fix errors for code checker

Signed-off-by: JunSung Park <writtic@gmail.com>

* Set default entityExpiration to max value

Signed-off-by: JunSung Park <writtic@gmail.com>
 2022-02-07 22:45:46 +08:00
XuanYang-cn
5eef9886b3
Add more specific log when exiting BackGroundGC goroutine (#15273) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-07 16:55:44 +08:00
Cai Yudong
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Cai Yudong
4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-02 00:35:43 +08:00
congqixia
273890a91e
Fix datanode flush manager injection unit test (#15358) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-27 10:23:40 +08:00
XuanYang-cn
8ab9d769de
Fix logs of compaction_executor (#15365) 
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-25 13:29:40 +08:00
congqixia
1e61112b5a
Add retry logic in pulsar consumer unsubscribe (#15284) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-20 01:23:37 +08:00
XuanYang-cn
87394e29b6
Fix compaction ut datarace (#15267) 
Fixes: #15241

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-18 17:49:39 +08:00
bigsheeper
7f775096c5
Unsubscribe when close pulsar consumer (#15221) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-01-17 18:53:36 +08:00
yah01
fd8d452cfd
remove unused fieldIDs and fieldTypes (#15229) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-01-16 17:37:35 +08:00
congqixia
2528b6850d
Refactor flush manager injection to reduce goroutine number (#15180) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-13 18:49:34 +08:00
XuanYang-cn
008e08a996
Add flowgraph manager and event manager (#15102) 
1. Add flowgraph manager to manager start and release of dataSyncService
2. Add event manager to manager etcd watchInfo event
  - Make put event able to retry
  - Make delete event able to terminate the retry loop

See also: #14604, #14300

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-13 14:21:34 +08:00
XuanYang-cn
6e847ff194
Make dn handleChannelInfo log more consist (#14936) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-11 09:37:35 +08:00
Cai Yudong
b6a48817b0
Add MinioConfig in GlobalParams for all components (#15099) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 17:29:34 +08:00
XuanYang-cn
45f1f97ac4
Fix deltalog savebinlogpath save log (#15092) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-10 15:45:34 +08:00
Cai Yudong
7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-09 23:45:33 +08:00
Cai Yudong
f8f61fb116
Use BaseParams etcd configuration for all components (#15012) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-07 16:15:26 +08:00
Xieql
8b06aac7a9
[skip e2e] Improve annotation (#14976) 
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
 2022-01-07 13:57:49 +08:00
zhenshan.cao
a5f8bdeb42
Fix bug: data race in MsgFactory (#14839) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-06 14:57:20 +08:00
Xieql
25b00230c8
[skip e2e] Improve annotation (#14908) 
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
 2022-01-06 14:15:20 +08:00
Xieql
c488c41dfb
[skip e2e] Improve annotation (#14912) 
Signed-off-by: Xieql <qianglong.xie@zilliz.com>
 2022-01-06 13:38:08 +08:00
congqixia
ecdc3f7bf7
[skip e2e] Improve comment for DataNode delete node (#14825) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-01-05 14:29:35 +08:00
godchen
98232b4942
[skip e2e]Sort imports (#14820) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:25:48 +08:00
godchen
e1eb721532
[skip e2e]Sort imports (#14819) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:23:52 +08:00
godchen
7031a511ce
[skip e2e]Sort imports (#14806) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:00:38 +08:00
zhenshan.cao
b10c361822
Fix bug: should use Lock instead of RLock (#14797) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-04 23:33:24 +08:00
zhenshan.cao
76a9687310
Adjust usage of lock in datanode (#14631) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-04 19:35:32 +08:00
zhenshan.cao
7abebe576c
Add more logs on write path (#14600) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-04 15:47:18 +08:00
XuanYang-cn
f0a40f1569
Make FieldBinlog in a segment unique for a field (#14632) 
See also: #14620

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-30 20:29:22 +08:00
Jiquan Long
ebb9b24b47
Improve name of flowgraph node (#14538) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-30 10:33:46 +08:00
XuanYang-cn
9d0bc136e0
Add vChannelName in IBN timetick log (#14527) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-30 10:09:48 +08:00
Xiaofan
69087ff8dd
Support embedded etcd (#14333) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-12-29 14:35:21 +08:00
congqixia
a8c7e0c827
[skip e2e] Add comment for Compaction of DataNode (#14350) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-28 09:38:07 +08:00
XuanYang-cn
3cfa88d6f2
Fix delete node not aware of merged segments error (#14190) 
This PR made merge gracefully remove merged segment by moving
them into `compactedSegments`

And everytime when delete_node operates, if compactedSegments
have contents, delete_node will replace the latest segment in delBuf
merged from the merged segments, and truely remove the compactedSegments

Fixes: #14085

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-27 11:02:18 +08:00
zhenshan.cao
9a2c91d326
Fix bug: refresh config after session init (#14194) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-12-24 22:22:17 +08:00
XuanYang-cn
3d24007efa
Add more logs for load sealed segment (#14079) 
Add logs before&after flowgraph_ddnode forwarding delete msg

See also: #13879

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-24 10:50:07 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components (#13885) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-23 18:39:11 +08:00
Jiquan Long
0871bd3d63
Add log when update pk range (#14006) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-12-23 11:57:08 +08:00
Bingyi Sun
6b91e5e9d1
Add warning log to trace old messages (#13939) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-22 16:59:05 +08:00
congqixia
2b352bcde5
Fix compact result segment log size always zero (#13953) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-22 14:27:06 +08:00
Xiangyu Wang
2a6168e937
Improve import statement (#13734) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-21 19:15:32 +08:00
JackLCL
87071f3f41
[skip e2e]Fix error log output format in segment_replica.go (#13806) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-20 22:51:25 +08:00
Xiangyu Wang
5d4c4f38c8
Improve import statement (#13735) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 15:49:06 +08:00
Xiangyu Wang
cabc40cb09
Improve import statement (#13737) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 14:39:12 +08:00
Xiangyu Wang
a7dfd53f53
Improve import statement (#13736) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 14:38:25 +08:00
Xiangyu Wang
08be6888d5
Improve import statement (#13733) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 14:33:47 +08:00
congqixia
2de5cf7762
[skip e2e] Improve log in flush manager (#13689) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-20 12:22:43 +08:00
congqixia
d4afc812d1
Normalize DataNode term in errors.go (#13688) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-20 12:20:50 +08:00
congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs (#13606) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-19 20:00:42 +08:00
JackLCL
20c956b675
[skip e2e]Fix error log output format in segment_replica.go (#13676) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-19 12:32:42 +08:00
godchen
154f11e3a1
[skip e2e]Fix redundant conversion (#13648) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 22:28:50 +08:00
godchen
e191b23249
[skip e2e]Fix error string (#13647) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:22:42 +08:00
godchen
fefeaa2e1d
[skip e2e]Fix compaction executor typo (#13646) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:20:46 +08:00
cai.zhang
9f23fc7f2a
Register the service when the component state is healthy (#13248) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2021-12-15 11:47:10 +08:00
Bingyi Sun
0c39e0370c
Move segment statistics update to datanode tt loop (#13233) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-15 10:53:16 +08:00
JackLCL
30afcf2311
[skip ci]Fix error log output format in segment_replica.go (#13350) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-14 16:51:09 +08:00
JackLCL
5f02a3738a
[skip ci]Fix error log output format in meta_service.go (#13349) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-14 16:43:25 +08:00
Xiangyu Wang
405b3cd932
Use the same lint rules with golint in revive (#13324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-14 15:31:07 +08:00
congqixia
8f87cc7ef1
Fix JoinIDPath logic problem (#13039) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-09 12:05:04 +08:00