test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,163 Commits 96 Branches 105 Tags 185 MiB
96dcee5dff
Commit Graph

96 Commits

Author SHA1 Message Date
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module (#33881) 
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-25 21:18:15 +08:00
yiwangdr
e895cfed84
fix: reduce redundant map operations in datacoord (#33343) 
More refactories will be added.
issue: #33342

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-05-24 12:47:40 +08:00
congqixia
8cf2cf5c94
enhance: Add go-deadlock as unittest only dependency (#33063) 
See also #33062

This PR:

- Add `lock.RWMutex` & `lock.Mutex` alias to switch implementation based
  on build flags
- When build flags has `test` in it, use `go-deadlock` to detect
  possible deadlocks
- Replace all `sync.RWMutex` & `sync.Mutex` in datacoord pkg

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-15 16:33:34 +08:00
yiwangdr
b1eacb2ae8
feat: datacoord/node watch based on rpc (#32036) 
issue: https://github.com/milvus-io/milvus/issues/25309

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-05-07 15:49:30 +08:00
yiwangdr
037de8e4d3
enhance: speed up minor functions calls in datacoord (#32389) 
Related to https://github.com/milvus-io/milvus/issues/32165

1. nodeid based channel store access should use map access instead of
iteration.

2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-04-20 07:55:21 +08:00
congqixia
83da08c388
enhance: Use map instead of slice to maintain channel info (#32273) 
See also #32165

`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.

This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-16 15:57:19 +08:00
wei liu
0d849a6c0a
fix: fix collectionInfo leak in datacoord (#32175) 
issue: #32029

lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-15 16:33:19 +08:00
smellthemoon
1c1f2a1371
enhance:change some logs (#29579) 
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-05 16:12:48 +08:00
XuanYang-cn
623939c9f5
enhance: Remove not in use policies (#29448) 
THe results don't meet our requirements, and the code hasn't been
maintained for a long time.

See also: #29447

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-28 10:38:46 +08:00
XuanYang-cn
ae180d1628
enhance: Change ChannelManager to interface (#29300) 
Rewrite cluster test
issue: #28854

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-25 19:24:46 +08:00
wei liu
fdbca10e23
fix: Fix channel manager bg checker exit when disable auto balance (#28459) 
issue: #28454

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-20 18:20:22 +08:00
XuanYang-cn
a153950b10
Change channel to Interface (#27839) 
This PR changes `*channel` into RWChannel interface

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-13 11:16:18 +08:00
wei liu
14c8a90517
Fix auto balance block channel reassign after datanode restart (#28275) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-09 19:00:25 +08:00
wei liu
5b45a138b1
disable auto balance when old node exists (#28191) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 14:02:20 +08:00
jaime
6749957e71
Refine RPC call in unwatch drop channel (#27864) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-24 17:46:15 +08:00
Xiaofan
2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 (#27483) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-23 21:44:09 +08:00
jaime
d2dbbbc11b
Reduce write lock scope in channel manager (#27823) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-21 07:58:16 +08:00
congqixia
49516d44b4
Add ctx parameter and log tracer for watch and selectNodes (#27809) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-20 04:22:11 +08:00
MrPresent-Han
cb71a3e235
rm dependency to rc when getting recovery info(#25363) (#27405) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-09 18:51:32 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia
8d13717cac
Fill Collection start position timestamp in WatchInfo (#26370) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 09:05:32 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yiwangdr
c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
Xiaofan
72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
congqixia
4a22af6e1a
Unwatch channel in watch buffer (#23548) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-20 10:34:31 +08:00
congqixia
d83654c33f
Add Close method for ChannelManager in datacoord (#23493) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-18 17:54:31 +08:00
zhenshan.cao
4a32b842e8
Improve the check logic of channel remove (#23473) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-18 02:58:30 +08:00
congqixia
ba84f52119
Fix watcher loop quit and channel shouldDrop logic (#23402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-14 09:54:28 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
MrPresent-Han
77c9e33e70
support dml channel balancer on datacoord (#22324) (#22377) (#22692) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-03-20 10:01:56 +08:00
zhenshan.cao
e768437681
Correct usage of Timer and Ticker (#22228) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-23 18:59:45 +08:00
aoiasd
148a024e05
Add tickle for datacoord watch event (#21193) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:20:34 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
Ten Thousand Leaves
0700e56008
Increase MaxWatchDuration and make it configurable (#20884) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-29 17:19:14 +08:00
bigsheeper
fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config (#20504) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-12 21:09:04 +08:00
bigsheeper
cd19d99ad7
Add channel level checkpoint (#20350) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-10 22:13:04 +08:00
SimFG
c604d2f933
Make param names clearer in the policy.go (#20136) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-31 11:35:34 +08:00
Xiaofan
192151bc11
Refine Data Log (#19996) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-25 19:31:30 +08:00
SimFG
a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
XuanYang-cn
28e949cf31
Add GetDataVChanPositions (#19657) 
The original GetVChanPositions are specialized for QueryCoord,
the recovery info needed by DataNode shouldn't take `indexed` state
into account.

This PR splits GetVChanPositions into GetDataVChanPositions and
GetQueryVChanPositions.

See also: #19653

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-10-11 16:51:22 +08:00
congqixia
d64f318a80
Add collection schema in watch channel info (#19663) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-10 01:37:21 +08:00
zhenshan.cao
c15b880f0e
Remove redundant logs (#19412) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-09-26 15:40:53 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
Jiquan Long
641afae22f
Watch channels with start positions (#18744) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-08-22 14:42:52 +08:00
congqixia
28aadc988d
Use vchannel name in datanode subName (#18519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-03 17:10:36 +08:00
SimFG
50d5d04552
Refine some codes about the datanode (#17720) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-23 09:58:14 +08:00
Enwei Jiao
746aeea35b
fix typo (#17668) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-21 18:54:13 +08:00
Ten Thousand Leaves
51a9e54b21
Fix a bug that causes infinite loop (#17550) 
This will cause AddNode() to keep its lock forever so DataCoord would just hang

issue: #17366
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 20:50:10 +08:00
XuanYang-cn
e6225d923b
Cancel timers while adding new timer (#17511) 
See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-13 14:46:08 +08:00