test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
14,137 Commits 96 Branches 105 Tags 185 MiB
8255b713ac
Commit Graph

438 Commits

Author SHA1 Message Date
Xiangyu Wang
08be6888d5
Improve import statement (#13733) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-20 14:33:47 +08:00
congqixia
2de5cf7762
[skip e2e] Improve log in flush manager (#13689) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-20 12:22:43 +08:00
congqixia
d4afc812d1
Normalize DataNode term in errors.go (#13688) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-20 12:20:50 +08:00
congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs (#13606) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-19 20:00:42 +08:00
JackLCL
20c956b675
[skip e2e]Fix error log output format in segment_replica.go (#13676) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-19 12:32:42 +08:00
godchen
154f11e3a1
[skip e2e]Fix redundant conversion (#13648) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 22:28:50 +08:00
godchen
e191b23249
[skip e2e]Fix error string (#13647) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:22:42 +08:00
godchen
fefeaa2e1d
[skip e2e]Fix compaction executor typo (#13646) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-17 19:20:46 +08:00
cai.zhang
9f23fc7f2a
Register the service when the component state is healthy (#13248) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2021-12-15 11:47:10 +08:00
Bingyi Sun
0c39e0370c
Move segment statistics update to datanode tt loop (#13233) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-15 10:53:16 +08:00
JackLCL
30afcf2311
[skip ci]Fix error log output format in segment_replica.go (#13350) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-14 16:51:09 +08:00
JackLCL
5f02a3738a
[skip ci]Fix error log output format in meta_service.go (#13349) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-14 16:43:25 +08:00
Xiangyu Wang
405b3cd932
Use the same lint rules with golint in revive (#13324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-12-14 15:31:07 +08:00
congqixia
8f87cc7ef1
Fix JoinIDPath logic problem (#13039) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-09 12:05:04 +08:00
XuanYang-cn
dbf757e08b
Change genKey into JoinIDPath (#12959) 
1. Simplify genKey(), make genKey a rpc call
2. Unify the usage of JoinIDPath

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-09 11:09:06 +08:00
XuanYang-cn
a9a332dbcf
Fix empty segment compact and load (#12710) 
1. Fix compaction save empty segment bug
2. Fix load empty segment bug
3. Add UT for dmlchannels to 100%

Resolves: #12450

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-12-09 11:03:08 +08:00
congqixia
1195f441b2
Update DataNode state before stop (#12946) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-08 14:19:04 +08:00
JackLCL
73b63b9a02
[skip ci]Fix error log output format in segment_replica.go (#12929) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-08 13:55:04 +08:00
JackLCL
a6ce7d7f53
[skip ci]Fix error log output format in meta_service.go (#12928) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-08 13:51:09 +08:00
JackLCL
e51ef53632
[skip ci]Fix error log output format in flush_manager.go (#12857) 
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
 2021-12-07 14:38:15 +08:00
congqixia
b20a238c38
Fix newDataSyncService panicking with normal segment has no dml pos (#12771) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-06 16:21:35 +08:00
congqixia
58c5b6093d
Fix close10000 ut has chances to fail (#12740) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-06 09:59:35 +08:00
Bingyi Sun
8bbaae04ff
Decrease compaction parallel (#12707) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-03 18:57:33 +08:00
congqixia
ffd05eb140
Fix deadlock in mergedTimeTickSender (#12697) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-03 16:39:33 +08:00
edward.zeng
45cbec1f7b
[skip ci] Refine datanode binlog io comment (#12638) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-02 19:11:32 +08:00
edward.zeng
c2b73e1b55
[skip ci] Refine data node flow graph comment (#12637) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-02 18:59:33 +08:00
congqixia
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode (#12563) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-02 16:39:33 +08:00
godchen
5e3e3b3b41
Remove grpc returned error (#12537) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-01 22:17:46 +08:00
edward.zeng
271aaa7537
[skip ci] Refine data node param table comment (#12560) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-01 19:31:36 +08:00
congqixia
e813867652
Implement Drop mode for flush manager (#12469) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-01 10:11:39 +08:00
godchen
f31ed089b5
Add timeout for reader next (#12308) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-26 22:45:24 +08:00
XuanYang-cn
48b45d82e5
Add ut for binlog_io to 100 coverage (#12283) 
Make DN ut coverage upto 90%
Resolves: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-26 17:43:17 +08:00
dragondriver
b6d0150dc5
Fix health check when disconnect to etcd (#12292) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-26 11:39:16 +08:00
bigsheeper
e245ba87a9
Add excluded segments for dropped segemnts (#12247) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-11-25 14:25:15 +08:00
XuanYang-cn
8a8ebed23d
Fix DN GC bug (#12258) 
Fixes: #12249

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-25 09:43:15 +08:00
congqixia
59d19384dc
Fix flush manager injection logic with multiple segments (#12260) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-24 18:41:16 +08:00
godchen
e3f49858c5
Add subscription prefix (#12251) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-24 17:47:15 +08:00
dragondriver
6c4c0ef6b5
Add more deploy metrics (#12248) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-24 15:43:15 +08:00
XuanYang-cn
e72c5b9351
Filter dropped segments in DN (#12242) 
See also: #12131, #12230

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-24 14:11:15 +08:00
XuanYang-cn
6bf0c4326d
Fix compaction generates wrong insert binlogs (#12204) 
This PR
- changes Get() to GetRow() in BinlogIterator
- removes Get() and Length() of FieldData
- adds ut in BinlogIterator for floatvector and binaryvector
- fix a log bug

Fixes: #12146

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-23 17:09:15 +08:00
congqixia
4121e31df1
Send SIGINT to runner goroutine after etcd disconnects (#12163) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-22 16:23:17 +08:00
godchen
51b353b52c
Add msgstream reader (#12130) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-19 15:57:12 +08:00
congqixia
7e91bcf115
Normalize datanode dminput node log (#12128) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-19 14:21:13 +08:00
dragondriver
ee0f753f7a
Fix datarace between GetComponentStates and Register (#11935) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-19 13:57:12 +08:00
Bingyi Sun
038f42851e
Add more log for debug (#11893) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-18 10:37:41 +08:00
XuanYang-cn
7dbb4d10e5
Fix DeltaLogs empty yet appending a deltaInfo bug (#11984) 
- Add more logs for startpositions

Fixes: #11970

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-17 17:57:11 +08:00
congqixia
5e550a8793
Add close for flushManager (#11874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-17 10:39:15 +08:00
XuanYang-cn
9bd3664672
Fix compaction bug (#11816) 
This PR fixes:
- timeout in compaction
- BinlogIterator using rawID as PKField bug

See also: #11757

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-17 10:07:13 +08:00
congqixia
5edbb82610
Add session revoke (#11908) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-16 22:31:14 +08:00
Xiangyu Wang
12f50cb22c
[skip ci]Update OWNERS files (#11898) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-16 15:41:11 +08:00
congqixia
fa6314393a
Fix compactor loop closure captured (#11814) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-15 19:44:10 +08:00
congqixia
579416e3d3
Apply retry for flush manager execution (#11808) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-15 17:19:10 +08:00
XuanYang-cn
f3852c1db0
Add more specific logs (#11724) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-15 15:25:09 +08:00
Bingyi Sun
5c8978e5d6
Fix drop segment bug (#11709) 
issue: #11708
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-12 17:31:11 +08:00
XuanYang-cn
c4bb73dc74
Complete drop collection procedure in DN (#11552) 
drop, cancel compaction, flush all buffers and release fg

See also: #11426

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-11 20:56:49 +08:00
godchen
5357e301f4
Fix retrieve search error (#11501) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-11 00:54:45 +08:00
godchen
14d2891a69
[skip ci]Fix download typo (#11577) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-10 20:11:35 +08:00
godchen
0fc1e12faf
[skip ci]Fix download typo (#11575) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-10 20:09:33 +08:00
godchen
038d0f1adf
[skip ci]Fix download typo (#11573) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-10 20:07:39 +08:00
congqixia
4c37e94a39
Add type assertion for compactTask (#11468) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-10 19:16:40 +08:00
Xiaofan
5f08b67156
Move node.yaml to component.yaml (#11363) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-11-10 14:27:37 +08:00
godchen
863f1bb34e
Fix multi delete data not effect (#11422) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-09 15:01:17 +08:00
Xiangyu Wang
0eb71f7ee5
Logging traceID in process of processing delete request (#11374) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-11-09 11:01:02 +08:00
godchen
639800241d
Make flow graph for delta channel (#11379) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-09 09:27:04 +08:00
XuanYang-cn
3cd0584a04
Add compaction in datanode (#11353) 
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-08 19:49:07 +08:00
congqixia
3ac7d7a9de
Add detailed comments for dataSyncService in DataNode (#11285) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-05 22:59:04 +08:00
sunby
c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
congqixia
0e5931a3da
Add mergedTimetickSender for datanode (#11297) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-05 14:59:32 +08:00
godchen
9186d5527e
Forward delete msg (#11210) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-05 11:59:02 +08:00
XuanYang-cn
8863c01ff7
Make DN fail faster when watching an invalid chan (#11102) 
See also: #11098

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-04 15:56:28 +08:00
congqixia
0ef95c5df1
Add timeticker logger for insert buffer node (#11216) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-04 15:40:14 +08:00
godchen
b63a9521c2
Make gen msg private (#10115) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-04 15:36:19 +08:00
godchen
a0a3a889e7
Add common endian for global use (#11092) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-02 18:16:32 +08:00
XuanYang-cn
225b2c66a0
Fix datanode ut using register bug (#10885) 
Resolves: #10881
See also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-01 11:01:49 +08:00
XuanYang-cn
1e3fc5076f
Change etcdkv clientv3 into MetaKv interface (#10903) 
This pr:
- changed etcdkv clientv3 into MetaKv interface
- replaced fmt.Sprintf with path.Join for kv key

See also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-01 10:19:55 +08:00
Xiaofan
2df2d488e1
Fix Unit test fatal fail (#10933) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-30 10:24:38 +08:00
congqixia
756be81bce
Add injectFlush method for flushManager interface (#10659) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-28 19:16:31 +08:00
Xiaofan
6fddcb3925
Fix Session checker for proxy (#10737) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-27 21:58:33 +08:00
XuanYang-cn
42ce9b18d1
Add methods in segment replica (#10582) 
see also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-27 17:14:28 +08:00
congqixia
169af67eb1
Fix segments not removed from flushingCache (#10738) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-27 16:06:24 +08:00
cai.zhang
aff5cd9598
Supprot to read the configuration of minio from environment variables (#10534) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-10-26 15:10:03 +08:00
godchen
095a2378ca
Add datanode ut (#10433) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-25 22:52:27 +08:00
congqixia
ae2a301662
Add injection logic for FlushManager (#10580) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-25 20:17:34 +08:00
XuanYang-cn
21d98c2825
Add binlog io for compaction (#10536) 
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 20:13:51 +08:00
godchen
d2e0943f21
Fix delete node flush error (#10549) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-25 18:49:20 +08:00
congqixia
2713988ff2
Merge auto manual flush with same segment id (#10550) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-25 18:03:42 +08:00
congqixia
bd90a3831c
Allow flushed segments trigger flushe process (#10539) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-25 18:01:50 +08:00
XuanYang-cn
cd06f50645
Remove schema in delete codec (#10517) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-24 09:59:10 +08:00
godchen
31393d8801
[skip ci]Fix datanode comment typo error (#10502) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-23 16:57:35 +08:00
godchen
ffc0c07610
Change delete data primary key to int64 (#10438) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 15:37:12 +08:00
godchen
d6bf0c928f
Get stats delta path from datacoord (#10362) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 14:31:13 +08:00
congqixia
7ce7cb7a5e
Add a timed flush trigger mechanism (#10197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 15:02:36 +08:00
godchen
6279fb817a
Fix stats log load with prefix error (#10216) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-19 20:50:35 +08:00
godchen
8daeb0d519
Change bloom filter use pk (#10193) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-19 20:18:47 +08:00
sunby
4031b8746f
Fix panic when flushing segment (#10182) 
issue: #10180
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-19 19:06:37 +08:00
congqixia
ed4e516ce7
Save stats and delta logs from DataNode (#10159) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 14:34:35 +08:00
congqixia
5737e0075f
Apply flush manager logic in datanode (#10142) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 11:04:34 +08:00
congqixia
3711fcd82f
Add more info for Delta log compaction (#10083) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-18 18:24:45 +08:00
congqixia
0438ed2bfe
Use single signal chan to notify flush (#10075) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-18 12:34:34 +08:00
congqixia
90f88c6559
Add flush manager structure (#9986) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-18 10:38:35 +08:00