test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-01 11:29:48 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
9,360 Commits 94 Branches 105 Tags 184 MiB
3dac2611b6
Commit Graph

92 Commits

Author SHA1 Message Date
Cai Yudong
3dac2611b6
Remove redundant variable load (#9452) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-08 14:49:08 +08:00
XuanYang-cn
e7e37448d2
[skip ci]Add comment for flowgraph insertBufferNode (#9361) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 23:30:21 +08:00
XuanYang-cn
7b14b0e63d
[skip ci]Add comment on readBinary (#9355) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:28:11 +08:00
XuanYang-cn
65a76b9bb5
[skip ci]Add comment in insertbuffernode (#9354) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:20:05 +08:00
godchen
83228c35ad
Add flowgraph insert buffer node error check (#9310) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-06 21:06:23 +08:00
congqixia
10a6f7f146
Defer buf removal from insertBuf after SaveBinlog succeeds (#8930) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-30 17:42:11 +08:00
XuanYang-cn
b150f57927
No buffer also means flushed (#8686) 
Resolves: #8684

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-28 11:18:03 +08:00
zhenshan.cao
b71bffe26f
Replace ChannelID with ShardName in DML Request (#8539) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-27 10:01:59 +08:00
XuanYang-cn
ebd894d8b7
Refactor buffer size calculation (#8565) 
This PR changes the datanode insertBufSize config
from 32000 number of rows to 16MB

Resolves: #7741

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:55:59 +08:00
Cai Yudong
b2e5627061
Rename insertMsg to flowGraphMsg in datanode flow graph (#8546) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-26 10:43:57 +08:00
XuanYang-cn
faeb1bf50f
Add comment on exposed struct (#8348) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 20:15:54 +08:00
XuanYang-cn
2b82482679
Fix golint and ruleguard (#8347) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 19:14:01 +08:00
XuanYang-cn
a830c28bd8
Fix golint in datanode (#8345) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 19:05:55 +08:00
XuanYang-cn
d816a81ee3
Make FlushSegments async (#8159) 
Resolves: #8122

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 16:03:54 +08:00
congqixia
bdfb8768d4
Enable gosec (#8202) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-18 14:45:50 +08:00
XuanYang-cn
46c72c57c2
Add BufferData in insertbufferNode (#8138) 
Optimize bufferInsertMsg param to avoid a second transfer
of endposition

See also: #8058, #7741

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-18 14:25:50 +08:00
Cai Yudong
d10c6abc9e
Apply for msgstream from pool when creating collection (#7738) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 09:13:50 +08:00
XuanYang-cn
6c434b4801
Improve codes in insertbuffernode (#8054) 
Add error handling in insertbuffernode

See also: #7624, #7684

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-17 16:27:56 +08:00
XuanYang-cn
82d4d19337
Remove not in use segmentCheckPoints (#8039) 
Resolves: #8037

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-17 16:25:55 +08:00
dragondriver
a872a4357c
Add more test cases for proxy implementation (#7685) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-11 11:36:22 +08:00
Cai Yudong
4e23ed9a18
Revert apply physical channels from pool (d5f7e358 and 3b960969) (#7687) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-10 13:00:00 +08:00
congqixia
69794fd32d
Refactor insertBufferNode and add unit tests (#7621) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-09 15:00:00 +08:00
Cai Yudong
3b9609692b
Apply for msgstream from pool when creating collection (#7377) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 18:35:58 +08:00
Cai Yudong
f4b53df727
Revert "Apply for msgstream from pool when creating collection (#7323)" (#7376) 
This reverts commit e6de86a433.
 2021-08-31 11:07:58 +08:00
Cai Yudong
e6de86a433
Apply for msgstream from pool when creating collection (#7323) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 09:47:58 +08:00
congqixia
65f168eac1
Fix datanode corner cases (#7336) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-30 10:03:58 +08:00
congqixia
daae05641a
Construct bloom filter when inserting (#7268) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-25 10:29:52 +08:00
sunby
4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
XuanYang-cn
9d81a90402
Fix mem leak of flush cache in DataNode (#6862) 
Resolves: #6858

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-11 14:24:09 +08:00
XuanYang-cn
47df4a2e57
Fix DataNode BackGroudGC problem (#6551) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-03 09:59:24 +08:00
godchen
db94d7771f
Read vector from disk (#6707) 
* Read vector from disk

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix git action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix test error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix caculate error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change var name

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix len error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused code

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change bytes to float method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change float to bytes method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-07-24 09:25:22 +08:00
sunby
7f2d7c1da3
Rename dataservice to datacoordinator (#5912) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-21 18:22:13 +08:00
XuanYang-cn
441300140d
Tidy replica (#5920) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-21 16:00:22 +08:00
sunby
33f5225968
Use physical channel to init msg stream (#5815) 
* Use physical channel to init msg stream

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix search collection error check (#1)

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: bigsheeper <954206947@qq.com>
 2021-06-18 16:02:05 +08:00
XuanYang-cn
18eb27aa7a
Tidy replica in datanode (#5805) 
- Remove not using functions
- Remove `initNode` logic in ParamTable
- Delete collection.go and collection_test.go

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-16 19:03:57 +08:00
XuanYang-cn
5bcb94294c Set start positions in datanode (#5757) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:36:36 +08:00
XuanYang-cn
dea0a4e522 Idempotent of FlushSegment (#5746) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:36:36 +08:00
XuanYang-cn
ab2fd34a2f fix double messages bug in datanode recovery (#5733) 
* fix double messages bug in datanode recovery

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* remove debug log

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:14:29 +08:00
XuanYang-cn
46699c4c2b fix datanode dropcollection error (#5690) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:06:13 +08:00
sunby
189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
neza2017
74c73fa75f fix datanode (#5635) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
7064937dc3 fix insertBuffer node (#5632) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
0c362a8831 fix bug and add unit test for insert buffer (#5630) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
3c70675313 data node, save binlog to minIO, and let data service save these meta (#5618) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
0a82c6381f add auto flush for data node (#5609) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
XuanYang-cn
e96d39bf6a refactor datanode flowgraph (#5538) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
6c02ae4eb1 Remove Old msgposition logic (#5487) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
8657251f41 DataNode stores segment msgpack positions (#5472) 
* DataNode stores segment msgpack positions

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix typo caused deadlock

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Use MsgPack EndPositions for flush pos

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
7a8b34ee9f Remove not using code in DataNode (#5476) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
55eac3ee7f Complete Flush procedure in DataNode (#5454) 
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00