test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-03 12:29:36 +08:00
Code Issues Actions 13 Packages Projects Releases Wiki Activity
19,059 Commits 96 Branches 105 Tags 184 MiB
b111f3b110
Commit Graph

924 Commits

Author SHA1 Message Date
congqixia
233bf90c55
Add SyncManager to replace flush manager (#27873) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-31 02:30:16 +08:00
congqixia
98e2aad752
Refine datanode metacache and implement CoW (#27985) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-28 11:12:11 +08:00
smellthemoon
403c6680cc
Add error description (#27959) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-27 10:10:13 +08:00
Filip Haltmayer
6b1a106a31
Moving etcd client into session (#27069) 
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
 2023-10-27 07:36:12 +08:00
yah01
f79c7370f4
Fix panic while flushing dropped/compacted segment (#27927) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-25 22:10:13 +08:00
yihao.dai
b9d5ef3599
Fix datanode ttNode goroutine leak (#27878) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-24 18:22:10 +08:00
aoiasd
9091a27832
Add meta cache to datanode for L0 Delta (#27768) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-23 19:42:10 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
smellthemoon
4b0ec156b3
Set channel work pool size in datanode (#27728) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-19 08:28:08 +08:00
XuanYang-cn
7358c3527b
Add iterators (#27643) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-18 19:34:08 +08:00
jaime
e386a62fae
Remove recollect segment stats during starting datacoord (#27410) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:26:09 +08:00
jaime
ec1fe3549e
Add a stop hook to clean session (#27564) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:24:10 +08:00
congqixia
82b2edc4bd
Replace manual composed grpc call with Broker methods (#27676) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-13 09:55:34 +08:00
yah01
be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
aoiasd
ec830e1240
fix use same logidx in different field (#27492) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-11 18:49:33 +08:00
congqixia
cbb350c552
Add broker for datanode grpc operations (#27631) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-11 17:03:34 +08:00
yah01
2df9908c6a
Map old error code to new error (#27616) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 12:43:33 +08:00
jaime
86ed2fd51f
[skip e2e] Fix unstable ut (#27618) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-11 07:21:32 +08:00
XuanYang-cn
294ff74ca5
Add new compaction types (#27608) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-11 06:49:32 +08:00
XuanYang-cn
56c94cdfa7
Add channel manager in DataNode (#27308) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-08 21:37:33 +08:00
congqixia
5d558623fe
Add revive sub-lints and fix existing problems (#27495) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:53:38 +08:00
congqixia
80eb5434d3
Fix var-naming caused by old PRs (#27501) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 14:43:32 +08:00
yah01
8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
Jiquan Long
0f14d18201
Optimize the codec code of session (#27360) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-10-01 10:33:30 +08:00
XuanYang-cn
5c5f9aa05e
Enhance newDataSyncService (#27277) 
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
  - config.maxQueueLength, config.maxParallelish

See also: #27207

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-27 11:07:25 +08:00
yah01
6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
wayblink
7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface (#27186) 
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-09-26 09:57:25 +08:00
XuanYang-cn
676024ff38
Add rpc to notify channel operation (#27172) 
- Add NotifyChannelOperation
- Add CheckChannelOperationProgress

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-25 14:05:27 +08:00
congqixia
1d76565894
Add metrics for garbage collection (#27303) 
Also fix second metrics usage in compaction

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 18:47:25 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
XuanYang-cn
916528f51a
Add initCtx in getDataSyncService (#27199) 
Passing initCtx to all IO funcs in newDataSyncService,
so when ctx.Canceled, newDataSyncService would return.

See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-20 16:03:23 +08:00
yah01
b4f86ea55e
Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
yah01
338848fbc3
Remove unused code and dup imports (#27228) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:55:23 +08:00
Xiaofan
6635398a6d
Fix Bin log concurrency by adding a pool (#27189) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-19 18:05:22 +08:00
XuanYang-cn
09505ea78e
Move etcd watch related code into eventmanager (#27192) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 15:27:38 +08:00
XuanYang-cn
10116b85ac
Release before remove when releasing flowgraphs (#27191) 
GetAndRemove removes the fg from manager immediately,
while the flowgraph is still releasing. This PR will remove
the fg from flowgraphManager AFTER flowgraphs released.

- Add Remove for ConcurrentMap
- Move collections() into flowgraph manager

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-19 11:53:22 +08:00
yihao.dai
4b2802033d
Fix datanode panic due to concurrent compaction and delete processing (#27167) 
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 19:35:29 +08:00
yah01
a6b98740b7
Replace all status with only error string (#27125) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-18 14:05:28 +08:00
yihao.dai
c162c6a4c8
Increase FlushChannel retry times (#27140) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-16 00:41:20 +08:00
yah01
168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
yihao.dai
dd2cb1d44a
Flush by flush channels (#26859) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-12 21:07:19 +08:00
yah01
00c65fa0d7
Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
XuanYang-cn
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
yiwangdr
337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
bjzhjing
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
congqixia
fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
XuanYang-cn
b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
yihao.dai
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
yihao.dai
7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia
e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
XuanYang-cn
8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
smellthemoon
6069a7d42b
Log out memory in MB (#26262) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-30 17:13:00 +08:00
congqixia
cc584551fb
Remove unused hash value setting in DDNode (#26575) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-28 11:30:26 +08:00
SimFG
b9bc6681ae
Use atomic.bool for the isWatchFailed param in the tickler (#26558) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-23 13:22:27 +08:00
XuanYang-cn
27691e843c
Fix channel checkpoint stuck (#26534) 
See also: #23621

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-22 18:10:22 +08:00
XuanYang-cn
08fd28b30b
Only do gracefully stop when DN Stop (#26399) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-21 11:16:20 +08:00
SimFG
3be4ac4022
Fix datanode/datacoord continuous restart (#26470) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-20 21:20:24 +08:00
Enwei Jiao
533f0ddf6d
Add amazonlinux image, prepare to remove Centos7 image, remove openblas image (#26368) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-18 18:32:19 +08:00
congqixia
fbb5d32cb6
Make write binlog in parallel (#26325) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 15:44:18 +08:00
congqixia
f371ec0316
Add data sync service close log for watch stuck debugging (#26332) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 10:57:32 +08:00
Enwei Jiao
78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-16 10:29:33 +08:00
congqixia
f8bcf60e55
Adjust some confusing Warning log to INFO (#26356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 17:49:33 +08:00
wayblink
21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact (#26203) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-08 21:17:21 +08:00
congqixia
b9850ce5c0
Fix copylocks linter errors (#26217) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-08 20:41:15 +08:00
MrPresent-Han
b84f5c560a
refine log out for datanode(#25763) (#26195) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-08 16:49:08 +08:00
MrPresent-Han
fb933fe64d
fix sync memory policy lose effect when inserting too many partitions(#25763) (#26154) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-07 09:59:07 +08:00
XuanYang-cn
830f06783a
Reduce log volumes of DC&DN (#26060) 
See also: #26057

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-02 17:13:11 +08:00
XuanYang-cn
84253f255e
Fix datanode graceful stop panic (#25932) 
See also: #25925

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-07-28 10:11:08 +08:00
groot
a6808e6484
Fix bulkinsert row count error (#25869) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-25 11:29:00 +08:00
cai.zhang
779d677eb2
Remove lock for compaction sync segments (#25711) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-24 17:25:00 +08:00
jaime
8e5d6193f9
Add a timeout config for bulkinsert request (#25789) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-07-24 14:23:00 +08:00
congqixia
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:23:01 +08:00
congqixia
31b40b3ccc
Use typeutil.ConcurrentMap instead of sync.Map in fg manager (#25835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:21:07 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yiwangdr
b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
wayblink
53b4b70316
Reduce log frequency in timetick_sender (#25532) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 14:38:29 +08:00
wei liu
012fd1152a
fix datacoord consume datanode tt metrics (#25524) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:26:29 +08:00
congqixia
dbfade77c5
Retry on SyncSegment failure (#25540) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-13 14:12:29 +08:00
wayblink
ed3e4b0bc5
Fix unstable ut in timetick_sender_test (#25539) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 10:04:29 +08:00
MrPresent-Han
ea3817fbbc
refinement towards sync_cp_lag_too_behind_policy to avoid submit syncTasks too frequently(#25441) (#25442) (#25500) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-12 20:22:29 +08:00
SimFG
f9e2d00f91
Prevent exclusive consumer exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
wayblink
fc12d3997c
Rename newTimeTickManager to newTimeTickSender (#25415) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-12 14:56:28 +08:00
groot
96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
wei liu
71d99d4ec2
fix unexpected metrics value cause by wrong time unit (#25240) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 08:48:27 +08:00
Xiaofan
63b6a4a639
use single instance ppol (#25159) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-28 14:54:45 +08:00
xige-16
33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
yiwangdr
c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
congqixia
736916222b
Stop retry lazy load stats after segment released (#25054) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 19:20:43 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan
e8911ebda7
Add retry time when lazy load BF (#25096) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-25 11:32:43 +08:00
Xiaofan
72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
aoiasd
7f5de9aab3
add read lock for getSegment (#25036) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-06-21 14:06:42 +08:00
PowderLi
3f4356df10
fix the spelling of field (#25008) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-06-21 14:00:42 +08:00
yihao.dai
6168a55c04
Lower the log frequency when sync tasks are skipped (#25006) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-20 17:56:41 +08:00
jaime
b75dcf90c0
Fix Flush hang after SyncSegments timeout (#24953) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-06-20 10:20:41 +08:00
Xiaofan
af1d84e5e1
Support sync when cp is lagged (#24989) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-20 10:00:41 +08:00
Xiaofan
f57fe6d70b
Add compaction log (#24976) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-19 14:18:41 +08:00
yihao.dai
c73219a54d
Limit the number of concurrent sync tasks and allow only one sync task for the same segment (#24881) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-16 14:14:39 +08:00
congqixia
d186f35895
Fix TestChannelMetaMockSkipBFLoad/transient_error not stable (#24928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-16 09:50:44 +08:00
wayblink
bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
groot
5c3b744b0c
Fix potential crash bug of bulkinsert (#24763) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-06-13 10:22:38 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01
ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
congqixia
31880ab427
Use conc.PoolOption instead of ants.Option (#24585) 
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-01 15:52:31 +08:00
wayblink
5958b42941
Add retry on getChannelLatestMsgID (#24518) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-30 20:03:28 +08:00
aoiasd
c84bdcea49
merge stats log when segment flushing or compacting (#23570) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-05-29 10:21:28 +08:00
Xiaofan
49f75e5e11
Make flow graph retry longer (#24355) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-05-25 09:53:30 +08:00
congqixia
03297a7ef7
Fix event manager not closed after datanode stop (#24206) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-18 11:43:23 +08:00
yah01
99cc24b283
Fix compaction doesn't support JSON data (#24151) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-17 19:09:22 +08:00
congqixia
73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
congqixia
084424d636
Make data node start only once (#24031) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-11 13:51:20 +08:00
congqixia
345855c984
Make datanode load statslog lazy if SkipBFStatsLog is true (#23877) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-06 15:12:39 +08:00
MrPresent-Han
b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
congqixia
796f9355ee
Fix global rateCollector init multiple times (#23703) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-26 10:16:36 +08:00
wei liu
4fb8919a97
support collection level rate limit (#22767) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-25 15:54:35 +08:00
zhenshan.cao
03ec804e68
Change default shard number to 1 (#23586) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-21 07:08:32 +08:00
congqixia
4a6c2b78a7
Fix multiple transferNewSegment and removeSegment causes panic (#23564) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-20 13:48:31 +08:00
Enwei Jiao
967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
congqixia
4fe363c4b2
Add KNN cgo pool (#23526) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 23:56:31 +08:00
congqixia
4508786819
Fix test_watch_channel unit test unstable (#23507) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 10:00:31 +08:00
wei liu
cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
wayblink
b23723535c
Add log in AddImportSegment.getChannelLatestMsgID (#23480) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-04-18 17:50:31 +08:00
zhenshan.cao
4a32b842e8
Improve the check logic of channel remove (#23473) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-18 02:58:30 +08:00
aoiasd
695cc769df
Fix watch event timeout when IO Pool busy (#23293) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 11:02:30 +08:00
aoiasd
f603456b72
remove segment from datanode meta when segment dropped (#23398) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 10:58:30 +08:00
congqixia
ba84f52119
Fix watcher loop quit and channel shouldDrop logic (#23402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-14 09:54:28 +08:00
cai.zhang
43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
XuanYang-cn
b90fa5f459
Refine codes of datanode buffer (#23168) 
This PR refines deltabuffer of datanode:
- Add last sync time for compacted segment, see also: #23210
- Ensure all deltabuffermanager handles all delete related operations
- Change usedMemory to atomic.Int64
- Remove allocator in delete buffer

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-04-10 18:42:30 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
aoiasd
1a485044a9
fix NumConsumers metric not work (#23167) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-05 08:18:28 +08:00
smellthemoon
ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
XuanYang-cn
be0827417a
Fix buffer calculate memorysize negative (#23152) 
Along with the following small patches:
- And fix logging dup segmentIDs for delete buffer status
- Add String method for PriorityQueue to debug

See also: #23105

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-31 18:30:23 +08:00
Jiquan Long
0f2c3ac832
Optimize log (#23121) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-30 10:44:22 +08:00
Enwei Jiao
50dab2d394
Add tracing for retrieve (#23033) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-29 18:10:02 +08:00
XuanYang-cn
9691315c21
Fix flush deltabuf with segmentID=0 (#23059) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-28 19:14:01 +08:00
aoiasd
5d172d0f4f
Refine errors of datanode (#22852) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-28 19:04:00 +08:00
zhenshan.cao
e3c3c949c4
Fix datarace of flow_graph_manager_test.go in package datanode (#23043) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-03-28 10:42:00 +08:00
yihao.dai
b21f03682a
Fix flowgraph manager test (#23009) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-27 18:00:05 +08:00
congqixia
4008ee6d60
Remove datanode ddNode forward delta logic (#23005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-27 14:18:00 +08:00
yah01
081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Xiaofan
5a914aedf3
Make Datacoord TT to be configurable (#22956) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-03-24 22:24:04 +08:00
XuanYang-cn
da58ba6a49
Fix endless appending sids (#22977) 
See also: #22976

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-24 18:00:03 +08:00
congqixia
8f847884c2
Fix misc copylocks and remove exclude rule (#22962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-24 11:17:58 +08:00
XuanYang-cn
93bc805933
Enhance ID allocator in DataNode (#22905) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-23 19:43:57 +08:00
yihao.dai
f2ffb5db26
Use buffer size at memory sync policy (#22825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-21 21:37:56 +08:00
SimFG
4a90490a67
Fix the segment not found error (#22772) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-17 17:27:56 +08:00