Commit Graph

18686 Commits

Author SHA1 Message Date
yah01
3ea0129eb3
enhance: improve the error messages and logs (#28684)
- better name for log fields
- make the error and log consistent

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-24 15:08:24 +08:00
XuanYang-cn
24616e7838
fix: Remove ignored files in path (#28696)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-24 15:04:26 +08:00
smellthemoon
79c0edb1d8
enhance:Remove msgbase unnecessary assignments (#28511)
remove some unnecessary assignments, for the reason that
commonpbutil.NewMsgBase has default value.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-24 15:02:39 +08:00
nico
824e278219
test: update test cases for new error code&msg (#28706)
issue: #23075

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-11-24 14:52:25 +08:00
cai.zhang
6f7a9264d5
enhance: Handle knowhere error for creare diskann index (#28690)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-24 11:58:23 +08:00
wayblink
5aedbd0af4
fix: Repair integration test framework (#28308)
#28307

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-11-24 10:26:23 +08:00
Bingyi Sun
2843664a20
fix: channel name can not contains '-partition-' (#28689)
issue: https://github.com/milvus-io/milvus/issues/28675

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-24 10:14:24 +08:00
sre-ci-robot
3199da3b77
[automated] Update Builder gpu image changes (#28702)
Update Builder gpu image changes
See changes:
90997a5ddb
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-24 10:00:24 +08:00
sammy.huang
fa66c2829d
enhance: [skip e2e] add auto prefix for the PR from bot (#28705)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2023-11-24 09:42:23 +08:00
wayblink
53646032d0
fix: rootcoord_collection_num metric is twice the real value (#28666)
#28665

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-11-23 20:42:23 +08:00
PowderLi
90997a5ddb
enhance: update the base image: milvusdb/milvus-env (#28505)
issue: #28683
to avoid downloading installation packages in CI workload install vcpkg
and install some packages in advance

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-23 19:58:25 +08:00
zhagnlu
0d9d098186
enhance: Add precheck when chunk manager init (#28330)
#28329

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:56:32 +08:00
wayblink
da339535d5
enhance: Merge flowgraph goroutines into 1 (#28654)
/kind enhancement
#24826

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-11-23 19:52:25 +08:00
wei liu
4b8d3eaa41
enhance: Increase heartbeat timeout (#28648)
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-23 19:50:24 +08:00
congqixia
098c1c926d
fix: Add IndexList check for load segment request (#28601)
See also #28022 #28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-23 19:42:30 +08:00
congqixia
a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661)
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-23 19:34:23 +08:00
zhagnlu
e9e9b30e8e
enhance: Disable stdout buffer (#28391)
#28390

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:26:25 +08:00
zhagnlu
bc43c5cf34
fix: Panic if minio port is empty (#28461)
#28460

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-23 19:24:25 +08:00
XuanYang-cn
b1f15fa0e8
fix: Ease the log level when sync task done (#28678)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 19:18:28 +08:00
XuanYang-cn
9b371067d2
feat: Add Compaction views and triggers (#27906)
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager

See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 17:30:25 +08:00
Bingyi Sun
4fedff6d47
feat: integrate storage v2 into the write path (#28440)
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:26:24 +08:00
Bingyi Sun
e5ce385ffd
enhance: remove -inl.h files (#28674)
issue: https://github.com/milvus-io/milvus/issues/28673
Move template implementations from -inl.h to .cpp file and make explicit
instantiation

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:20:25 +08:00
cai.zhang
31f442915b
fix: Fix bug for parsing expression that include quotes (#28416)
issue: #28365 
Fix bug for parsing error when a string enclosed in single quotes in an
expression contains multiple double quotes.
such as:
```
expr = "tag == '\"blue\"'"
```

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-23 17:18:32 +08:00
SimFG
de13865769
enhance: Add load/release partitions to replicate msg stream (#28399)
/kind improvement
issue: #25655

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-23 15:38:24 +08:00
yah01
c96d07682e
enhance: enhance the accuracy of memory usage (#28554)
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage

resolve #28553

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-23 15:12:23 +08:00
XuanYang-cn
55800ade84
fix: Remove logging extra segmentIDs (#28662)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 14:40:23 +08:00
Enwei Jiao
3c3a57cb92
fix: Fix kafka config type error (#28642)
issue https://github.com/milvus-io/milvus/issues/28588

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-23 14:36:22 +08:00
congqixia
af18aa709b
fix: Remove not needed BlockAll call in SyncSegments (#28632)
See also #28628
Previous compaction task blocked the segment sync task and may block the
flowgraph when sync task is generated by auto sync policy This
`BlockAll` call will block forever and cause whole fg stuck

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-22 14:52:23 +08:00
sammy.huang
ca42cbdad1
enhance: update cmake version to 3.27.5 (#28599)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2023-11-22 12:02:22 +08:00
Gao
3e77365de5
fix: correct autoindex segment num (#28387)
Fix #28386 
Current code snippet 
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```

We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-11-22 11:12:22 +08:00
smellthemoon
29249c4bd3
enhance: create goroutine only once (#28594)
create goroutine only once when getOrCreateMergedTimeTickerSender

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-22 10:26:28 +08:00
Bingyi Sun
e8c5e75786
enhance: ignore proto generated files (#28537)
ignore proto generated files
issue: https://github.com/milvus-io/milvus/issues/28566

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-22 10:20:22 +08:00
zhuwenxing
9fd81025ea
test:skip a bulk insert testcase temporarily (#28619)
skip a bulk insert test case temporarily.
It is a known issue but needs more time to solve. skip the test case is
for not blocking other PR

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2023-11-22 10:14:22 +08:00
smellthemoon
73f2bab454
enhance:add some log when create client and get component states (#28160)
/kind improvement

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-22 09:12:22 +08:00
congqixia
2fc743992a
fix: syncmgr unstable TestCompact unittest logic (#28630)
fix #28629
orignal unit test close channel before setting the segment id, so there
is a chance that test read segment id before setting it change unit test
behavior to wait future return now

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-22 00:52:23 +08:00
yah01
d73dac52c7
enhance: validate JSON data while inserting (#28602)
some SDKs doesn't check the JSON data validation,
add this in server.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 23:30:22 +08:00
congqixia
e47fe4e4d1
enhance: Reduce ListImportTasks log content (#28605)
In proxy `ListImportTasks` may print all task information from
rootcoord, this may cause log content too large to process
This PR make this log print taskID and importState only

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 20:58:23 +08:00
congqixia
faf38ed8dd
fix: collection not Unref in local worker release (#28590)
See also #28589

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 20:20:24 +08:00
congqixia
efe3fe7b2e
fix: collection released when localWorker load segments (#28598)
See also #28596
Increase ref for collection during load and unref after load completed.
Use the same logic protection from services.go `LoadSegments`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 20:18:22 +08:00
yah01
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition (#28456)
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
yah01
bfccfcd0ca
enhance: refine error messages (#28424)
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 17:02:24 +08:00
aoiasd
b1fbc1f4e7
fix: datacoord l0 segment bugs (#28490)
relate: https://github.com/milvus-io/milvus/issues/27675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-21 16:48:22 +08:00
congqixia
6521b519d7
enhance: Use bufSize instead of row number in sync policy (#28498)
See also #27675, comment from #27874

This PR changes the `IsFull` logic of insert buffer
Changing the limit from rowNum to buffer size,
this shall help form better binlog file when schema has variable-length
field

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 15:02:25 +08:00
Bingyi Sun
0f985c9961
enhance: remove concurrency level in checkSegmentSize (#28533)
`concurrency` is no longer needed in `checkSegmentSize`

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-21 15:00:23 +08:00
congqixia
2b3fa8f67b
fix: Add length check for storage.NewPrimaryKeyStats (#28576)
See also #28575
Add zero-length check for `storage.NewPrimaryKeyStats`. This function
shall return error when non-positive rowNum passed.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-21 10:28:21 +08:00
PowderLi
c238bff9fb
fix: symbol 'GetStorageMetrics' and 'enableDynamicField' (#28580)
/kind bug
to #28579 #28504

1. replace enableDynamic with enableDynamicField
2. cgo directly link to milvus_storage

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-21 10:20:22 +08:00
Bingyi Sun
d7145e2c06
enhance: Update golangci_lint version (#28535)
Update golangci lint and fix some warnings

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-21 10:04:21 +08:00
yah01
02c5a649cf
enhance: store system fields in segcore (#28524)
we need the system fields info for some usacase
fix: #28523

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 09:28:22 +08:00
Enwei Jiao
7445d3711c
feat: trigger compaction to handle index version (#28442)
issue: https://github.com/milvus-io/milvus/issues/28441

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-21 09:26:22 +08:00
XuanYang-cn
e88bbaac24
feat: Add universal levelzero segment switch (#28483)
See also: #27349

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-20 19:04:22 +08:00