before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.
we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage
resolve#28553
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
See also #28628
Previous compaction task blocked the segment sync task and may block the
flowgraph when sync task is generated by auto sync policy This
`BlockAll` call will block forever and cause whole fg stuck
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Fix#28386
Current code snippet
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```
We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.
Signed-off-by: chasingegg <chao.gao@zilliz.com>
create goroutine only once when getOrCreateMergedTimeTickerSender
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
skip a bulk insert test case temporarily.
It is a known issue but needs more time to solve. skip the test case is
for not blocking other PR
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
fix#28629
orignal unit test close channel before setting the segment id, so there
is a chance that test read segment id before setting it change unit test
behavior to wait future return now
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
In proxy `ListImportTasks` may print all task information from
rootcoord, this may cause log content too large to process
This PR make this log print taskID and importState only
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #28596
Increase ref for collection during load and unref after load completed.
Use the same logic protection from services.go `LoadSegments`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675, comment from #27874
This PR changes the `IsFull` logic of insert buffer
Changing the limit from rowNum to buffer size,
this shall help form better binlog file when schema has variable-length
field
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #28575
Add zero-length check for `storage.NewPrimaryKeyStats`. This function
shall return error when non-positive rowNum passed.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
/kind bug
to #28579#28504
1. replace enableDynamic with enableDynamicField
2. cgo directly link to milvus_storage
Signed-off-by: PowderLi <min.li@zilliz.com>
These is no enough unittest case for EventDispatcher, see also #28540.
This PR add unit test case for all methods for EventDispatcher
Related to #28538
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
- Fix LevelZero segment cannot be flushed
- Add level option for syncTask
- Invoke `AddSegment` when new LevelZero segment is allocated
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This check rejects load request if running out the pool workers, but
small segment would be loaded soon, another segments would been loading
again after a check interval, which leads to slow loading for collection
Block the request by go pool
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
See also #27675
Since `MetaWriter` need `*datapb.FieldBinlog` struct, sync task now
generate FieldBinlog directly
Also fix merged statslog not generated if last task has no insert
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #28332
during querycoord's recover, it try to call `DescribeCollection` and
`ShowPartitions` to root coord, to checker whether collection or
partition has been released in rootcoord. but if rootcoord isn't not
ready yet, the rpc will fail, the querycoord panic.
to fix this, we remove rpc call during querycoord's start
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
add 2 more types:
- test: for add tests to existing functionality.
- doc: for modifying documentation.
Also requires related issue for large PR
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
according to the issue: #28312
below is the feature added
## change 1:
discard the 'kind/improvement' label and instead use the
'kind/enhancement' label"
## change 2:
delete the existing two rule in mergify file, to allow multiple commits
in a PR
- name: Add `needs-rebase` label when more than one commit in pr
- name: Remove `needs-rebase` label when only one commit in pr
## change 3:
add two new rules for PR format
- name: Add 'do-not-merge/invalid-pr-format' label for invalid PR titles
- name: Remove 'do-not-merge/invalid-pr-format' label for valid PRs
## change 4:
mapping PR type to corresponding label, below is the logic
if PR title start with 'fix:', add 'kind/bug' label
else if PR title start with 'feat:',add 'kind/feature' label
else if PR title start with 'enhance:', add 'kind/enhancement' label
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
See also #28466
In `taskDispatcher.schedule`, same task may be resubmitted if the
previous round did not finish
In this case, TaskObserver.check may set current target by mistake,
which may cause the random search/query failure
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR removes all the commented code and files from PR #28320
For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving
Signed-off-by: yah01 <yah2er0ne@outlook.com>
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
