Related to #28736#28748
See also #27675
Previous PR: #28646
This PR fixes `SegmentNotFound` issue when compaction happens multiple
times and the buffer of first generation segment is sync due to stale
policy
Now the `CompactSegments` API of metacache shall update the compactTo
field of segmentInfo if the compactTo segment is also compacted to keep
the bloodline clean
Also, add the `CompactedSegment` SyncPolicy to sync the compacted
segment asap to keep metacache clean
Now the `SyncPolicy` is an interface instead of a function type so that
when it selects some segments to sync, we colud log the reason and
target segment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #28622
query node with delegator will has more rows than other query node due
to delgator loads all growing rows.
This PR enable the balance segment which based on the num of growing
rows in leader view.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
fix macOS compilation fail, related #28715
1) install_deps.sh missed to install some dependencies
2) llvm can not be found without a soft link
Signed-off-by: xiaofan luan <xiaofanluan@xiaofandeMacBook-Pro.local>
Co-authored-by: xiaofan luan <xiaofanluan@xiaofandeMacBook-Pro.local>
See also #27675
When L0 segment contains only delta data, merged statslog shall be
skiped when performing sync task
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
Compacted segment info shall be removed after all buffer belongs to it
is sync-ed.
This PR add the cleanup function after triggerSyncTask logic:
- The buffer is stable and protected by mutex
- Cleanup fetches compacted & non-sync segment
- Remove segment info only there is no buffered maintained in manager
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
remove some unnecessary assignments, for the reason that
commonpbutil.NewMsgBase has default value.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
issue: #28683
to avoid downloading installation packages in CI workload install vcpkg
and install some packages in advance
Signed-off-by: PowderLi <min.li@zilliz.com>
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
See also #28022#28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #28660
This pr add request timeout config item for etcd kv request timeout
Sync the default timeout value to same value for etcdKV & tikv config
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #28365
Fix bug for parsing error when a string enclosed in single quotes in an
expression contains multiple double quotes.
such as:
```
expr = "tag == '\"blue\"'"
```
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.
we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage
resolve#28553
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
See also #28628
Previous compaction task blocked the segment sync task and may block the
flowgraph when sync task is generated by auto sync policy This
`BlockAll` call will block forever and cause whole fg stuck
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Fix#28386
Current code snippet
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```
We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.
Signed-off-by: chasingegg <chao.gao@zilliz.com>
create goroutine only once when getOrCreateMergedTimeTickerSender
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
skip a bulk insert test case temporarily.
It is a known issue but needs more time to solve. skip the test case is
for not blocking other PR
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
fix#28629
orignal unit test close channel before setting the segment id, so there
is a chance that test read segment id before setting it change unit test
behavior to wait future return now
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
