test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-05 05:18:52 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,850 Commits 96 Branches 105 Tags 186 MiB
ddc3e76803
Commit Graph

138 Commits

Author SHA1 Message Date
congqixia
ba8197fafb
fix: Filter channel level zero segments when build level delete cache (#31129) 
See also #31125

Delegator shall build level zero delete cache from l0 segments belongs
to it. Previously it build cache from all existing level zero segments
in the querynode which may lead to high memory usage and even panicking
when pk types are not matched

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-07 23:01:02 +08:00
smellthemoon
a4f3e01a3a
fix: add the range search params check in proxy (#30423) 
if check in Segcore, will not do the it when not insert data.
so, check "radius" and "range_filter" in proxy.
related with #30365

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-02-28 11:24:58 +08:00
xige-16
060c8603a3
fix: Support mvcc with hybrid serach (#30114) 
issue: https://github.com/milvus-io/milvus/issues/29656
/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-02-01 16:03:03 +08:00
congqixia
7c086a4608
enhance: Set delete scope for LoadSegment streaming data (#30245) 
See also #29474

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-30 11:13:02 +08:00
chyezh
d300bc7bcb
fix: querynode num entity metric is broken by illegal label (#29948) 
issue: #29766
also see pr: #29825

Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2024-01-14 10:23:00 +08:00
congqixia
adf0c8885c
enhance: add trace span for wait tsafe (#29911) 
Add tracing span for search/query operation waiting tsafe duration

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-12 14:03:04 +08:00
yah01
d357139064
fix: the entities num metric may be contributed more than once (#29767) 
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions

fix: #29766

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-10 10:00:51 +08:00
zhenshan.cao
60e88fb833
fix: Restore the MVCC functionality. (#29749) 
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-09 11:38:48 +08:00
congqixia
fe47deebf3
fix: Set & Return correct SegmentLevel in querynode segment manager (#29740) 
See also #27349

The segment level label in querynode used `Legacy` before segment level
was correctly passed in Load request. Now this attribute is still using
legacy so the metrics does not look right.

This PR add paramter for `NewSegment` and passes corrent values for each
invocation.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-08 14:16:48 +08:00
yah01
a0cec4047a
fix: make the entity num metric accurate (#29643) 
fix #29642

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-05 18:24:47 +08:00
congqixia
da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp (#29626) 
See also #29625

This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
  - holds cacheBlock slice
  - `Put` method append new delete data into last block
  - when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
  - For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-04 17:02:46 +08:00
MrPresent-Han
ed644983e2
enhance: add param for bloomfilter(#29388) (#29490) 
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-28 18:10:46 +08:00
congqixia
b251c3a682
enhance: add ctx for HandleCStatus and callers (#29517) 
See also #29516

Make `HandleCStatus` print trace id for better logging

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-27 16:10:47 +08:00
congqixia
02bc0d0dd5
fix: Add scope limit for querynode DeleteRequest (#29474) 
See also #27515

When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-26 14:28:47 +08:00
congqixia
ac95c52171
enhance: change protection to RLock for loadStreamDelete (#29450) 
See also: #29332

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-25 23:27:02 +08:00
congqixia
1eacdc591b
fix: delegator may mark segment offline by mistake (#29343) 
See also #29332

The segment may be released before or during the request when delegator
tries to forward delete request to yet. Currently, these two situation
returns different error code.

In this particular case, `ErrSegmentNotLoaded` and `ErrSegmentNotFound`
shall both be ignored preventing return search service unavailable by
mistake.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-20 21:22:43 +08:00
yah01
9b3e06ae86
enhance: add more metrics for level zero segments (#29029) 
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-07 14:34:35 +08:00
Gao
3e77365de5
fix: correct autoindex segment num (#28387) 
Fix #28386 
Current code snippet 
```
// get delegator
sd, ok := node.delegators.Get(channel)
if !ok {
err := merr.WrapErrChannelNotFound(channel)
log.Warn("Query failed, failed to get shard delegator for search", zap.Error(err))
return nil, err
}
req, err = node.optimizeSearchParams(ctx, req, sd)
if err != nil {
log.Warn("failed to optimize search params", zap.Error(err))
return nil, err
}
// do search
results, err := sd.Search(searchCtx, req)
```

We could move these into `ShardDelegator`, and directly use sealed
segment num in `Search` methods, also segment num got outside could be
wrong when we specify partitions.

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-22 11:12:22 +08:00
yah01
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition (#28456) 
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 18:24:22 +08:00
yah01
d20ea061d6
Fix panic while forwarding empty deletions to growing segment (#28213) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 16:42:21 +08:00
yah01
ece592a42f
Deliver L0 segments delete records (#27722) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-07 01:44:18 +08:00
wei liu
ecec5dfcfd
fix retry on offline node (#28079) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-03 10:14:16 +08:00
congqixia
e4fdf5e68e
Refine offline segments logic in shard delegator (#28073) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-01 20:04:24 +08:00
congqixia
13877a07ff
Add ctx parameter for tsafe pkg & NewDelegator method (#27877) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-26 19:14:10 +08:00
wei liu
8041fc3c75
avoid add empty growing segment to delegator distribution (#27930) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-26 10:10:10 +08:00
aoiasd
9ba79a97e4
Add some log for delete task when it need query (#27317) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-24 14:54:12 +08:00
yihao.dai
d9431266bb
Fix panic at loadSegment during rolling upgrade (#27671) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-13 16:31:35 +08:00
yah01
be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
wei liu
0695c88844
refine_log_of_update_target_version (#27572) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-11 06:51:33 +08:00
congqixia
258e1ccd66
Refine querynode scheduler lifetime (#26915) 
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:21:26 +08:00
congqixia
8c59dba329
Refine queryHook mockery (#27394) 
This PR move `QueryHook` interface to `optimizers` pkg
Update all mockery generated files to latest
Add makefile entry for `QueryHook`

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:01:26 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia
cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
Xiaofan
e51cb739eb
Remove some logs in querynode delegator (#27156) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-09-18 10:57:20 +08:00
aoiasd
e107d0794c
support complex delete expression (#25752) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-09-12 10:19:17 +08:00
congqixia
2a5d574a0d
Fix querynodev2 concurrent load logic (#26959) 
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done

Make delegator control concurrency for each same segment
Related to #26908

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-10 07:41:18 +08:00
XuanYang-cn
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek (#26686) 
See also: #25309

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-08 09:51:17 +08:00
congqixia
af5c01082b
Refine delegator lifetime control (#26881) 
- Add SafeChan interface in lifetime package
- Embed SafeChan into  interface
- Replace private lifetime struct in delegator package with
- Refine delegator on-going task lifetime control and wait all accepted task done
- Fix potential goroutine leakage from  if delegator closed concurrently

/kind improvement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-07 10:11:15 +08:00
MrPresent-Han
528948559f
fix false load failure for long unserviable period(#26813) (#26818) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-06 12:57:15 +08:00
congqixia
4b58c71908
Add ctx parameter for organizeTask and GetWorker method (#26835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 10:05:48 +08:00
congqixia
1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia
89fc9aad82
Improve sync target version logic (#26630) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-29 23:12:27 +08:00
wei liu
7af0f7d90c
avoid concurrent sub/unsub on same channel (#26454) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-23 10:10:22 +08:00
congqixia
855a04539a
Set delegator lastUpdateTs to startTs (#26416) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 12:24:18 +08:00
wei liu
23baecd70f
set sealed segment to unreadable before sync target version (#26338) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-15 17:27:35 +08:00
Enwei Jiao
7d61355ab0
Refactor log for Query (#26310) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-14 18:57:32 +08:00
wei liu
b47a72bfcf
fix set dirty segment distribution to leader view (#26180) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-11 11:21:32 +08:00
congqixia
eea9197306
Perform alter distribution under mutex protection (#26229) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 13:05:15 +08:00
yah01
99c633954a
Downgrade the log level in search path (#26188) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-08 18:35:14 +08:00
Bingyi Sun
54c0e64059
Fix search on empty segments set bug (#26136) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-08-08 11:17:08 +08:00
wei liu
a132122370
fix time unit in insert/delete cost metrics value (#25893) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 15:07:01 +08:00
aoiasd
b533c68632
Forbid update checkpoint without msgID (#25694) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-07-25 10:43:04 +08:00
wei liu
0e26486abc
fix segment num in log (#25876) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-25 09:13:00 +08:00
wei liu
fc19b85a40
fix count(*）retrieve redundant growing segment (#25825) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-24 14:09:00 +08:00
wei liu
32827f538a
add metrics for delegator insert/delete cost (#25733) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-21 15:30:59 +08:00
Cai Yudong
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
congqixia
cc71e065e3
Support load index after segment loaded (#25567) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 10:51:19 +08:00
Zhihong Yu
a7023ab3cc
Narrow the scope of deleteMut lock in loadStreamDelete (#25436) 
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
 2023-07-17 09:30:34 +08:00
wei liu
b62c82af22
fix set target version in loading sealed segment (#25603) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-14 20:00:31 +08:00
yah01
d216f9abda
Clear collection meta after all channels/segments released (#25486) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-14 10:28:30 +08:00
yah01
205a7c430a
Fix QueryNode panic while inserting to released segment (#25495) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-12 19:48:29 +08:00
SimFG
f9e2d00f91
Prevent exclusive consumer exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
congqixia
9e3a591bda
Fix segment released early in case of balancing (#25453) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-10 18:58:28 +08:00
wei liu
d4704ab9b6
fix set shard unservicable when sync target version (#25418) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-07 19:32:25 +08:00
Jiquan Long
8d193a3e56
Skip unnecessary reduce phase during search (#25166) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-27 19:22:45 +08:00
congqixia
b3362c297a
Make snapshot.expired atomic (#25177) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-27 18:12:45 +08:00
wei liu
68ae199a9f
load segment with target version, avoid read redundant segment (#24929) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-27 11:48:45 +08:00
wei liu
e40d95e093
add search/query request cost metrics on qn (#24413) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-15 14:24:38 +08:00
yihao.dai
e81eb56b92
Add metric of waiting for tSafe latency (#24765) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-09 11:26:35 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
Enwei Jiao
d3af451d92
Upgrade golangci-lint (#24707) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-06-07 19:34:36 +08:00
yihao.dai
5f84923bfa
Fix search/query unloaded data (#24368) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-26 15:01:26 +08:00
wei liu
ce03248b1a
simplify logic circle in search (#24110) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-23 16:01:26 +08:00
congqixia
73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
yah01
400364483d
Improve predict the resource usage (#23968) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-11 15:33:24 +08:00
yihao.dai
7e0d1492c7
Load delete from channel checkpoint (#23961) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-09 19:10:41 +08:00
foxspy
6f4ed517de
add growing segment index (#23615) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-04-26 10:14:41 +08:00
wei liu
3ad9ff7a9a
fix release segment failed (#23462) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-17 18:46:30 +08:00
congqixia
a9a5f76f74
Fix AddDistribution unit test (#23418) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-13 20:46:29 +08:00
congqixia
6cd73d4e2a
Wait safe signal before release growing (#23358) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-11 17:56:31 +08:00
congqixia
0527d2706c
Fix offline set always nil (#23340) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-11 14:42:30 +08:00
MrPresent-Han
ba02e70301
fix bug when syncing distribution without schema and remove unused logic(#23085) (#23216) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-04-07 19:32:29 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
congqixia
e3e3ac1525
Use peek segment info in querynodev2 (#23183) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-03 16:10:24 +08:00
yihao.dai
b6f69fe7f2
Check if loaded before delegator search/query (#23162) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-03 14:26:23 +08:00
wei liu
e2096965c7
fix leader view (#23038) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-03-29 14:06:02 +08:00
yah01
081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00