test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 03:48:37 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
20,928 Commits 94 Branches 105 Tags 184 MiB
72dc07ba48
Commit Graph

559 Commits

Author SHA1 Message Date
aoiasd
72dc07ba48
fix: bm25 search failed when nq > 1 and remove idf oracle when no bm25 field exist. (#36886) 
relate: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-16 12:51:23 +08:00
yihao.dai
f3b6792a25
enhance: Enhance segment log (#36848) 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-15 20:43:30 +08:00
congqixia
ba25320aea
fix: Unify loaded partition check to delegator (#36879) 
Related to #36370

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-15 19:15:23 +08:00
aoiasd
5ec4163d0f
feat: support bm25 logs mixcompaction (#36072) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-14 16:57:22 +08:00
Buqian Zheng
383350c120
feat: added more checks for function creation check (#36766) 
issue: https://github.com/milvus-io/milvus/issues/35853

* BM25 Function now takes no params, k1, b should be passed via index
params
* support BM25 full text search when metric type is not present in
search request
* add more strict validation with functions at collection creation time

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-10-13 17:43:22 +08:00
Buqian Zheng
16b533cbf0
feat: Restful support for BM25 function (#36713) 
issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-10-13 17:41:21 +08:00
Bingyi Sun
a75bb85f3a
feat: support chunked column for sealed segment (#35764) 
This PR splits sealed segment to chunked data to avoid unnecessary
memory copy and save memory usage when loading segments so that loading
can be accelerated.

To support rollback to previous version, we add an option
`multipleChunkedEnable` which is false by default.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-10-12 15:04:52 +08:00
aoiasd
db34572c56
feat: support load and query with bm25 metric (#36071) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-11 10:23:20 +08:00
yihao.dai
3685edb264
enhance: Use common gc config (#36668) 
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 19:47:19 +08:00
SimFG
130a923dec
enhance: the estimate method when loading the collection (#36307) 
- issue: #36530

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-09 17:35:19 +08:00
congqixia
ddc3e76803
fix: Add defer Unpin when error happens (#36620) 
Resolves: #36619

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 19:49:17 +08:00
wei liu
470bb0cc3f
enhance: Enable balance on querynode with different mem capacity (#36466) 
issue: #36464
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 16:15:17 +08:00
cai.zhang
ecb2b242e2
enhance: Add sorted for segment info (#36469) 
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-30 10:01:16 +08:00
congqixia
4fd9b0a8e3
enhance: Return segment id hint in QueryStream response (#36487) 
Related to #36482

This PR reuses `SealedSegmentIDsRetrieved` field in `RetrieveResults`
struct to store segment id hint.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-26 10:13:14 +08:00
congqixia
ed95568a05
enhance: Fix PR conflict in reduce unit test (#36470) 
Related to #36433 #36180

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-24 18:01:13 +08:00
congqixia
98a917c5d4
enhance: [skip e2e] Add unittest for reducing duplicated pk from multi segments (#36433) 
Related to #35505 #36362

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-24 14:11:13 +08:00
Chun Han
df7ae08851
fix: iterator cursor progress too fast(#36179) (#36180) 
related: #36179

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-24 11:45:13 +08:00
congqixia
1833913f44
enhance: Add streaming forward policy switch for delegator (#36330) 
Related to #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-23 18:01:12 +08:00
yihao.dai
763fd0dfc5
enhance: Use a separate mmap config for chunk cache (#36276) 
issue: https://github.com/milvus-io/milvus/issues/35273

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-15 16:23:09 +08:00
wei liu
329fb421cd
fix: fix search/query/count may access same growing and sealed segment (#36258) 
issue: #36257
during syncTargetVersion, sealed segment should be excluded, to avoid
it's growing segment be conusmed from stream again.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-14 14:21:07 +08:00
congqixia
3bc7d63be9
fix: overwrite correct selection when pk duplicated (#35826) 
Related to #35505

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-14 10:27:08 +08:00
aoiasd
c22a2cebb6
fix: split stream query result to avoid grpc response too large error (#36090) 
relate: https://github.com/milvus-io/milvus/issues/36089

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-13 15:07:09 +08:00
congqixia
11dbe1e755
enhance: Add L0 forward policy to support remote load (#36189) 
Related to #35303

This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-12 12:01:08 +08:00
Jiquan Long
89bf226f0b
feat: support keyword text match (#35923) 
fix: #35922

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-09-10 15:11:08 +08:00
Chun Han
9d0aa5c202
fix: empty result when having only one subReq(#36098) (#36128) 
related: #36098

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-10 14:25:07 +08:00
zhagnlu
208c8a2328
fix:support config index offsetcache and fix create same index again (#35985) 
#35971

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-08 18:23:05 +08:00
Chun Han
e480b103bd
feat: supporing hybrid search group_by (#35982) 
related: #35096

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-08 17:09:04 +08:00
XuanYang-cn
7859faf8ea
fix: Change deltalog memory estimation factor to one (#36033) 
See also: #36031

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-09-06 18:21:05 +08:00
XuanYang-cn
5e3f700e5d
enhance: Remove too frequent logs in Delete (#35980) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-09-05 14:25:03 +08:00
congqixia
8593c4580a
enhance: Add delete buffer related quota logic (#35918) 
See also #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-05 11:39:03 +08:00
jaime
24fb10114b
enhance: remove cooling off in rate limiter for read requests (#35935) 
issue: #35934

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-04 14:39:10 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK (#35054) 
issue: #33744 

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-02 14:19:03 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) 
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-29 10:03:08 +08:00
Chun Han
bfd9d86fe9
feat: support groupby size on go-layer(#33544) (#33845) 
related: #33544

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-08-27 14:21:00 +08:00
aoiasd
fe83805d56
fix: loss data bug for deprecated querynode DoubleBuffer (#35128) 
relate: https://github.com/milvus-io/milvus/issues/31548

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-08-27 14:10:59 +08:00
yihao.dai
f2b83d316b
enhance: Support memory mode chunk cache (#35347) 
Chunk cache supports loading raw vectors into memory.

issue: https://github.com/milvus-io/milvus/issues/35273

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-25 15:42:58 +08:00
Gao
e8e3544a11
enhance: add hit segment num metrics for queryHook (#35577) 
issue: #35576

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-08-23 12:49:02 +08:00
Zhen Ye
a773836b89
enhance: optimize milvus core building (#35610) 
issue: #35549,#35611,#35633

- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-23 12:35:02 +08:00
SimFG
731d45abbe
enhance: provide more general configuration to control mmap behavior (#35359) 
- issue: #35273

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-21 00:22:54 +08:00
Ted Xu
41646c8439
feat: integrate new deltalog format (#35522) 
See #34123

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-20 19:06:56 +08:00
congqixia
2fbc628994
feat: Support field partial load collection (#35416) 
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-20 16:49:02 +08:00
congqixia
f87af9bc54
enhance: Exclude L0 segment from readable snapshot (#35507) 
L0 segments now do not contain insert data and may cause confusion for
query hook optimizer if counted as sealed segment number.

This PR add segment level flag in segment entry and exclude L0 segments
while get readable segment snaphsot

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-16 15:28:53 +08:00
congqixia
6ff238e88a
fix: Set corresponding DataScope for loadStreamDelete (#35312) 
Related to #35311

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-06 22:32:23 +08:00
zhagnlu
4b553b0333
enhance: revert remove duplicated pk function (#35103) 
issue: #34778
 Revert "fix: fix query count(*) concurrently"
 Revert "enhance: mark duplicated pk as deleted "

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-05 10:48:17 +08:00
Chun Han
3faef63a25
enhance: add log for partition stats( #30376) (#35219) 
related:  #30376

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-08-02 19:34:22 +08:00
Bingyi Sun
3641ae6611
fix: Fix index memory estimation (#35225) 
issue: https://github.com/milvus-io/milvus/issues/35229

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-08-02 16:24:15 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling (#35194) 
See also #35196

Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 11:24:19 +08:00
wei liu
27b6d58981
fix: Set legacy level to l0 segment after qc restart (#35197) 
issue: #35087
after qc restarts, and target is not ready yet, if dist_handler try to
update segment dist, it will set legacy level to l0 segment, which may
cause l0 segment be moved to other node, cause search/query failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 10:18:13 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
Gao
6695c6d0a3
enhance: add channel num for queryHook optimization (#35104) 
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-31 18:59:49 +08:00