test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-03 04:19:18 +08:00
Code Issues Actions 13 Packages Projects Releases Wiki Activity
20,092 Commits 95 Branches 105 Tags 184 MiB
9bcab2ebee
Commit Graph

20092 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
SimFG
9bcab2ebee
enhance: [2.4] use the insert rate limit for the upsert request (#34616) 
- issue: #34614
- pr: #34615

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-12 16:19:36 +08:00
zhagnlu
4e02e57044
enhance: mark duplicated pk as deleted (#34619) 
pr: #34586

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-12 10:27:37 +08:00
wayblink
86b57b7827
enhance: [cherry-pick] Alter compactTo segments before compactFrom to avoid data loss if crash (#34532) 
issue: #34512 
pr: #34513

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-12 00:47:37 +08:00
jaime
1d2062a68a
fix: encountering orphan channel-cp meta after GC (#34609) 
issue: #34545
pr: #34612

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 22:51:37 +08:00
XuanYang-cn
be41892c48
fix: Compaction task l0 state transfer wrong (#34599) 
This bug caused failed L0 compaction tasks never end.

See also: #34460
pr: #34597

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-11 20:21:37 +08:00
wayblink
9dc98ac76d
enhance: [cherry-pick] Refine clustering compaction latency metrics (#34539) 
issue: #34527
pr: #34528

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-11 17:43:43 +08:00
edward.zeng
3d29ebfffb
enhance: Refine default histogram metric bucket (#34592) 
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2024-07-11 16:51:35 +08:00
XuanYang-cn
7e2a9d6bca
fix: Milvus panic when compaction disabled and dropping a collection (#34206) 
See also: #31059
pr: #34103

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-11 14:20:52 +08:00
aoiasd
d6fc6a9f41
fix: [Cherry-pick] Restful API use deprecate error code cause access log panic. (#34579) 
Avoid panic when use deprecate error code and use merr code replace
deprecate common code.
relate: https://github.com/milvus-io/milvus/issues/34578
pr： https://github.com/milvus-io/milvus/pull/34576

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-11 11:58:51 +08:00
jaime
bfd386aad7
fix: unstable UT for level0 deletion (#34525) 
issue: #34533
pr: #34524

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 10:02:56 +08:00
jaime
073a501fc7
enhance: skip orphan channel cp meta when checking cp lag (#34556) 
issue: #34545
pr: #34555

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 09:36:56 +08:00
XuanYang-cn
2bef9c38c6
enhance: Add back compactionTaskNum metrics (#34534) 
Fix L0 compaction task recover unable to set segment
not isCompacting

See also: milvus-io#34460
pr: milvus-io#34583

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-11 00:08:52 +08:00
yihao.dai
cbba5481f8
fix: Prevent dispatcher merging if curTs is 0 (#34562) (#34563) 
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.

issue: https://github.com/milvus-io/milvus/issues/34255

pr: https://github.com/milvus-io/milvus/pull/34562

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-10 22:46:51 +08:00
SimFG
15adb2feac
enhance: [2.4] add the seal segment when dispatch delete msgs (#34566) 
/kind improvement
- pr: #34565

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-10 19:18:51 +08:00
wei liu
a7190b042d
fix: Indexnode stuck at stopping progress cause by wrong lifetime control (#34559) 
issue: #34557
pr: #34558

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-10 15:14:13 +08:00
yihao.dai
1589d68413
fix: Fix no L0 compaction tasks are executed (#34537) (#34538) 
The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.

issue: https://github.com/milvus-io/milvus/issues/34460

pr: https://github.com/milvus-io/milvus/pull/34537

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-10 11:08:12 +08:00
wei liu
f0d12aa68d
fix: RootCoord panic during check health (#34542) 
issue: #34540
pr: #34541

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-10 09:50:38 +08:00
congqixia
0d8defb1f9
fix: [2.4] Write padding at end of mmap file not chunk (#34530) 
Cherry-pick from master
pr: #34529
Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-09 20:32:14 +08:00
jaime
60aab15ef4
enhance: support setting properties in create database request (#34511) 
issue: #34493
pr: #34510

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-09 18:16:18 +08:00
congqixia
ddb1015563
fix: [2.4] Check err is ErrKeyNotFound when CASCachedValue (#34489) 
Cherry-pick from master
pr: #34488
See also #33785

When config item is not present in paramtable, CAS fails due to
GetConfig returns error.

This PR make this returned err instance of ErrKeyNotFound and check
error type in \`CASCachedValue\` methods.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-09 18:12:13 +08:00
SimFG
737bd7c734
enhance: [2.4] release the record in delete codec and add some log for compaction (#34506) 
/kind improvement
- pr: #34454

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-09 15:40:17 +08:00
wayblink
7788decb37
fix: [cherry-pick] Fix compaction start time (#34505) 
bug: #34497 #34498
pr: #34496

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-09 13:30:14 +08:00
yihao.dai
42b38fba00
fix: Fix unstable ut of msgdispatcher (#34485) (#34486) 
issue: https://github.com/milvus-io/milvus/issues/34484

pr: https://github.com/milvus-io/milvus/pull/34485

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-09 10:06:24 +08:00
wei liu
7034260721
fix: Query may return deleted records (#34502) 
issue: #34500
pr: #34501

cause the sort in `GetLevel0Deletions` will broken the corresponed order
between pks and tss, then the pks and tss will be sorted in
segment.Delete() interface.

This PR remove this uncessary and incorrect sort progress to avoid query
may return deleted records.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-09 10:00:14 +08:00
Gao
f6cd84161c
enhance: ensure autoindex default metric type compatibility (#34479) 
issue: #34304 
pr: #34261

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-08 19:48:14 +08:00
Chun Han
2f38483418
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics(#30376) (#34475) 
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34477

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-07-08 19:44:13 +08:00
wayblink
56a74e72f7
fix: [cherry-pick]fix can't enqueue when compaction queue is full(#34445) (#34469) 
issue: #30633
pr: #34445

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-08 19:20:13 +08:00
yihao.dai
0d7ba810b3
enhance: Check segment existence when FlushSegments and add some key logs (#34438) (#34472) 
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

pr: https://github.com/milvus-io/milvus/pull/34438

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-08 19:00:13 +08:00
yihao.dai
0732167c87
fix: Fix incorrect segment num rows (#34441) (#34474) 
Repeated calls to UpdateStatistics, this PR correct it.

issue: https://github.com/milvus-io/milvus/issues/34440

pr: https://github.com/milvus-io/milvus/pull/34441

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-08 17:30:12 +08:00
XuanYang-cn
26ac76944b
fix: Accidently exit the check loop (#34480) 
See also: #34460
pr: #34481

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-08 17:14:15 +08:00
congqixia
7c3c9c2ed4
fix: [2.4] Add nbits parameter check for IVF_PQ (#34451) (#34473) 
Cherry-pick from master
pr: #34451
See also #34426

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-08 17:10:12 +08:00
zhuwenxing
f2d0517f96
test: [cherry-pick]add testcase for count query (#34471) 
pr: https://github.com/milvus-io/milvus/pull/34453

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-07-08 12:54:12 +08:00
Gao
a60e2a65ff
enhance: change autoindex default metric type (#34277) 
issue: #34304 
pr: #34261

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-08 10:52:14 +08:00
congqixia
014820e9d2
fix: [2.4] Write padding into mmap file in case of SIGBUS (#34443) (#34455) 
Cherry-pick from master
pr: #34443
See also #34442

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-08 10:06:19 +08:00
jaime
326370c1be
enhance: add disk quota and max collections into db properties (#34386) 
issue: https://github.com/milvus-io/milvus/issues/34385
pr: #34368

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 18:22:17 +08:00
wei liu
d3e94f9861
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl (#34377) 
issue: #32995
pr: #33405
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.

WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.

In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.

Block BF construct time {"time": "54.128131ms"}
Block BF size {"size": 3021578}
Block BF Test cost {"time": "55.407352ms"}
Basic BF construct time {"time": "210.262183ms"}
Basic BF size {"size": 2396308}
Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.

Block BF TestLocation cost {"time": "529.97183ms"}
Basic BF TestLocation cost {"time": "3.197430181s"}

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-05 17:04:10 +08:00
zhagnlu
173c02902e
enhance: refactor variable column to reduce memory cost (#33875) (#34367) 
cherry-pick commit from master:
pr: #33875

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-05 15:52:09 +08:00
congqixia
f5a0353fd1
enhance: [2.4] Continue loop when reassign channel fails (#34331) (#34425) 
Cherry-pick from master
pr: #34331
Log will be confusing when `Reassign` channel operation failed for both
success & failure log will be printed in row. This PR continue the loop
to avoid this output.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-05 11:14:08 +08:00
Gao
261b61e875
fix: centroids file not removed when data skew in major compaction (#34359) 
issue: https://github.com/milvus-io/milvus/issues/30633
pr: #34050

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-05 10:42:10 +08:00
zhagnlu
74da97796b
enhance: Enhance and correct exception module (#34366) 
cherry-pick commit from master:
pr: #33705

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-05 10:06:10 +08:00
PowderLi
ba2c232331
fix: [cherry-pick] [restful v2] count(*) & hook (#34433) 
issue: #31224 #34374
pr: #34369

for query api:

1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]

add hook about grpc call

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-07-05 09:52:10 +08:00
zhagnlu
190a2ca7b8
enhance: reduce cpp ut test cost time (#34414) 
#34413
cherry-pick part from master commit: pr: #33358

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-04 20:40:10 +08:00
congqixia
b402485292
fix: [2.4] Skip l0 segments when syncing segments to datanodes (#34389) 
Cherry-pick from master
pr: #34388
See also #34387

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-04 18:02:14 +08:00
yihao.dai
a57c9e61fc
enhance: [cherry-pick] optimize datanode cpu usage and correct the update logic of ttchecker (#34383) 
This PR cherry-picks the following commits:
- Try to improve cpu usage by refactoring the ttchecker logic and
caching string. https://github.com/milvus-io/milvus/pull/33267
- Correct the update logic of timerecorder in the flowgraph to avoid
false failure: "some node(s) haven't received input".
https://github.com/milvus-io/milvus/pull/34339

issue: https://github.com/milvus-io/milvus/issues/33266,
https://github.com/milvus-io/milvus/issues/34337

pr: https://github.com/milvus-io/milvus/pull/33267,
https://github.com/milvus-io/milvus/pull/34339

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Xiaofan <83447078+xiaofan-luan@users.noreply.github.com>
 2024-07-04 16:34:17 +08:00
Chun Han
5831908aa2
enhance: reconstruct scalar part's code for segment-pruner(#30376) (#34365) 
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34346
1. support more complex expr
2. add more ut test for unrelated fields

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-04 16:30:10 +08:00
XuanYang-cn
0f1915ef24
fix: DataNode might OOM by estimating based on MemorySize (#34203) 
See also: #34136
pr: #34201

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 15:24:10 +08:00
shaoting-huang
dd4dfbcd8d
enhance: [cherry-pick] Batch pick PRs related to data codec (#34345) 
This PR cherry-picks the following commits related to data codec
- Fix data codec writer close. #33818
- Legacy code clean up. #33838

issue: #33813 #33839 

pr: #33818 #33838

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-07-04 15:08:11 +08:00
foxspy
b33243f236
enhance: Update Knowhere version (#34405) 
1. cherry-pick pr #34223 (update the parameter dataset from reference to
share_ptr)
2. update knowhere version from v2.3.5 to v2.3.6
(https://github.com/zilliztech/knowhere/releases/tag/v2.3.6)

/kind branch-feature

---------

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-07-04 14:46:11 +08:00
wayblink
b3aec4c8e1
fix:[cherry-pick] minor fixs for major compaction (#34402) 
This PR cherry-picks the following commits:

- fix: Avoid datarace in clustering compaction #34288
- fix: remove isFull check in compaction.enqueue #34338

issue: #30633 
pr: #34288 #34338

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-04 14:44:10 +08:00
chyezh
a1a0a56f86
enhance: async search and retrieve in cgo (#34200) 
issue: #33132
pr: #33133
other pr: #33228, #34084, #33946

- implement future-based cgo utility
- async search and retrieve in cgo
- modify gc configuration document

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-04 13:02:09 +08:00