test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 21:09:06 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,002 Commits 96 Branches 105 Tags 186 MiB
c344083f22
Commit Graph

1051 Commits

Author SHA1 Message Date
jaime
b37d6fa0f9
enhance: decrease cpu overhead during filter segments on datacoord (#34231) 
issue: https://github.com/milvus-io/milvus/issues/33129
pr: #33130 
pr: #33373

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:20:08 +08:00
jaime
0992f10694
enhance: improve check health (#34265) 
issue: https://github.com/milvus-io/milvus/issues/34264
pr: #33800

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:18:07 +08:00
cai.zhang
85aa929749
enhance: Set the timeout for SyncSegments based on the number of segments (#34228) 
issue: #32809 

master pr: #34226

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-28 14:18:06 +08:00
cai.zhang
1c6e850f73
enhance: [cherry-pick] Periodically synchronize segments to datanode watcher (#33420) (#34186) 
This PR primary picks up the SyncSegments functionality, including the
following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33420
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33664
  - https://github.com/milvus-io/milvus/pull/33829
  - https://github.com/milvus-io/milvus/pull/34056
  - https://github.com/milvus-io/milvus/pull/34156

issue: #32809 
master pr: #33420, #33664, #33829, #34056, #34156

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:24:05 +08:00
cai.zhang
42096a2337
enhance: [cherry-pick] Reduce the frequency of index ralted logs (#33050) (#34198) 
issue:  #33001


pr: #33050 , #33212 , #33321

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:08:05 +08:00
jaime
6423b6c718
enhance: move rocksmq from internal to pkg (#34165) 
pr:  https://github.com/milvus-io/milvus/pull/33881
issue:  https://github.com/milvus-io/milvus/issues/33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 13:36:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
yihao.dai
3ae1865ffb
fix: Prevent import from generating orphaned files (#34070) (#34071) 
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: https://github.com/milvus-io/milvus/issues/34068

pr: https://github.com/milvus-io/milvus/pull/34070

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 17:00:04 +08:00
XuanYang-cn
a33b68678d
enhance: [cherry-pick] Move compactor into sub package (#34098) 
This PR consists of the following commits:

- enhance: Tidy compactor and remove dup codes (#32198)
- fix: Fix l0 compactor may cause DN from OOM (#33554)
- enhance: Add deltaRowCount in l0 compaction (#33997)
- enhance: enable stream writer in compactions (#32612)
- fix: turn on compression on stream writers (#34067)
- fix: adding blob memory size in binlog serde (#33324)

See also: #32451, #33547, #33998, #31679
pr: #32198, #33554, #33997, #32612

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-25 11:16:02 +08:00
Patrick Weizhi Xu
1168e8c107
enhance: [2.4] support integral type for MV and skip MV if there is only one category (#34005) 
pr: #33161
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 19:12:02 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
XuanYang-cn
e55fee6b04
enhance: Add deltaRowCount in l0 compaction (#33843) 
See also: #33998 
pr: #33997

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 19:32:02 +08:00
congqixia
4424c9e5e7
fix: [2.4] Remove loopclosure issue in ChannelManagerImplV2 (#33989) (#34004) 
Cherry-pick from master
pr: #33989
See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 14:42:00 +08:00
XuanYang-cn
dff7ec7bb2
enhance: Donot mark compaction tasks to be timeouted (#33833) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-14 10:45:58 +08:00
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33793) 
issue: #33610
pr: #33792

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 14:14:04 +08:00
chyezh
74b4bd1c77
fix: lost DroppedAt field when DropCollection (#33767) 
issue: #33097
pr: #33766

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:19:56 +08:00
yihao.dai
396f8608dd
fix: Fix multiple vector fields import (#33723) (#33724) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

pr: https://github.com/milvus-io/milvus/pull/33723

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:55:55 +08:00
yihao.dai
b71a404776
fix: Check if the import job exists (#33672) (#33673) 
issue: https://github.com/milvus-io/milvus/issues/33671

pr: https://github.com/milvus-io/milvus/pull/33672

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:50:29 +08:00
yihao.dai
ed1dee9e38
enhance: Support L0 import (#33514) (#33712) 
issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33514

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-08 11:17:52 +08:00
yihao.dai
68e5e6434b
enhance: Print more disk quota info (#33596) (#33597) 
/kind enhancement

pr: https://github.com/milvus-io/milvus/pull/33596

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 16:51:56 +08:00
yihao.dai
30fd4a955a
enhance: Avoid redundant meta operations of import (#33518) (#33519) 
issue: https://github.com/milvus-io/milvus/issues/33513

pr: https://github.com/milvus-io/milvus/pull/33518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-03 14:15:53 +08:00
yihao.dai
8ff5d2793c
fix: Fill stats log id and check validity (#33477) (#33478) 
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

pr: https://github.com/milvus-io/milvus/pull/33477

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-31 14:13:46 +08:00
XuanYang-cn
0c5aaeb34c
enhance: Make channel meta able to writer 200k plus segments (#33300) 
See also: #33125
pr: #33279

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-28 10:09:48 +08:00
SimFG
6585227c9c
fix: [2.4] not found database name in the datacoord meta object (#33412) 
- issue: #33410
- pr: #33411

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-28 10:07:43 +08:00
wayblink
deebae70a7
fix:[cherry-pick]Panic if ProcessActiveStandBy returns error (#33372) 
pr:#33369
issue:#33368

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-05-27 10:13:59 +08:00
jaime
8990b8b051
fix: correct error of metrics stats (#33305) 
issue: #32980
cherry pick from master
pr:  #33075 #33255

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-24 09:15:41 +08:00
XuanYang-cn
e5ca1f7c45
fix: Return error when startup Delete/AddNode fail (#33193) (#33258) 
See also: #33151, #33149
pr: #33193

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-22 14:49:40 +08:00
congqixia
5370c39a23
fix: [2.4] Mark channel checkpoint dropped prevent cp lag metrics leakage (#32454) (#33201) 
Cherry-pick from master
pr: #33198
See also #31506 #31508

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-21 11:57:51 +08:00
congqixia
cf7765915c
enhance: [2.4]Refine channel mgr v2 implementation (#33156) (#33172) 
Cherry-pick from master
pr: #33156
Related to #25309

- Remove ctx from struct
- Add ctx parameters for internal check logic methods
- Add Waitgroup to make sure worker goroutine quit before close returns

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-20 18:25:38 +08:00
wayblink
259bc97a2b
fix: Fix segments lost in flush response (#33061) 
#33055

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-05-15 13:49:34 +08:00
cai.zhang
6ea7633bd5
enhance: Add memory size for binlog (#33025) 
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2024-05-15 12:59:34 +08:00
jaime
f48a7ff8ff
enhance: use Delete instead of DeletePartialMatch to remove metrics (#33029) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-14 18:49:33 +08:00
congqixia
dea7faf394
fix: Remove RLock re-entry in GetNumRowsOfPartition (#33045) 
Resolves #33044

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-14 17:51:32 +08:00
XuanYang-cn
40e5c0be70
fix: Enable to balance released standby channel (#32986) 
See also: #32879

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-14 11:51:32 +08:00
jaime
ba625835bc
enhance: Add metrics for segment index files size (#32979) 
issue:#32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-13 17:59:32 +08:00
XuanYang-cn
bc015df962
fix: [skip-e2e] Fix unstable policy_test (#32883) 
See also: #32849

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-09 15:39:40 +08:00
congqixia
cedb33ceec
enhance: Improve datacoord segment filtering with collection (#32831) 
See also #32165

This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-08 21:37:29 +08:00
XuanYang-cn
0b0d279515
fix: [skip-e2e] Fix AvgAssignByCount unstable (#32850) 
See also: #32849

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-08 19:09:35 +08:00
yihao.dai
a8db16ad4b
enhance: Change ddl log to info (#32838) 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-08 14:17:50 +08:00
XuanYang-cn
6843d6d376
fix: Compaction trigger choose 2 same segments (#32800) 
DataNode would stuck at compactor try to lock the
same segmentID

See also: #32765

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-07 19:01:31 +08:00
yiwangdr
b1eacb2ae8
feat: datacoord/node watch based on rpc (#32036) 
issue: https://github.com/milvus-io/milvus/issues/25309

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-05-07 15:49:30 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
congqixia
6af5275066
fix: Refine index param check for Create/Alter Index (#32712) 
See also #32711

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-06 15:43:29 +08:00
chyezh
77f4f0c5a8
fix: gc only do once (#32722) 
issue: #29655

produced by pr: #31740

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-30 14:19:27 +08:00
wayblink
42d0412e93
enhance: Add channelCPs in FlushResponse (#32044) 
#32609

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-04-30 09:45:27 +08:00
jaime
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation (#32611) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-04-26 15:53:29 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 
issue: #19095,#29655,#31718

- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.

- File garbage collection is performed in other goroutine.

- Segment Index Recycle clean index file too.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-25 20:41:27 +08:00
wei liu
04f355a802
enhance: Enable alter database props in rootcoord (#32458) 
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-25 10:53:25 +08:00
Ted Xu
fef7812254
enhance: trigger compaction by size (#32326) 
See #32328

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-04-23 12:11:24 +08:00
yiwangdr
037de8e4d3
enhance: speed up minor functions calls in datacoord (#32389) 
Related to https://github.com/milvus-io/milvus/issues/32165

1. nodeid based channel store access should use map access instead of
iteration.

2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-04-20 07:55:21 +08:00