test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 20:09:57 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
20,167 Commits 94 Branches 105 Tags 184 MiB
e5d691d854
Commit Graph

1070 Commits

Author SHA1 Message Date
wayblink
73ffc1b424
Refine clustering_compaction_task retry mechanism (#34194) 
#32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-30 20:22:09 +08:00
Chun Han
96dcee5dff
fix:load major compaction partial result(#34051) (#34052) 
related: #34051

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-06-28 14:04:05 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module (#33881) 
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-25 21:18:15 +08:00
cai.zhang
c65f41dc60
fix: Only sync flushed segments to datanode (#34156) 
issue: #33540

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 20:46:07 +08:00
yihao.dai
940a0ac0a3
fix: Prevent import from generating orphaned files (#34070) 
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: https://github.com/milvus-io/milvus/issues/34068

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 16:58:02 +08:00
zhenshan.cao
d18c49013b
enhance: Refine compaction (#33982) 
issue : https://github.com/milvus-io/milvus/issues/32939

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-06-25 10:08:03 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161) 
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 10:20:01 +08:00
Bingyi Sun
622be36ad7
fix: fix l0 compaction may miss some sealed segments (#33980) 
issue: https://github.com/milvus-io/milvus/issues/33955

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-06-21 16:52:15 +08:00
XuanYang-cn
04edb07d82
enhance: Add deltaRowCount in l0 compaction (#33997) 
See also: #33998

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 17:46:01 +08:00
congqixia
6a15b6e2be
fix: Remove loopclosure issue in ChannelManagerImplV2 (#33989) 
See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-19 21:25:59 +08:00
wayblink
5fac2fa1d2
fix: Panic if ProcessActiveStandBy returns error (#33369) 
#33368

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-19 11:16:00 +08:00
wayblink
5cb0760187
fix: Small fixs of major compaction (#33929) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-18 16:53:58 +08:00
cai.zhang
95148866ed
fix: Don't remove growing L0 segment in datanode metacache (#33829) 
issue: #33540 
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-17 10:09:57 +08:00
yihao.dai
8537f3daeb
enhance: Rename Compaction to CompactionV2 (#33858) 
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: https://github.com/milvus-io/milvus/issues/32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:07:57 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
chyezh
b3ee1a4bd5
fix: lost DroppedAt field when DropCollection (#33766) 
issue: #33097

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:21:56 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
yihao.dai
b1d46eb34b
fix: Fix multiple vector fields import (#33723) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:57:54 +08:00
yihao.dai
eb5d4de390
fix: Check if the import job exists (#33672) 
issue: https://github.com/milvus-io/milvus/issues/33671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:51:55 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
yihao.dai
3540eee977
enhance: Support L0 import (#33514) 
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-07 14:17:20 +08:00
smellthemoon
c61fb1eff5
enhance: do check when add not empty logpath (#33640) 
meta only store logid

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-07 10:19:51 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
cai.zhang
cfea3f43cf
fix: Don't sync L0 segments to channel watcher (#33664) 
issue: #33540

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-06 15:59:50 +08:00
cai.zhang
feeb869ff9
enhance: Remove compaction plans on the datanode (#33548) 
issue: #33546

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-05 20:27:51 +08:00
yihao.dai
01ce32caa1
enhance: Print more disk quota info (#33596) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 16:51:56 +08:00
yihao.dai
35532a3e7d
fix: Fill stats log id and check validity (#33477) 
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 11:17:56 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability (#33265) 
issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-06-05 10:17:50 +08:00
yiwangdr
180d754158
fix: speed up segment lookup via channel name in datacoord (#33530) 
issue: #33342

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-06-03 14:47:47 +08:00
yihao.dai
7a2127b09f
enhance: Avoid redundant meta operations of import (#33518) 
issue: https://github.com/milvus-io/milvus/issues/33513

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-03 14:15:53 +08:00
cai.zhang
77637180fa
enhance: Periodically synchronize segments to datanode watcher (#33420) 
issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-30 13:37:44 +08:00
congqixia
54797b4286
enhance: Refine frequent log in datacoord (#33449) 
This PR changes:
- Frequent `ListIndexes` success log to debug level
- Aggregate collection missing log after collection dropped in
`meta.GetCollectionIndexFilesSize`

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-29 19:15:43 +08:00
jaime
3d29907b6e
enhance: decrease cpu overhead during filter segments on datacoord (#33130) 
issue: #33129

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-28 19:17:43 +08:00
XuanYang-cn
5e39aa9272
enhance: Make channel meta able to writer 200k plus segments (#33279) 
See also: #33125

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-28 12:33:42 +08:00
SimFG
2453181218
fix: not found database name in the datacoord meta object (#33411) 
- issue: #33410

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-28 10:09:48 +08:00
yiwangdr
e895cfed84
fix: reduce redundant map operations in datacoord (#33343) 
More refactories will be added.
issue: #33342

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-05-24 12:47:40 +08:00
yihao.dai
7730b910b9
enhance: Decouple compaction from shard (#33138) 
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-24 09:07:41 +08:00
cai.zhang
be77ceba84
enhance: Use proto for passing info in cgo (#33184) 
issue: #33183

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-23 10:31:40 +08:00
XuanYang-cn
22bddde5ff
enhance: Tidy compactor and remove dup codes (#32198) 
See also: #32451

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-23 09:53:40 +08:00
XuanYang-cn
819a624753
fix: Return error when startup Delete/AddNode fail (#33193) 
See also: #33151, #33149

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-22 11:17:38 +08:00
cai.zhang
ed39a38953
enhance: Reduce the frequency of logs describing indexing failures (#33212) 
issue: #33001 #33102

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-21 19:27:39 +08:00
yihao.dai
017fd7bc25
enhance: Select L2 segments in L0Compaction as well (#32991) 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-21 16:13:39 +08:00
congqixia
f31a20faad
fix: [Backport] Mark channel checkpoint dropped prevent cp lag metrics leakage (#32454) (#33198) 
Cherry-pick from 2.3
pr: #32454
See also #31506 #31508

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-21 11:59:39 +08:00
jaime
0d99db23b8
fix: metrics leak on the coord nodes (#33075) 
issue: #32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-20 22:03:39 +08:00
congqixia
f76d16780b
enhance: Refine channel mgr v2 implementation (#33156) 
Related to #25309

- Remove ctx from struct
- Add ctx parameters for internal check logic methods
- Add Waitgroup to make sure worker goroutine quit before close returns

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-20 14:13:38 +08:00
yihao.dai
32560263fa
enhance: Query slot for compaction task (#32881) 
Query slot of compaction in datanode, and transfer the control logic for
limiting compaction tasks from datacoord to the datanode.

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-17 18:19:38 +08:00
congqixia
8cf2cf5c94
enhance: Add go-deadlock as unittest only dependency (#33063) 
See also #33062

This PR:

- Add `lock.RWMutex` & `lock.Mutex` alias to switch implementation based
  on build flags
- When build flags has `test` in it, use `go-deadlock` to detect
  possible deadlocks
- Replace all `sync.RWMutex` & `sync.Mutex` in datacoord pkg

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-15 16:33:34 +08:00
cai.zhang
4382cf5283
enhance: Reduce the frequency of index ralted logs (#33050) 
issue: #33001

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-15 14:37:34 +08:00
wayblink
259bc97a2b
fix: Fix segments lost in flush response (#33061) 
#33055

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-05-15 13:49:34 +08:00
cai.zhang
6ea7633bd5
enhance: Add memory size for binlog (#33025) 
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2024-05-15 12:59:34 +08:00